test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-02 03:48:37 +08:00
Code Issues Actions 12 Packages Projects Releases Wiki Activity
21,008 Commits 94 Branches 105 Tags 184 MiB
bf956a3ec2
Commit Graph

434 Commits

Author SHA1 Message Date
yiwangdr
32cff25f97
enhance: decrease coordinator init time (#29822) 
This PR mainly improve two items:
1. Target observer should refresh loading status during init time. An
uninitialized loading status blocks search/query. Currently, the target
observer refreshes every 10 seconds, i.e. we'd need to wait for 10s for
no reason. That's also the reason why we constantly see false log
"collection unloaded" upon mixcoord restarts.
2. Delete session when service is stopped. So that the new service
doesn't need to wait for the previous session to expire (~10s).

Item 1 is the major improvement of this PR, which should speed up init
time by 10s.
Item 2 is not a big concern in most cases as coordinators usually shut
down after stop(). In those cases, coordinator restart triggers serverID
change which further triggers an existing logic that deletes expired
session. This PR only fixes rare cases where serverID doesn't change.

integration test:
`go test -tags dynamic -v -coverprofile=profile.out -covermode=atomic
tests/integration/coordrecovery/coord_recovery_test.go -timeout=20m`
Performance after the change:
Average init time of coordinators: 10s
Hardware: M2 Pro
Test setup: 1000 collections with 1000 rows (dim=128) per collection.


issue: #29409

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2024-02-05 14:00:12 +08:00
pingliu
5cdf0f2490
doc: fix standalone stop cannot start issue (#30481) 
Signed-off-by: ping.liu <ping.liu@zilliz.com>
 2024-02-02 21:05:05 +08:00
pingliu
e4a033858e
doc: Add Milvus standalone all in one script (#30459) 
Signed-off-by: ping.liu <ping.liu@zilliz.com>
 2024-02-02 18:05:04 +08:00
yah01
ec688f5bf6
fix: failed to download OpenDAL (#30380) 
OpenDAL's url has been changed
fix #30379

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-02-02 14:35:04 +08:00
sammy.huang
265453f400
enhance: [skip e2e]revert back to original way to archive, upload and download (#30248) 
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
 2024-02-01 10:15:04 +08:00
congqixia
9f8eb0e527
enhance: make integration test case timeout configurable (#30073) 
currently integration test may timeout if any case run time is above 3
minutes. This duration was hard coded.

This PR change this duration into a customized parameter and could be
passed via test running commands.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-18 12:22:54 +08:00
yah01
031243fee7
feat: support mmap for marisa trie (#29613) 
this supports mmap for marisa trie index
related https://github.com/milvus-io/milvus/issues/21866

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-11 10:22:50 +08:00
Jiquan Long
4b3de64733
enhance: add rust to install_dep.sh (#29586) 
fix: #29585

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-01-07 20:04:12 +08:00
yihao.dai
3561586edf
feat: Add import reader for binlog (#28910) 
This PR defines the new import reader interfaces and implement a binlog
reader for import.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-01-05 11:48:47 +08:00
PowderLi
5f00bad4b8
fix: link with install path's libblob-chunk-manager (#29496) 
issue: #29494

1. link with install path's libblob-chunk-manager
2. performance of `ShouldBindWith` is better than `ShouldBindBodyWith`
3. the middleware shouldn't read the unrefreshed parameter repeatly

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-12-31 20:02:48 +08:00
wei liu
e41fd6fbde
enhance: Move proxy client manager to util package (#28955) 
issue:  #28898

This PR move the `ProxyClientManager` to util package, in case of
reusing it's implementation in querycoord

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-12-20 19:22:42 +08:00
zhagnlu
56d7225673
fix: Add LD_LIBRARY_PATH when local start milvus (#29287) 
#28165

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-12-19 10:08:41 +08:00
zhagnlu
a602171d06
enhance: Refactor runtime and expr framework (#28166) 
#28165

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-12-18 12:04:42 +08:00
Jiquan Long
ed79505d31
enhance: use specific rust version to compile opendal (#29106) 
/kind improvement

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-12-13 16:34:38 +08:00
XuanYang-cn
d2207cac27
fix: Fix rustup update (#29093) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-12-11 11:36:37 +08:00
Bingyi Sun
36f69ea031
feat: integrate storagev2 in building index of segcore (#28768) 
issue: https://github.com/milvus-io/milvus/issues/28655

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-12-05 16:48:54 +08:00
congqixia
e79c3eaa90
enhance: [skip e2e] Update install_deps.sh cmake version (#28946) 
cmake version needs to be greater or equal than 3.26.4, sync
`install_deps.sh` scripts version to 3.26.5

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-04 18:26:34 +08:00
PowderLi
cac802ef7f
enhance: use already installed vcpkg (#28703) 
issue #28686
1. Update Builder gpu image changes, see changes #28505
2. update azure-identity-cpp from beta to release

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-11-30 15:58:32 +08:00
Xiaofan
f13ef1665a
fix : macos clean compilation fail, brew install some missed library (#28717) 
fix macOS compilation fail, related #28715
1) install_deps.sh missed to install some dependencies 
2) llvm can not be found without a soft link

Signed-off-by: xiaofan luan <xiaofanluan@xiaofandeMacBook-Pro.local>
Co-authored-by: xiaofan luan <xiaofanluan@xiaofandeMacBook-Pro.local>
 2023-11-27 10:06:26 +08:00
yah01
da03feb79a
enhance: upgrade OpenDAL to v0.42.0 and optimize the building (#28556) 
- Upgrade OpenDAL to v0.42.0
- Clone OpenDAL with depth 1
- Add build log for OpenDAL

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-20 14:58:20 +08:00
yah01
6f59510723
Upgrade go to 1.20 (#24390) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-11-10 07:46:18 +08:00
Enwei Jiao
cbf25f8585
Use OpenDAL 0.42 (#28258) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-11-09 11:26:21 +08:00
PowderLi
9f9726f8b7
print azure sdk log (#28240) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-11-08 17:50:18 +08:00
PowderLi
7bb0fa9c70
reduce useless ObjectExists (#28156) 
replace ListBlobs() with GetProperties()
unified style std::string& / char*
config azure requestTimeoutMs

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-11-07 16:32:20 +08:00
wei liu
0c69f48ba4
refine stop order (#28016) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-01 11:40:15 +08:00
Enwei Jiao
8ae9c947ae
Use OpenDAL to access object store (#25642) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-11-01 09:00:14 +08:00
XuanYang-cn
6ce98cf542
Fix sript stop unable to find milvus process (#27957) 
/kind improvement

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-10-26 20:44:14 +08:00
Enwei Jiao
4faba61647
Upgrade conan to 1.61.0 (#27870) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-10-24 17:50:10 +08:00
jaime
ac2d1bb5c2
Support receive signals from parent process (#27756) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-10-18 20:20:11 +08:00
chen
a66d1cf4ff
remove unused code (#27134) 
Signed-off-by: xiedeyantu <czjourney@163.com>
 2023-10-16 23:06:08 +08:00
jaime
ec1fe3549e
Add a stop hook to clean session (#27564) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-10-16 10:24:10 +08:00
BUPTAnderson
ff19cb2719
Fix cannot find -latomic (#27598) 
Signed-off-by: hg.anderson <yunlongkong@126.com>
 2023-10-15 20:08:04 +08:00
zhagnlu
2efc4ec703
Remove deprecated cmake parameter (#27695) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-10-14 02:57:35 +08:00
Sheldon
fc1d59f64b
only build cpu in default devcontainer (#27570) 
now the default devcontainer will use gpubuilder, which is so big,
almost >8G

Signed-off-by: Sheldon <chuanfeng.liu@zilliz.com>
 2023-10-13 14:23:36 +08:00
xige-16
bf46ffd6c4
Fix build with diskann failed (#27672) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-10-12 19:25:35 +08:00
Enwei Jiao
0f2f4a0a75
Remove useless parameters for Makefile (#27622) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-10-11 20:45:35 +08:00
PowderLi
07809880b2
fix vcpkg install path (#27635) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-10-11 17:27:33 +08:00
PowderLi
11b521392e
install ninja in advance for arm64 (#27579) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-10-11 11:51:32 +08:00
Enwei Jiao
42db013a7b
Support llvm17 on MacOS (#27582) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-10-11 02:47:33 +08:00
foxspy
370b6fde58
milvus support multi index engine (#27178) 
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-22 09:59:26 +08:00
PowderLi
4feb3fa7c6
support azure (#26398) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-09-19 10:01:23 +08:00
Jiquan Long
36e29ef859
Auto generate parser code (#26716) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-08-31 11:13:01 +08:00
congqixia
d343888f3c
Fix pkg codecov & static-check script (#26720) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-31 09:59:01 +08:00
PowderLi
30ca45868d
install zip because we want to use vcpkg (#26664) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-08-30 14:19:00 +08:00
Enwei Jiao
a73eb1c831
Sync pb files (#26643) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-08-29 12:36:26 +08:00
Xiaofan
7b3a1dc93f
Fix load index panic (#26457) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-08-21 09:56:20 +08:00
Enwei Jiao
533f0ddf6d
Add amazonlinux image, prepare to remove Centos7 image, remove openblas image (#26368) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-08-18 18:32:19 +08:00
aoiasd
54d6f0ea23
Add rpath ldflag for unittest (#26269) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-08-11 11:25:32 +08:00
chen
703f0a187c
optimise install_deps.sh (#25942) 
Signed-off-by: xiedeyantu <czjourney@163.com>
 2023-08-08 11:15:14 +08:00
Enwei Jiao
f97d5a7d08
Fix compile failed with llvm16 (#26112) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-08-03 17:29:07 +08:00
zhagnlu
65cb52d06b
Support dynamic simd framework and using term expr as example (#25260) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-07-13 16:22:30 +08:00
Enwei Jiao
4aed32ff61
Use librdkafka for all platform (#25538) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-07-13 15:34:33 +08:00
congqixia
549c4a0e1b
Enlarge integration test timeout (#25283) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-03 17:40:25 +08:00
wei liu
a3437e0ab5
refactor replica selection and retry policy on channel (#24367) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-06-13 10:20:37 +08:00
congqixia
dacf8b305d
Fix list milvus-proto go.mod when try to parse API_VERSION (#24788) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-10 19:30:36 +08:00
congqixia
93ee113c12
Fix update-api-version.sh for master branch (#24773) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-09 13:10:35 +08:00
congqixia
41af0a98fa
Use go-api/v2 for milvus-proto (#24770) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-09 01:28:37 +08:00
Enwei Jiao
4048caa8ca
Use only one go.mod (#24669) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-06-07 14:22:36 +08:00
Enwei Jiao
c9d9940918
Remove Ubuntu18.04 support (#24666) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-06-06 14:06:36 +08:00
Bryanw
11db41745d
add pkg-config to brew install when using x86 Mac (#24627) 
Signed-off-by: bryanwux <bryanwu610@gmail.com>
 2023-06-05 15:34:36 +08:00
zhenshan.cao
f7924724b0
Add consistency_level paramter in search/query request (#24499) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-05-30 21:01:29 +08:00
congqixia
96d07edbfb
Add coverage report for integration test (#24428) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-26 13:45:26 +08:00
congqixia
52191fe3e1
Seperate integration test package to resolve interference between cases (#24331) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-23 17:51:25 +08:00
congqixia
3a66e1de65
Use suite for integration test (#24253) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-19 15:57:24 +08:00
zhagnlu
113f9a0ebc
Support SIMD of several Expr (#23715) (#23717) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-05-12 14:11:20 +08:00
congqixia
eb27a8a8c0
Fix integration test not wait for index built (#24037) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-12 09:51:25 +08:00
congqixia
db5c4052da
Execute set -e after source setenv.sh (#23974) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-10 16:05:20 +08:00
congqixia
38e60804a4
Add integration test workflow (#23976) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-09 19:18:41 +08:00
Enwei Jiao
2ced6660e2
Simplify makefile (#23583) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-04-21 14:16:33 +08:00
zhenshan.cao
03ec804e68
Change default shard number to 1 (#23586) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-04-21 07:08:32 +08:00
Enwei Jiao
967a97b9bd
Support json & array types (#23408) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
 2023-04-20 11:32:31 +08:00
Enwei Jiao
f14c221123
Revert "Opt compile time for Mac (#23453)" (#23533) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-04-19 16:52:30 +08:00
Enwei Jiao
e9dcba3050
Opt compile time for Mac (#23453) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-04-19 10:24:30 +08:00
Enwei Jiao
288582b2d9
Optimize mac build time (#23390) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-04-14 17:56:29 +08:00
congqixia
5276529524
Make pkg code change trigger ci workflow (#23312) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-11 10:32:31 +08:00
XuanYang-cn
b90fa5f459
Refine codes of datanode buffer (#23168) 
This PR refines deltabuffer of datanode:
- Add last sync time for compacted segment, see also: #23210
- Ensure all deltabuffermanager handles all delete related operations
- Change usedMemory to atomic.Int64
- Remove allocator in delete buffer

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-04-10 18:42:30 +08:00
Enwei Jiao
1fc33e2e22
split go and cpp ut (#23239) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-04-06 15:56:28 +08:00
Ji Bin
0d0dc95f9d
improvement for macos build script (#23082) 
Signed-off-by: Ji Bin <matrixji@live.com>
 2023-03-29 10:10:01 +08:00
Gao
8b3e5189e1
Add querynode plugin (#22379) 
Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-03-24 15:21:59 +08:00
Enwei Jiao
fd974df09f
OTEL integration in cpp (#22175) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-03-21 16:31:57 +08:00
congqixia
2a0ad67021
Disable go cache test (#22816) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-03-20 10:27:56 +08:00
Jiquan Long
8139106b51
Feat: count entities by expression (#22765) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-03-16 19:31:55 +08:00
jaime
d126f06946
Decouple mq module from internal proto definition (#22536) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-03-04 23:21:50 +08:00
Enwei Jiao
6fa8d62879
Lock conan version at 1.58.0 (#22503) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-03-01 19:59:47 +08:00
Enwei Jiao
b25b3ef431
Integreation with Velox (#22102) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-02-16 17:26:35 +08:00
Enwei Jiao
5351fee733
Upgrade gcc11 for centos7 (#22190) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-02-15 17:16:33 +08:00
Enwei Jiao
e1ec24895d
Upgrade gcc in CentOS (#22086) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-02-09 17:32:31 +08:00
congqixia
66feb295da
Use AdressSantizer build when cpp files changed (#21809) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-01-19 13:43:44 +08:00
Enwei Jiao
fb42466c65
Use opentelemetry (#21509) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-01-12 16:09:39 +08:00
cai.zhang
e127cf7b99
Reset indexpb for upgrade (#21620) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-11 14:35:40 +08:00
cai.zhang
aa203acfb3
Low IndexCoord weight (#21548) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-06 14:21:37 +08:00
cai.zhang
e5f408dceb
Merge IndexCoord and DataCoord (#21267) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-04 19:37:36 +08:00
Jiquan Long
d7156812c1
Try using ASAN in ci ut (#21089) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2022-12-29 15:29:31 +08:00
SimFG
6a29a964df
Fix queryCoord panic during query node down (#21400) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-12-28 10:17:30 +08:00
Enwei Jiao
62a098d42d
Fix dev container make failed (#21301) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-12-19 16:07:25 +08:00
Enwei Jiao
ae41a6583f
Fix compile on M1 (#21298) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-12-19 15:05:24 +08:00
Enwei Jiao
d553b5108e
Fix jemalloc.so not found (#21106) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-12-13 11:27:21 +08:00
SimFG
f8cff79804
Support the graceful stop for the query node (#20851) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-12-06 22:59:19 +08:00
Enwei Jiao
9b61170bf0
Fix compile proto failed (#20820) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-11-28 09:47:14 +08:00
congqixia
83d3007744
Fix install linux dep priviledge (#20772) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-11-24 19:49:11 +08:00