test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-04 04:49:08 +08:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity
19,928 Commits 96 Branches 105 Tags 185 MiB
5b847e93fc
Commit Graph

19928 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
zhuwenxing
5b847e93fc
test: [cherry-pick]update import test case to support different dim (#33709) (#33866) 
pr: #33709

add test case for https://github.com/milvus-io/milvus/issues/33681

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
(cherry picked from commit ca1f7ab019)
 2024-06-14 16:23:57 +08:00
zhuwenxing
f06be79a4d
test: [cherry-pick]refine load state check in restful test (#33862) 
pr: https://github.com/milvus-io/milvus/pull/33861

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-14 16:22:10 +08:00
zhagnlu
b071ebe49b
fix: readd timestamp index because segment timestamp not ordered (#33857) 
pr: #33856

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-14 14:34:01 +08:00
chyezh
dd6c982bdb
fix: load operation when segment is on releasing (#33699) 
issue: #30857
pr: #31340

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-14 11:47:57 +08:00
XuanYang-cn
dff7ec7bb2
enhance: Donot mark compaction tasks to be timeouted (#33833) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-14 10:45:58 +08:00
congqixia
62bd51ec6d
fix: [2.4] Compare config value then swap when caching param value (#33785) (#33797) 
Cherry-pick from master
pr: #33785
See also #33784

This PR change the behavior of `SetCacheValue` of config manager:

- Use mutex and map instead of concurrent map for `configCache`
- Compare config raw value before set cache value

With this implementation, concurrent caching & eviction shall always
have current output:

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |get     |        |old value|null     |
|t1  |CAS OK  |        |old value|old value|
|t2  |        |update  |new value|old value|
|t3  |        |eviction|new value|null     |

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |get     |        |old value|null     |
|t1  |        |update  |new value|null     |
|t2  |CAS fail|        |old value|null     |
|t3  |        |eviction|new value|null     |

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |        |update  |new value|null     |
|t1  |get     |        |new value|null     |
|t2  |CAS OK  |        |new value|new value|
|t3  |        |eviction|new value|null     |

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |        |update  |new value|null     |
|t1  |get     |        |new value|null     |
|t2  |        |eviction|new value|null     |
|t3  |CAS OK  |        |new value|new value|

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |        |update  |new value|null     |
|t1  |        |eviction|new value|null     |
|t2  |get     |        |new value|null     |
|t3  |CAS OK  |        |new value|new value|

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-13 17:51:57 +08:00
XuanYang-cn
a446e754b4
fix: [2.4]DeleteData merge wrong data casuing data loss (#33821) 
See also: #33819
pr: #33820

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-13 16:07:56 +08:00
wei liu
c9fa167a38
enhance: Add rbac support on describe/alter database api (#33804) 
issue: #32707
pr: #33803

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-13 14:15:56 +08:00
wei liu
25d8b74f71
enhance: Execute bloom filter apply in parallel to speed up segment predict (#33793) 
issue: #33610
pr: #33792

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-13 14:14:04 +08:00
Cai Yudong
ebd0af14f4
enhance: Handle Float16Vector/BFloat16Vector numpy bulk insert as same as BinaryVector (#33760) (#33788) 
pr: #33760
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-06-13 10:49:57 +08:00
congqixia
86f3433053
enhance: [2.4]Use fastjson lib for unmarshal delete log (#33787) (#33802) 
Cherry-pick from master
pr: #33878
```
goos: linux
goarch: amd64
GOMAXPROC=1
cpu: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
BenchmarkJsonSerdeStd             343872              3568 ns/op            1335 B/op         25 allocs/op
BenchmarkJsonSerdeFastjson       5124177               234.9 ns/op            16 B/op          1 allocs/op
```

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-13 10:27:57 +08:00
congqixia
efd1fa8b8a
fix: [2.4] Prevent restart timetick sender creating ut datanode (#33790) (#33801) 
Cherry-pick from master
pr: #33790
See also #33789

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-13 10:03:57 +08:00
sammy.huang
5a14f65ee4
fix:[skip e2e] make more retries on codecov uploader (#33700) 
pr: #33683

Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
 2024-06-12 20:40:01 +08:00
chyezh
74b4bd1c77
fix: lost DroppedAt field when DropCollection (#33767) 
issue: #33097
pr: #33766

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-12 20:19:56 +08:00
Gao
7ef28927e6
fix: [2.4] fix binary vector data size (#33751) 
issue #22837 
pr: #33750

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-06-12 20:11:55 +08:00
wei liu
54feef30e7
enhance: Use BatchPkExist to reduce bloom filter func call cost (#33752) 
issue: #33610
pr: #33611

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-12 17:45:58 +08:00
edward.zeng
eeba85118e
test: [skip e2e] fix the push order for milvus image (#33762) (#33768) 
/cc @yanliang567

Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
 2024-06-12 09:55:55 +08:00
Aldrin
ff9bb17cc5
fix: Masked kafka credentials getting printed in logs (#33747) 
Cherry-pick from master
pr: #33730 
issue: https://github.com/milvus-io/milvus/issues/33727

Signed-off-by: Aldrin <imagesai32@gmail.com>
 2024-06-11 15:59:54 +08:00
SimFG
f664b51ebe
enhance: [2.4] try to speed up the loading of small collections (#33746) 
- issue: #33569
- pr: #33570

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-06-11 15:07:55 +08:00
SimFG
c331aa4ad3
enhance: [2.4] add the includeCurrentMsg param for the Seek method (#33743) 
/kind improvement

- issue: #33325
- pr: #33326

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-06-11 15:01:55 +08:00
congqixia
ee22750104
fix: [2.4] Write back dbid modification for nonDB id collection (#33641) (#33694) 
Cherry-pick from master
pr: #33641
See also #33608

Make `fixDefaultDBIDConsistency` also write back collection dbid
modification when nonDB id collection is found.

This fix shall prevent dropped collections of this kind show up again
after dropping and restart.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-11 11:25:55 +08:00
yihao.dai
396f8608dd
fix: Fix multiple vector fields import (#33723) (#33724) 
1. Fix dim mismatch with multi-vector fields and JSON import
2. Enhance: do not display file ID in GetImportResponse.

issue: https://github.com/milvus-io/milvus/issues/33681,
https://github.com/milvus-io/milvus/issues/33682

pr: https://github.com/milvus-io/milvus/pull/33723

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-10 21:55:55 +08:00
yihao.dai
b71a404776
fix: Check if the import job exists (#33672) (#33673) 
issue: https://github.com/milvus-io/milvus/issues/33671

pr: https://github.com/milvus-io/milvus/pull/33672

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-10 21:50:29 +08:00
yihao.dai
ed1dee9e38
enhance: Support L0 import (#33514) (#33712) 
issue: https://github.com/milvus-io/milvus/issues/33157

pr: https://github.com/milvus-io/milvus/pull/33514

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-08 11:17:52 +08:00
Buqian Zheng
39e341e83a
fix: [2.4] update check for sparse hnsw index (#33714) 
issue: #29419
pr: #33713

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-06-07 18:36:07 +08:00
zhagnlu
729001c3a8
fix: clean vector memory (#33692) (#33708) 
pr: #33692

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-07 18:17:53 +08:00
chyezh
93348af5c0
fix: async warmup will be blocked by state lock (#33687) 
issue: #33685
pr: #33686

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-07 14:23:54 +08:00
XuanYang-cn
26db471968
enhance: [2.4]Add consts of MsgDispatcher to configs (#33680) 
See also: #33676
pr: #33679

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-07 14:21:58 +08:00
zhuwenxing
29efd69857
test: refine import test (#33691) 
pr: #33600 #33356
 2024-06-07 09:21:53 +08:00
zhuwenxing
b78d7edca6
test: add coo format sparse vector in restful test (#33689) 
pr: https://github.com/milvus-io/milvus/pull/33677

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-07 09:19:58 +08:00
cqy123456
3562ef83b2
enhance:[cherry-pick]Allow hnsw to build sparse (#33653) 
issue:https://github.com/milvus-io/milvus/issues/22837
related:https://github.com/milvus-io/milvus/pull/33675

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-06 19:15:51 +08:00
Xiaofan
d331b403c3
enhance: Remove l0 delete cache (#33537) 
Cherry pick from master
pr: #32989
remove l0 cache and build delete pk and ts everytime. this reduce the
memory and also increase the code readability

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
 2024-06-06 17:13:50 +08:00
wei liu
a425a041a8
enhance: Remove unnecessary log info during load segment (#33663) (#33670) 
pr: #33663

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-06 17:11:50 +08:00
foxspy
beaa94f170
enhance: [2.4] support sparse cardinal hnsw index (#33662) 
issue: https://github.com/milvus-io/milvus/issues/29419
pr: https://github.com/milvus-io/milvus/pull/33656

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
Co-authored-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-06-06 14:47:54 +08:00
congqixia
7203ff8898
enhance: [2.4] Print UseDefaultConsistency param in read requests (#33617) (#33644) 
Cherry-pick from master
pr: #33617 
`UseDefaultConsistency` param is crucial for debugging slow query
problems. It could be confusing when guarantee timestamp is 1 while this
param is not logged

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-06 14:19:56 +08:00
Buqian Zheng
0ecdd5a59c
enhance: [2.4] Improve GetVectorById of Sparse Float Vector (#33652) 
issue: https://github.com/milvus-io/milvus/issues/29419
pr: #33209 

codecov will fail due to newly added ut in test_sealed.cpp skipped due
to https://github.com/milvus-io/milvus/issues/33210

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-06-06 13:55:51 +08:00
congqixia
084140b093
enhance: [2.4] Make hasMoreResult accurate when hit number larger than limit(#33609) (#33642) 
Cherry-pick from master
pr: #33609

See also milvus-io/milvus-sdk-go#756

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-06 11:45:51 +08:00
wei liu
0c6354018b
enhance: Avoid load bf in delegator when qn worker has no more memory(#33557) (#33650) 
pr: #33557

query coord send load request to delegator, delegator load bf first,
then forward load request to qn worker. but when qn worker has no more
memory, it will return load failed immediatelly. then delegator roll
back the loaded bf. query coord wil retry the load request, and
delegator will load and roll back bf again and again.

this PR delay the loading bf step until load segment succeed in worker.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-06 10:41:52 +08:00
wei liu
f2917f5bdf
enhance: Remove StringPrimaryKey to reduce unnecessary copy and function call cost (#33486) (#33649) 
issue: #33497
pr: #33486

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-06 10:40:01 +08:00
foxspy
58a7111599
enhance: [cherry-pick] add autoindex mapping for binary/sparse datatype (#33625) 
issue: #22837 
pr: #33624

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
 2024-06-06 10:33:52 +08:00
cqy123456
443197bdfc
enhance: [cherry-pick]disk index support binary_vector (#33575) 
issue:https://github.com/milvus-io/milvus/issues/22837
related https://github.com/milvus-io/milvus/pull/33631

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-05 19:37:57 +08:00
zhagnlu
f94a72252c
enhance: optimize some cache to reduce memory usage (#33560) 
pr: https://github.com/milvus-io/milvus/pull/33534
pr: #33599

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-05 19:21:52 +08:00
Chun Han
627b787aed
fix: query iterator lack results(#33137) (#33422) (#33506) 
related: #33137 
pr: https://github.com/milvus-io/milvus/pull/33422

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-06-05 18:51:52 +08:00
yihao.dai
e81ae1e5a4
fix: Fix import segment size is uneven (#33605) (#33634) 
The data coordinator computed the appropriate number of import segments,
thus when importing in the data node, one can randomly select a segment.

issue: https://github.com/milvus-io/milvus/issues/33604

pr: https://github.com/milvus-io/milvus/pull/33605

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-05 18:49:52 +08:00
yihao.dai
68e5e6434b
enhance: Print more disk quota info (#33596) (#33597) 
/kind enhancement

pr: https://github.com/milvus-io/milvus/pull/33596

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-05 16:51:56 +08:00
zhuwenxing
c39d7c3f49
test: wait all tasks finished in import test (#33629) 
pr: None

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-05 15:51:52 +08:00
Gao
b32c14f96b
fix: [2.4] correct get vector data size for bf16/fp16/binary vector (#33488) 
issue: #22837 
pr: #33377

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-06-05 14:31:57 +08:00
yihao.dai
e282e1408e
enhance: Abstract Execute interface for import/preimport task (#33234) (#33607) 
Abstract Execute interface for import/preimport task, simplify import
scheduler.

issue: https://github.com/milvus-io/milvus/issues/33157

pr: https://github.com/milvus-io/milvus/pull/33234

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-05 11:17:56 +08:00
XuanYang-cn
95582b0208
fix: [2.4] L0 compactor may cause DN OOM (#33564) 
See also: #33547
pr: #33554

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-05 10:51:50 +08:00
aoiasd
aaf6c85095
fix: paramtable cache cause dynamic config non-dynamic (#33473) (#33590) 
relate: https://github.com/milvus-io/milvus/issues/33461
pr: https://github.com/milvus-io/milvus/pull/33473

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-06-05 10:15:51 +08:00