XuanYang-cn
68c9e7db8c
fix: Sync dropped segment for dropped partition ( #33331 )
...
See also: #33330
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-06-06 10:25:52 +08:00
sre-ci-robot
fd191dd7db
[automated] Update Knowhere Commit ( #33655 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-06-06 01:53:50 +08:00
cai.zhang
feeb869ff9
enhance: Remove compaction plans on the datanode ( #33548 )
...
issue: #33546
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-06-05 20:27:51 +08:00
cqy123456
703fc73f71
enhance: disk index support binary vector ( #33631 )
...
issue:https://github.com/milvus-io/milvus/issues/22837
related https://github.com/milvus-io/milvus/pull/33575
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-05 19:37:57 +08:00
cai.zhang
412ccfbb20
enhance: Refine IndexNode code and ensure compatibility ( #33458 )
...
issue: #33432 , #33183
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-06-05 19:35:52 +08:00
zhagnlu
8ad26093ba
fix: fix load failure ( #33599 )
...
issue: #33533
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-05 19:19:51 +08:00
aoiasd
b9bafc76b4
enhance: access log support print with consistency level ( #33503 )
...
relate: https://github.com/milvus-io/milvus/issues/33563
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-06-05 17:13:50 +08:00
yihao.dai
01ce32caa1
enhance: Print more disk quota info ( #33596 )
...
/kind enhancement
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-05 16:51:56 +08:00
yihao.dai
bbdf99a45e
fix: Fix import segment size is uneven ( #33605 )
...
The data coordinator computed the appropriate number of import segments,
thus when importing in the data node, one can randomly select a segment.
issue: https://github.com/milvus-io/milvus/issues/33604
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-05 15:41:51 +08:00
Gao
545d4725fb
fix: correct get vector data size for bf16/fp16/binary vector ( #33377 )
...
related #22837
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-06-05 14:31:57 +08:00
congqixia
597f4c5e03
enhance: Make hasMoreResult accurate when hit number larger than limit ( #33609 )
...
See also milvus-io/milvus-sdk-go#756
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-05 11:51:51 +08:00
zhuwenxing
4ec7cbcae7
test: remove some local config ( #33567 )
...
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-06-05 11:27:50 +08:00
yihao.dai
35532a3e7d
fix: Fill stats log id and check validity ( #33477 )
...
1. Fill log ID of stats log from import
2. Add a check to validate the log ID before writing to meta
issue: https://github.com/milvus-io/milvus/issues/33476
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-05 11:17:56 +08:00
zhuwenxing
05a80f4def
test: refine file dir in import test ( #33600 )
...
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-06-05 10:29:51 +08:00
congqixia
25080bb455
enhance: Print UseDefaultConsistency param in read requests ( #33617 )
...
`UseDefaultConsistency` param is crucial for debugging slow query
problems. It could be confusing when guarantee timestamp is 1 while this
param is not logged
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-05 10:27:50 +08:00
GenkenWei
516f39b616
fix: remove date in the end of ubuntu base image version ( #33073 )
...
CVE issues https://github.com/milvus-io/milvus/issues/31997
This PR will fix **32** LOW and **42** MEDIUM CVEs
**ubuntu:focal-20220426 (ubuntu 20.04)**
Total: 99 (UNKNOWN: 0, LOW: 55, MEDIUM: 44, HIGH: 0, CRITICAL: 0)
**ubuntu:focal (ubuntu 20.04)**
Total: 25 (UNKNOWN: 0, LOW: 23, MEDIUM: 2, HIGH: 0, CRITICAL: 0)
---------
Signed-off-by: weizhenkun <zhenkun.wei@qq.com>
Co-authored-by: weizhenkun <zhenkun.wei@qq.com>
2024-06-05 10:25:50 +08:00
zhenshan.cao
ac4f3997ce
enhance: Reconstructing Compaction to possess persistence capability ( #33265 )
...
issue #33586
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-06-05 10:17:50 +08:00
aoiasd
387b7cd7f4
enhance:avoid maintain checkpoint info in sync manager ( #33413 )
...
relate: https://github.com/milvus-io/milvus/issues/32915
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-06-05 10:05:50 +08:00
jaime
8858fcb40a
fix: fix loaded entity num is inaccurate ( #33521 )
...
issue: #33520
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-04 20:09:54 +08:00
Bingyi Sun
d610fdf033
enhance: change Allocator to generic ( #33581 )
...
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-06-04 19:05:49 +08:00
zhagnlu
c6f8a73bb2
enhance: optimize some cache to reduce memory usage ( #33534 )
...
#33533
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-04 14:09:47 +08:00
aoiasd
2422084a29
fix: paramtable cache cause dynamic config non-dynamic ( #33473 )
...
relate: https://github.com/milvus-io/milvus/issues/33461
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-06-04 11:39:46 +08:00
ThreadDao
ac5e098e13
feat: Add e2e test cases for GoSDK ( #33378 )
...
- Add e2e test cases for Go `milvusclient`
- Fix client SparseEmbedding to vector
issue: #33419
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2024-06-04 10:57:47 +08:00
congqixia
7f4698f4a7
enhance: Use map PK to timestamp in buffer insert ( #33566 )
...
Related to #27675
Store pk to minimal timestamp in `inData` instead of bloom filter to
check whether some delete entry hit current insert batch
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-04 10:07:48 +08:00
sre-ci-robot
d25c755480
[automated] Update Knowhere Commit ( #33573 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-06-04 01:55:46 +08:00
wei liu
22a059d4de
enhance: update dependency for blobloom ( #33565 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-03 21:59:46 +08:00
SimFG
44d7e03e56
fix: reset the RootCoordQuotaStates metric before recording this metric ( #33553 )
...
- issue: #33539
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-06-03 21:09:47 +08:00
Ted Xu
d0a0eac0a4
enhance: adding virtual resource allocator ( #33508 )
...
See #33559
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-06-03 19:45:46 +08:00
congqixia
2b285e5573
fix: Wrap init segcore tracing with golang timeout ( #33494 )
...
See also #33483
Wrap `C.InitTrace` & `C.SetTrace` with timeout preventing otlp
initializtion hangs forever when endpoint is not set correctly
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-03 19:25:51 +08:00
wei liu
34c6a989ab
enhance: Avoid load bf in delegator when qn worker has no more memory ( #33557 )
...
query coord send load request to delegator, delegator load bf first,
then forward load request to qn worker. but when qn worker has no more
memory, it will return load failed immediatelly. then delegator roll
back the loaded bf. query coord wil retry the load request, and
delegator will load and roll back bf again and again.
this PR delay the loading bf step until load segment succeed in worker.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-03 19:23:45 +08:00
yiwangdr
180d754158
fix: speed up segment lookup via channel name in datacoord ( #33530 )
...
issue: #33342
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-06-03 14:47:47 +08:00
yihao.dai
7a2127b09f
enhance: Avoid redundant meta operations of import ( #33518 )
...
issue: https://github.com/milvus-io/milvus/issues/33513
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-03 14:15:53 +08:00
XuanYang-cn
0382628668
enhance: Add more tracing for l0 compactor ( #33435 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-06-03 10:19:49 +08:00
smellthemoon
2c7bb0b8ac
fix: replace removeWithPrefix with remove to avoid delete redundantly ( #33328 )
...
#33288
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-05-31 18:05:45 +08:00
wei liu
c6a1c49e02
enhance: Use Blocked Bloom Filter instead of basic bloom fitler impl. ( #33405 )
...
issue: #32995
To speed up the construction and querying of Bloom filters, we chose a
blocked Bloom filter instead of a basic Bloom filter implementation.
WARN: This PR is compatible with old version bf impl, but if fall back
to old milvus version, it may causes bloom filter deserialize failed.
In single Bloom filter test cases with a capacity of 1,000,000 and a
false positive rate (FPR) of 0.001, the blocked Bloom filter is 5 times
faster than the basic Bloom filter in both querying and construction, at
the cost of a 30% increase in memory usage.
- Block BF construct time {"time": "54.128131ms"}
- Block BF size {"size": 3021578}
- Block BF Test cost {"time": "55.407352ms"}
- Basic BF construct time {"time": "210.262183ms"}
- Basic BF size {"size": 2396308}
- Basic BF Test cost {"time": "192.596229ms"}
In multi Bloom filter test cases with a capacity of 100,000, an FPR of
0.001, and 100 Bloom filters, we reuse the primary key locations for all
Bloom filters to avoid repeated hash computations. As a result, the
blocked Bloom filter is also 5 times faster than the basic Bloom filter
in querying.
- Block BF TestLocation cost {"time": "529.97183ms"}
- Basic BF TestLocation cost {"time": "3.197430181s"}
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-31 17:49:45 +08:00
wei liu
322a4c5b8c
enhance: Remove StringPrimaryKey to reduce unnecessary copy and function call cost ( #33486 )
...
issue: #33497
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-31 15:41:45 +08:00
zhuwenxing
6b51bae5be
test: add different nq for restful v2 test ( #33456 )
...
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-05-31 13:57:45 +08:00
zhuwenxing
3336b91ce6
test: add channel exclusive balance test and resource group test ( #33093 )
...
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-05-31 13:55:52 +08:00
zhuwenxing
4159a4d5d7
test: update nightly ci ( #33430 )
...
update nightly ci
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-05-31 13:54:07 +08:00
aoiasd
0cf225fa75
enhance: access log support restful api ( #33155 )
...
relate: https://github.com/milvus-io/milvus/issues/31823
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-31 11:13:44 +08:00
Buqian Zheng
4171414222
enhance: update knowhere version ( #33490 )
...
issue: https://github.com/milvus-io/milvus/issues/33489
update knowhere version to latest. remove usage of `seed_ef` as it be
replaced by existing `ef`.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-05-31 10:17:50 +08:00
congqixia
842fb02f74
fix: Use localStorage path to check disk cap for indexnode ( #33450 )
...
See also #30943 #30944
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-31 10:15:45 +08:00
Jiquan Long
0c5d8660aa
feat: support inverted index for array ( #33452 )
...
issue: https://github.com/milvus-io/milvus/issues/27704
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-05-31 09:47:47 +08:00
sre-ci-robot
23dedc2cbf
Update all contributors
...
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2024-05-30 12:01:00 +00:00
Chun Han
416a2cf507
fix: query iterator lack results( #33137 ) ( #33422 )
...
related: #33137
adding has_more_result_tag for various level's reduce to rectify
reduce_stop_for_best
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-05-30 17:51:44 +08:00
cai.zhang
77637180fa
enhance: Periodically synchronize segments to datanode watcher ( #33420 )
...
issue: #32809
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-05-30 13:37:44 +08:00
zhagnlu
589d4dfd82
enhance: optimize bitmap index ( #33358 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-05-30 13:09:43 +08:00
SimFG
8f46a20957
fix: show empty collection when has granted the all privilege ( #33445 )
...
- issue: #33382
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-29 20:51:44 +08:00
congqixia
54797b4286
enhance: Refine frequent log in datacoord ( #33449 )
...
This PR changes:
- Frequent `ListIndexes` success log to debug level
- Aggregate collection missing log after collection dropped in
`meta.GetCollectionIndexFilesSize`
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-29 19:15:43 +08:00
smellthemoon
08b94ea81d
enhance:change wrong log ( #33447 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-05-29 18:35:44 +08:00
