cai.zhang
a5be322ab2
enhance: Don't set the timeout for SyncSegments ( #34226 )
...
issue: #32809
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-02 14:28:07 +08:00
yihao.dai
ff51c7e628
fix: Fix panic due to failed to seek ( #34229 )
...
Converting the same msgposition's vchannel to a pchannel multiple times
would result in an invalid pchannel, leading to seek failure and panic.
This PR:
1. Make a copy of msgposition in msgdispatcher.
2. Check if channel is already a pchannel, no further channel conversion
is performed.
issue: https://github.com/milvus-io/milvus/issues/34221
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-01 16:08:12 +08:00
aoiasd
186757e622
enhance: support mark error as user error ( #33498 )
...
relate: https://github.com/milvus-io/milvus/issues/33492
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-07-01 14:56:12 +08:00
jaime
d6afb31b94
enhance: make subfunctions of datanode component modular ( #33992 )
...
issue: #33994
also remove deprecated channel manager based on the etcd implementation
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-01 14:46:07 +08:00
jaime
0426390f06
enhance: improve check health ( #33800 )
...
issue: #34264
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-01 10:16:06 +08:00
wayblink
73ffc1b424
Refine clustering_compaction_task retry mechanism ( #34194 )
...
#32939
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-30 20:22:09 +08:00
XuanYang-cn
dda70aa81b
fix: LegacyVersionWithoutRPCWatch default value to 2.4.1 ( #34184 )
...
See also: #31933
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-06-27 19:36:06 +08:00
chyezh
d2bc4a53be
enhance: implement rmq and pulsar as wal ( #34046 )
...
issue: #33285
- use reader but not consumer for pulsar
- advanced test framework
- move some streaming related package into pkg
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-27 15:11:05 +08:00
cqy123456
51ebe95be5
enhance: remove CheckVecIndexWithDataTypeExist function in pkg and remove some cgo call ( #34102 )
...
issue: https://github.com/milvus-io/milvus/issues/22837
related pr: https://github.com/milvus-io/milvus/pull/34104
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-26 21:13:44 +08:00
cqy123456
381d43717d
enhance: update new mmap config parmeters version ( #34143 )
...
issue: https://github.com/milvus-io/milvus/issues/32984
related pr: https://github.com/milvus-io/milvus/pull/34145
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-25 21:20:05 +08:00
jaime
9630974fbb
enhance: move rocksmq from internal to pkg module ( #33881 )
...
issue: #33956
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-25 21:18:15 +08:00
wayblink
f9a0f7bb25
Add an option to enable/disable vector field clustering key ( #34097 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-25 18:52:04 +08:00
jaime
d08cb885ca
enhance: enable flush rate limiter of collection level ( #33837 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-24 10:52:03 +08:00
Patrick Weizhi Xu
b961767005
enhance: support integral type for MV and skip MV if there is only one category ( #33161 )
...
issue: #29892
---------
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-06-24 10:20:01 +08:00
Chun Han
ca7ef26e4b
fix: sync part stats task cannot be finished( #30376 ) ( #34027 )
...
related: #30376
also: refine log output for query_coord task by rephrasing action string
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-06-24 10:16:02 +08:00
chyezh
259a682673
enhance: async search and retrieve in cgo ( #33228 )
...
issue: #30926 , #33132
related pr: #33133
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-22 09:38:02 +08:00
Ted Xu
958ecd500b
feat: adding grouped virtual resource allocator ( #33669 )
...
See #33559
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-06-21 15:58:01 +08:00
congqixia
cc77363b66
enhance: Set maxPartitionNum default value to 1024 ( #33949 )
...
See also #30059
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-20 09:11:59 +08:00
smellthemoon
2a1356985d
enhance: support null in go payload ( #32296 )
...
#31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-06-19 17:08:00 +08:00
cqy123456
298e50b834
enhance: check index with data type ( #33880 )
...
issue: https://github.com/milvus-io/milvus/issues/22837
related: https://github.com/milvus-io/milvus/pull/33878
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-19 10:15:59 +08:00
chyezh
2f6f964bc8
enhance: [skip e2e] modify gc configuration document ( #33946 )
...
issue: #31740
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-19 09:57:59 +08:00
Gao
a789c60380
enhance: autoindex for multi data type ( #33868 )
...
issue: #22837
contain https://github.com/milvus-io/milvus/pull/33625
https://github.com/milvus-io/milvus/pull/33867
https://github.com/milvus-io/milvus/pull/33911 which already merged to
2.4 branch
Signed-off-by: chasingegg <chao.gao@zilliz.com>
Co-authored-by: foxspy <xianliang.li@zilliz.com>
2024-06-18 21:34:01 +08:00
congqixia
3fdaae8792
fix: Return record with largest timestamp for entires with same PK ( #33936 )
...
See also #33883
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-18 15:55:59 +08:00
cqy123456
32f685ff12
enhance: growing segment support mmap ( #32633 )
...
issue: https://github.com/milvus-io/milvus/issues/32984
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-18 14:42:00 +08:00
wei liu
b72026c859
enhance: Add rbac support on describe/alter database api ( #33803 )
...
issue: #32707
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-17 10:07:57 +08:00
zhagnlu
d43ec4db0b
enhance: support array bitmap index ( #33527 )
...
#32900
---------
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-16 21:51:58 +08:00
wei liu
4987067375
enhance: Execute bloom filter apply in parallel to speed up segment predict ( #33792 )
...
issue: #33610
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-14 11:37:56 +08:00
congqixia
c689ef4822
fix: Remove eviction operations out of lock ( #33834 )
...
See also #33823
`EvictCacheValueByFormat` may be block by on going `CASCacheValue` and
cause possible deadlock
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-13 21:15:55 +08:00
wei liu
ab93d9c23d
enhance: Use BatchPkExist to reduce bloom filter func call cost ( #33611 )
...
issue:#33610
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-13 17:57:56 +08:00
cqy123456
be3559e022
enhance: proxy check hnsw with sparse is legal ( #33697 )
...
issue:https://github.com/milvus-io/milvus/issues/22837
related:https://github.com/milvus-io/milvus/pull/33653
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-12 20:13:55 +08:00
congqixia
b90999b741
fix: Compare config value then swap when caching param value ( #33785 )
...
See also #33784
This PR change the behavior of `SetCacheValue` of config manager:
- Use mutex and map instead of concurrent map for `configCache`
- Compare config raw value before set cache value
With this implementation, concurrent caching & eviction shall always
have current output:
|time|caching |eviction|config |cached |
|----|--------|------- |---------|---------|
|t0 |get | |old value|null |
|t1 |CAS OK | |old value|old value|
|t2 | |update |new value|old value|
|t3 | |eviction|new value|null |
|time|caching |eviction|config |cached |
|----|--------|------- |---------|---------|
|t0 |get | |old value|null |
|t1 | |update |new value|null |
|t2 |CAS fail| |old value|null |
|t3 | |eviction|new value|null |
|time|caching |eviction|config |cached |
|----|--------|------- |---------|---------|
|t0 | |update |new value|null |
|t1 |get | |new value|null |
|t2 |CAS OK | |new value|new value|
|t3 | |eviction|new value|null |
|time|caching |eviction|config |cached |
|----|--------|------- |---------|---------|
|t0 | |update |new value|null |
|t1 |get | |new value|null |
|t2 | |eviction|new value|null |
|t3 |CAS OK | |new value|new value|
|time|caching |eviction|config |cached |
|----|--------|------- |---------|---------|
|t0 | |update |new value|null |
|t1 | |eviction|new value|null |
|t2 |get | |new value|null |
|t3 |CAS OK | |new value|new value|
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-12 18:51:56 +08:00
Chun Han
f7af323d1e
fix: sync partitiion stats blocking balance task( #33741 ) ( #33742 )
...
related: #33741
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-11 14:21:56 +08:00
wayblink
a1232fafda
feat: Major compaction ( #33620 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-10 21:34:08 +08:00
Aldrin
8a4ef1b836
fix: Masked kafka credentials getting printed in logs ( #33730 )
...
issue: https://github.com/milvus-io/milvus/issues/33727
Signed-off-by: Aldrin <imagesai32@gmail.com>
2024-06-10 00:05:54 +08:00
chyezh
f53ab54c5d
enhance: async cgo utility ( #33133 )
...
issue: #30926 , #33132
- implement future-based cgo utility.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-09 22:55:53 +08:00
XuanYang-cn
1629833060
enhance: Add consts of MsgDispatcher to configs ( #33679 )
...
See also: #33676
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-06-07 14:21:59 +08:00
SimFG
ecee7d90d4
enhance: try to speed up the loading of small collections ( #33570 )
...
- issue: #33569
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-06-07 08:25:53 +08:00
cai.zhang
27cc9f2630
enhance: Support analyze data ( #33651 )
...
issue: #30633
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>
2024-06-06 17:37:51 +08:00
aoiasd
b9bafc76b4
enhance: access log support print with consistency level ( #33503 )
...
relate: https://github.com/milvus-io/milvus/issues/33563
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-06-05 17:13:50 +08:00
zhenshan.cao
ac4f3997ce
enhance: Reconstructing Compaction to possess persistence capability ( #33265 )
...
issue #33586
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-06-05 10:17:50 +08:00
jaime
8858fcb40a
fix: fix loaded entity num is inaccurate ( #33521 )
...
issue: #33520
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-04 20:09:54 +08:00
Bingyi Sun
d610fdf033
enhance: change Allocator to generic ( #33581 )
...
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-06-04 19:05:49 +08:00
aoiasd
2422084a29
fix: paramtable cache cause dynamic config non-dynamic ( #33473 )
...
relate: https://github.com/milvus-io/milvus/issues/33461
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-06-04 11:39:46 +08:00
Ted Xu
d0a0eac0a4
enhance: adding virtual resource allocator ( #33508 )
...
See #33559
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-06-03 19:45:46 +08:00
congqixia
2b285e5573
fix: Wrap init segcore tracing with golang timeout ( #33494 )
...
See also #33483
Wrap `C.InitTrace` & `C.SetTrace` with timeout preventing otlp
initializtion hangs forever when endpoint is not set correctly
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-03 19:25:51 +08:00
wei liu
c6a1c49e02
enhance: Use Blocked Bloom Filter instead of basic bloom fitler impl. ( #33405 )
...
issue: #32995
To speed up the construction and querying of Bloom filters, we chose a
blocked Bloom filter instead of a basic Bloom filter implementation.
WARN: This PR is compatible with old version bf impl, but if fall back
to old milvus version, it may causes bloom filter deserialize failed.
In single Bloom filter test cases with a capacity of 1,000,000 and a
false positive rate (FPR) of 0.001, the blocked Bloom filter is 5 times
faster than the basic Bloom filter in both querying and construction, at
the cost of a 30% increase in memory usage.
- Block BF construct time {"time": "54.128131ms"}
- Block BF size {"size": 3021578}
- Block BF Test cost {"time": "55.407352ms"}
- Basic BF construct time {"time": "210.262183ms"}
- Basic BF size {"size": 2396308}
- Basic BF Test cost {"time": "192.596229ms"}
In multi Bloom filter test cases with a capacity of 100,000, an FPR of
0.001, and 100 Bloom filters, we reuse the primary key locations for all
Bloom filters to avoid repeated hash computations. As a result, the
blocked Bloom filter is also 5 times faster than the basic Bloom filter
in querying.
- Block BF TestLocation cost {"time": "529.97183ms"}
- Basic BF TestLocation cost {"time": "3.197430181s"}
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-31 17:49:45 +08:00
congqixia
842fb02f74
fix: Use localStorage path to check disk cap for indexnode ( #33450 )
...
See also #30943 #30944
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-31 10:15:45 +08:00
Jiquan Long
0c5d8660aa
feat: support inverted index for array ( #33452 )
...
issue: https://github.com/milvus-io/milvus/issues/27704
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-05-31 09:47:47 +08:00
Chun Han
416a2cf507
fix: query iterator lack results( #33137 ) ( #33422 )
...
related: #33137
adding has_more_result_tag for various level's reduce to rectify
reduce_stop_for_best
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-05-30 17:51:44 +08:00
cai.zhang
77637180fa
enhance: Periodically synchronize segments to datanode watcher ( #33420 )
...
issue: #32809
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-05-30 13:37:44 +08:00
