yellow-shine
03627150d3
enhance new jenkinsfile for nightly and e2e ( #35715 )
...
https://github.com/milvus-io/milvus/pull/35672
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-08-27 14:23:05 +08:00
XuanYang-cn
ff68eb53d7
fix: [skip e2e]unstable l0 it ( #35613 )
...
See also: #35617
pr: #35612
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-26 18:53:04 +08:00
sre-ci-robot
216f33c529
[automated] Update Pytest image changes ( #35639 )
...
Update Pytest image changes
See changes:
d7d4525c2esre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-08-22 19:18:58 +08:00
wei liu
421a00bfe8
enhance: Refresh proxy cache after restore rbac meta ( #35636 )
...
issue: #35443
pr: #35635
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-22 19:09:01 +08:00
yanliang567
d7d4525c2e
test:[cherry-pick] Add upsert with autoid and update init collection method ( #35624 )
...
Related issue: https://github.com/milvus-io/milvus/issues/32653
pr: #35596
---------
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2024-08-22 14:00:57 +08:00
wei liu
e2542a1bf5
enhance: Update protobuf-go to protobuf-go v2 ( #34394 ) ( #35555 )
...
issue: #34252
pr: #34394 #35072 #35084
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-21 18:50:58 +08:00
wei liu
4bf4cbad85
enhance: Mark query node as read only after suspend ( #35492 ) ( #35586 )
...
issue: #34985 #35493
pr: #35492
after querynode has been suspended, it's not allow to load
segment/channel on it, which means the node is read only. to be
compatible with resource group design, after query node has been
suspend, we remove it from it's original resource group, make it a read
only query node in replica. then two things will happens:
1. it's original resource group will be lacking of query nodes, query
coord will assign new node to it.
2. querycoord will try to move out all segments/channels after querynode
has been suspended
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-20 19:00:56 +08:00
wei liu
4610dafb2e
enhance: make configure load param feature be compatible with old sdk( #35520 ) ( #35573 )
...
issue: #31570 #35521
pr: #35520 #35546
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-20 18:20:57 +08:00
Chun Han
cf8494ef45
enhance: support httpv1/v2 throttle and add it for httpV2( #35350 ) ( #35504 )
...
related: #35350
pr: https://github.com/milvus-io/milvus/pull/35470
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-08-20 16:32:56 +08:00
XuanYang-cn
4a5e6bc6f6
enhance: Add integration tests for l0 ( #35430 )
...
pr: #35429
See also: #34796
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-19 10:54:55 +08:00
wei liu
248a6ea401
enhance: Add BackupRBAC/RestoreRBAC API to enable rbac backup ( #35444 ) ( #35513 )
...
issue: #35443
pr: #35444
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-18 13:42:55 +08:00
smellthemoon
23052f6ac2
fix: upsert use the previous pk in insert when autoid( #34672 ) ( #35130 )
...
issue: #34668
pr: #34672
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-12 18:58:25 +08:00
zhagnlu
db9e4b898a
enhance: support bitmap index ( #35336 )
...
pr: #32902
cherry-pick bitmap index from master
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-07 20:12:18 +08:00
wei liu
0201e00a2f
enhance: enable to set load config in cluster level ( #35293 )
...
issue: #35170
pr: #35169
This PR enable to set load configs in cluster level, such as replicas
and resource groups. then when load collections will use the load
config.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-07 12:38:21 +08:00
nico
18de5f56ad
test: update test cases ( #35210 )
...
pr: #35218
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-08-07 09:32:17 +08:00
wei liu
2ac1bf7532
enhance: Enable setting the replica number and resource group during collection creation ( #34403 ) ( #34561 )
...
issue: #30040
pr: #34403
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-06 15:06:17 +08:00
sre-ci-robot
fc61f66bb3
[automated] Update Pytest image changes ( #35245 )
...
Update Pytest image changes
See changes:
3a997e8056sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-08-05 09:24:33 +08:00
yellow-shine
3a997e8056
enhance: docker-compose first then try to use docker compose ( #35227 )
...
https://github.com/milvus-io/milvus/issues/35209
https://github.com/milvus-io/milvus/pull/35208
---------
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-08-02 19:37:09 +08:00
nico
9bebca472b
test: update pymilvus version ( #35143 )
...
pr: #34843
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-08-01 11:04:13 +08:00
cai.zhang
c340f387cf
enhance: [cherry-pick] Change the fixed value to a ratio for clustering segment size ( #35075 )
...
issue: #34495
master pr: #35076
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-31 10:32:00 +08:00
zhagnlu
866055527b
enhance: revert remove duplicated pk function ( #35102 )
...
pr: #35103
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-30 20:03:50 +08:00
congqixia
079276c6ff
fix: [2.4] Unify hook singleton implementation in proxy ( #34888 )
...
Cherry-pick from master
pr: #34887
Related to #34885
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-26 18:07:53 +08:00
wayblink
1e5c71d550
fix: [cherry-pick] fix dropped segment still visible after dropped by L2 single compaction ( #35006 )
...
bug: #35003
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-26 13:47:48 +08:00
zhagnlu
6f0f3034c6
fix: fix query count(*) concurrently ( #35010 )
...
pr: #35007
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-26 11:11:46 +08:00
zhagnlu
9eb42bbda0
fix: fix parse plan proto failed for search type ( #34957 )
...
pr: #34944
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-25 20:27:46 +08:00
cai.zhang
74adedf750
enhance: Optimized the GC logic to ensure that memory is released in time ( #34950 )
...
issue: #34703
master pr: #34949
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-24 14:07:43 +08:00
Chun Han
ae1636c2be
fix: refine handling type for segment pruner( #34923 ) ( #34926 )
...
related: #34923
pr: https://github.com/milvus-io/milvus/pull/34925
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-24 12:05:44 +08:00
wayblink
8f3c126129
enhance: [cherry-pick] support l2 single compaction ( #34929 )
...
#34928
pr: #34935
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-24 11:47:50 +08:00
cai.zhang
4ed62e9dbb
enhance: [cherry-pick] Add integration test for clustering compaction ( #34860 )
...
issue: #34792
master pr: #34881
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2024-07-22 17:49:42 +08:00
wayblink
21973a600d
enhance: [cherry-pick] refine clustering compaction basic it ( #34794 )
...
issue: #34792
pr: #34793
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-21 20:05:41 +08:00
yihao.dai
07bc1b6717
enhance: Seal by total growing segments size ( #34692 ) ( #34779 )
...
Seals the largest growing segment if the total size of growing segments
of each shard exceeds the size threshold(default 4GB). Introducing this
policy can help keep the size of growing segments within a suitable
level, alleviating the pressure on the delegator.
issue: https://github.com/milvus-io/milvus/issues/34554
pr: https://github.com/milvus-io/milvus/pull/34692
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-19 18:25:50 +08:00
XuanYang-cn
edefc3cbb5
enhance: [skip e2e]Enable compaction it test ( #34526 ) ( #34720 )
...
pr: #34526
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-16 18:19:38 +08:00
SimFG
13f129aa9c
enhance: [2.4] format the sdk code ( #34699 )
...
- issue: #31293
- pr: #34033
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-16 14:15:38 +08:00
elstic
40f6febf9b
test: [cherry-pick] Fix index creation mismatch for sparse vector columns in multivector case ( #34620 )
...
pr: https://github.com/milvus-io/milvus/pull/34618
Signed-off-by: elstic <hao.wang@zilliz.com>
2024-07-16 09:39:38 +08:00
smellthemoon
0fdb288de7
enhance: upsert support autoid( #30342 ) ( #34633 )
...
pr: #30342
issue: #29258
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
2024-07-15 20:53:39 +08:00
wei liu
cf701a9bf0
enhance: Preserve fixed-size memory in delegator node for growing segment ( #34600 )
...
issue: #34595
pr: #34596
When consuming insert data on the delegator node, QueryCoord will move
out some sealed segments to manage its memory usage. After the growing
segment gets flushed, some sealed segments from other workers will be
moved back to the delegator node. To avoid the frequent movement of
segments, we estimate the maximum growing row count and preserve a
fixed-size memory in the delegator node.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-15 20:51:46 +08:00
zhuwenxing
f2d0517f96
test: [cherry-pick]add testcase for count query ( #34471 )
...
pr: https://github.com/milvus-io/milvus/pull/34453
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-07-08 12:54:12 +08:00
Gao
a60e2a65ff
enhance: change autoindex default metric type ( #34277 )
...
issue: #34304
pr: #34261
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-07-08 10:52:14 +08:00
wei liu
d3e94f9861
enhance: Use Blocked Bloom Filter instead of basic bloom fitler impl ( #34377 )
...
issue: #32995
pr: #33405
To speed up the construction and querying of Bloom filters, we chose a
blocked Bloom filter instead of a basic Bloom filter implementation.
WARN: This PR is compatible with old version bf impl, but if fall back
to old milvus version, it may causes bloom filter deserialize failed.
In single Bloom filter test cases with a capacity of 1,000,000 and a
false positive rate (FPR) of 0.001, the blocked Bloom filter is 5 times
faster than the basic Bloom filter in both querying and construction, at
the cost of a 30% increase in memory usage.
Block BF construct time {"time": "54.128131ms"}
Block BF size {"size": 3021578}
Block BF Test cost {"time": "55.407352ms"}
Basic BF construct time {"time": "210.262183ms"}
Basic BF size {"size": 2396308}
Basic BF Test cost {"time": "192.596229ms"}
In multi Bloom filter test cases with a capacity of 100,000, an FPR of
0.001, and 100 Bloom filters, we reuse the primary key locations for all
Bloom filters to avoid repeated hash computations. As a result, the
blocked Bloom filter is also 5 times faster than the basic Bloom filter
in querying.
Block BF TestLocation cost {"time": "529.97183ms"}
Basic BF TestLocation cost {"time": "3.197430181s"}
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-05 17:04:10 +08:00
wayblink
c62bf8a0b0
fix: [Cherry-pick]Pick major compaction fixs and optimizations ( #34360 )
...
This PR cherry-picks the following commits:
- fix: sync partitiion stats blocking balance task #33742
- fix: Fix meta prefix overlap bug #33830
- fix: Small fixs of major compaction #33929
- fix: Fix memory buffer error & some renaming #33850
- fix: sync part stats task cannot be finished #34027
- Add an option to enable/disable vector field clustering key #34097
- fix: fix error ignore in compactor #34169
- fix:load major compaction partial result #34052
- Use new stream segment reader in clustering compaction #34232
issue: #30633
pr: #33742 #33830 #33929 #33850 #34027 #34097 #34169 #34052 #34232
---------
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Chun Han <116052805+MrPresent-Han@users.noreply.github.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-03 09:53:37 +08:00
elstic
fb88267855
test: [cherry-pick] update test case ( #34109 )
...
pr: https://github.com/milvus-io/milvus/pull/34108
Signed-off-by: elstic <hao.wang@zilliz.com>
2024-07-02 22:04:08 +08:00
wayblink
99586066f5
feat: [cherry-pick] Major compaction ( #34326 )
...
This PR cherry-picks the following commits:
fix: speed up segment lookup via channel name in datacoord (#33530 )
needed by the next commit
feat: Major compaction (#33620 )
issue: #30633
pr: #33620
---------
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: yiwangdr <80064917+yiwangdr@users.noreply.github.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-07-02 18:29:01 +08:00
zhenshan.cao
14a11e379c
enhance: Refactor Compaction to enable persistence( #33265 ) ( #34268 )
...
pr : #33265
issue #33586
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-07-01 19:32:07 +08:00
zhuwenxing
9cb3658d74
test: [cherry-pick]remove dup code ( #34243 )
...
pr: https://github.com/milvus-io/milvus/pull/34242
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-06-28 10:44:05 +08:00
yihao.dai
b1e74dc7cb
enhance: [cherry-pick] Decouple compaction from shard ( #34157 )
...
This PR cherry-picks the following commits:
- Implement task limit control logic in datanode.
https://github.com/milvus-io/milvus/pull/32881
- Load bf from storage instead of memory during L0 compaction.
https://github.com/milvus-io/milvus/pull/32913
- Remove dependencies on shards (e.g. SyncSegments, injection).
https://github.com/milvus-io/milvus/pull/33138
- Rename Compaction interface to CompactionV2.
https://github.com/milvus-io/milvus/pull/33858
- Remove the unused residual compaction logic.
https://github.com/milvus-io/milvus/pull/33932
issue: https://github.com/milvus-io/milvus/issues/32809
pr: https://github.com/milvus-io/milvus/pull/32881 ,
https://github.com/milvus-io/milvus/pull/32913 ,
https://github.com/milvus-io/milvus/pull/33138 ,
https://github.com/milvus-io/milvus/pull/33858 ,
https://github.com/milvus-io/milvus/pull/33932
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-25 20:22:03 +08:00
sammy.huang
2ecfe1599c
[automated] Update cpu Builder image changes ( #34079 )
...
Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
2024-06-25 19:02:03 +08:00
yanliang567
59d910320d
test:[cherry-pick]Update tests for range search and add test for query with dup ids ( #34069 )
...
related issue: https://github.com/milvus-io/milvus/issues/33883
pr: #34057
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2024-06-24 11:34:03 +08:00
Jiquan Long
22e6807e9a
feat: support inverted index for array ( #33452 ) ( #34053 )
...
pr: https://github.com/milvus-io/milvus/pull/33184
pr: https://github.com/milvus-io/milvus/pull/33452
pr: https://github.com/milvus-io/milvus/pull/33633
issue: https://github.com/milvus-io/milvus/issues/27704
Co-authored-by: xiaocai2333 <cai.zhang@zilliz.com>
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
2024-06-24 10:50:03 +08:00
zhuwenxing
630a726f35
test: refine restful testcases trace ( #34065 )
...
pr: https://github.com/milvus-io/milvus/pull/34066
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-06-24 10:40:03 +08:00
wei liu
061a00c58f
enhance: Enable database level replica num and resource groups for loading collection ( #33052 ) ( #33981 )
...
pr: #33052
issue: #30040
This PR introduce two database level props:
1. database.replica.number
2. database.resource_groups
User can set those two database props by AlterDatabase API, then can
load collection without specified replica_num and resource groups. then
it will use database level load param when try to load collections.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-21 16:56:02 +08:00
