wayblink
8f3c126129
enhance: [cherry-pick] support l2 single compaction ( #34929 )
...
#34928
pr: #34935
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-24 11:47:50 +08:00
wei liu
c13c48d99a
fix: Failed to unmarshal field stats's bloom filter ( #34922 )
...
pr #34377 introduce this issue, which miss some new changes during the
cherry-pick
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-23 16:45:47 +08:00
wei liu
8c96026722
fix: Segment may bounce between delegator and worker ( #34904 )
...
issue: #34595
pr: #34830
pr#34596 to we add an overloaded factor to segment in delegator, which
cause same segment got different score in delegator and worker. which
may cause segment bounce between delegator and worker.
This PR use average score to compute the delegator overloaded factor, to
avoid segment bounce between delegator and worker.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-23 15:57:49 +08:00
chyezh
0f1bb9c697
enhance: recover stack info when non-cgo thread crash ( #34866 )
...
issue: #34864
pr: #34865
Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-23 14:21:46 +08:00
aoiasd
48063252b1
fix: [Cheryy-Pick] evict paramtable cache failed ( #34772 )
...
relate: https://github.com/milvus-io/milvus/issues/33461
pr: https://github.com/milvus-io/milvus/pull/34771
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-07-23 12:25:44 +08:00
wei liu
ebbccb870c
fix: Avoid segment lack caused by deduplicate segment task ( #34782 ) ( #34903 )
...
issue: #34781
pr: #34782
when balance segment hasn't finished yet, query coord may found 2 loaded
copy of segment, then it will generate task to deduplicate, which may
cancel the balance task. then the old copy has been released, and the
new copy hasn't be ready yet but canceled, then search failed by segment
lack.
this PR set deduplicate segment task's proirity to low, to avoid balance
segment task canceled by deduplicate task.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-23 11:06:15 +08:00
yellow-shine
0951fe4e64
feat: [skip e2e] effective way to use cache ( #34859 )
...
pr: https://github.com/milvus-io/milvus/pull/34774
issue: https://github.com/milvus-io/milvus/issues/34876
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-07-23 10:49:51 +08:00
cai.zhang
6986dfdd5b
enhance:[cherry-pick]Send flush signal when the water level reaches the high watermark ( #34908 )
...
issue: #30633
master pr: #34907
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-23 10:27:42 +08:00
cai.zhang
88da0251af
fix: [cherry-pick] Get current index version from knowhere ( #34902 )
...
issue: #34900
master pr: #34901
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-23 10:25:52 +08:00
jaime
176034d1d0
fix: collection meta is not removed after gc in DataCoord ( #34884 )
...
issue: #34847
pr: #34883
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-22 21:11:48 +08:00
cai.zhang
4ed62e9dbb
enhance: [cherry-pick] Add integration test for clustering compaction ( #34860 )
...
issue: #34792
master pr: #34881
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2024-07-22 17:49:42 +08:00
SimFG
4e95f3835a
fix: [2.4] execute method should return an error when result is a failure ( #34872 )
...
- issue: #34812
- pr: #34813
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-22 12:13:42 +08:00
congqixia
7540714c1b
enhance: [2.4] Add l0 segment entry num quota ( #34733 ) ( #34837 )
...
Cherry-pick from master
pr: #34733
See also #34670
This PR add quota configuration for l0 segment entry number per
collection. If l0 compaction cannot keep up the insertion/upsertion
rate, this feature could back press the related rate.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-22 12:05:43 +08:00
Patrick Weizhi Xu
606bdd7d41
feat: [restful v2][2.4] add partition key isolation prop ( #34869 )
...
issue: https://github.com/milvus-io/milvus/issues/34332
pr: https://github.com/milvus-io/milvus/pull/34710
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit 703feaf6b4a94d1747141cea073b5dcfa36eac73)
2024-07-22 11:43:41 +08:00
yihao.dai
ffee6b9c27
enhance: Optimize ConsistencyLevel and GuaranteeTimestamp log ( #34738 ) ( #34824 )
...
/kind improvement
pr: https://github.com/milvus-io/milvus/pull/34738
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-22 11:32:04 +08:00
wayblink
9b3bd650e6
fix: [cherry-pick] fix collection is compacting logic ( #34862 )
...
related: #34861 #34792
pr: #34855
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-22 11:09:45 +08:00
congqixia
c06a0ebef2
enhance: [2.4] Remove useless ops when there is no write ( #34767 ) ( #34839 )
...
Cherry pick from master
pr: #34767
Related to: #33235
THe querynode pipeline will make map & call ProcessInsert when there is
no write messages. So querynodes will have high CPU usage even when
there is no workload.
This PR check msg length before composing data struct and calling method
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-22 10:23:42 +08:00
wayblink
21973a600d
enhance: [cherry-pick] refine clustering compaction basic it ( #34794 )
...
issue: #34792
pr: #34793
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-21 20:05:41 +08:00
wayblink
c0c3c5f528
enhance: [cherry-pick] refine clustering compaction configs and logs ( #34818 )
...
issue: #30633
pr: #34784
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-21 19:27:41 +08:00
wayblink
33bbc614df
enhance: [cherry-pick] add ut for clustering_compactor ( #34817 )
...
issue: #34792
pr: #34852
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-21 19:25:41 +08:00
congqixia
d1e0d4ea4a
enhance: [2.4] Add dynamic cgo pool for proxy CGO call ( #34768 ) ( #34842 )
...
Cherry-pick from master
pr: #34768
Related to #34705
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-21 18:05:41 +08:00
Min Tian
8834649650
enhance: [2.4] new knowhere param for range_search ( #34709 )
...
Cherry-pick from master
pr: #34686
issue: #34685
Signed-off-by: min.tian <min.tian.cn@gmail.com>
2024-07-19 19:45:41 +08:00
cai.zhang
323dee2fbc
fix: [cherry-pick] Fix the issue of concurrent packing of the same segment ( #34838 )
...
issue: #34703
master pr: #34840
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-19 18:27:40 +08:00
yihao.dai
07bc1b6717
enhance: Seal by total growing segments size ( #34692 ) ( #34779 )
...
Seals the largest growing segment if the total size of growing segments
of each shard exceeds the size threshold(default 4GB). Introducing this
policy can help keep the size of growing segments within a suitable
level, alleviating the pressure on the delegator.
issue: https://github.com/milvus-io/milvus/issues/34554
pr: https://github.com/milvus-io/milvus/pull/34692
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-19 18:25:50 +08:00
wayblink
8632582520
fix: [cherry-pick] fix unstable ut of compaction pickSlot ( #34829 )
...
bug: #34827
pr: #34828
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-19 17:53:48 +08:00
XuanYang-cn
ff4bd2c9d2
enhance: [cp24]Add ut for l0CompactionTask processExecuting ( #34801 )
...
See also: #34796
pr: #34800
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-19 16:31:41 +08:00
Jiquan Long
34873a7f76
feat: record the duration waiting in the proxy queue ( #34744 ) ( #34788 )
...
fix: https://github.com/milvus-io/milvus/issues/34743
pr: #34744
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-07-19 15:37:44 +08:00
foxspy
1f028f3f4b
enhance: [2.4] add scalar filtering and vector search latency metrics ( #34783 )
...
issue: #34780
pr: #34785
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-07-19 11:49:46 +08:00
Chun Han
e11aab3b37
fix: non-init seg_offset for growing raw-data when doing groupby( #34748 ) ( #34750 )
...
related: #34713
pr: https://github.com/milvus-io/milvus/pull/34748
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-19 10:21:44 +08:00
SimFG
0e226502e4
enhance: [2.4] pick default root password and log level pr ( #34777 )
...
default root password
- issue: #33058
- pr: #34752
set log level
- issue: #34756
- pr: #34757
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-18 13:45:43 +08:00
wayblink
a26e965e6a
enhance:[cherry-pick] Add compaction task slot usage logic ( #34625 )
...
issue: #34544
pr: #34581
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-18 09:55:43 +08:00
cai.zhang
d74a5bdc1d
fix: [cherry-pick] Fix bug where binlogs already flushed with new segment during pack ( #34760 )
...
issue: #34703
master pr: #34762
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-18 09:15:47 +08:00
wayblink
83fc26c31a
fix: [cherry-pick] compaction task not be cleaned correctly ( #34766 )
...
1.fix compaction task not be cleaned correctly
2.add a new parameter to control compaction gc loop interval
3.remove some useless configs of clustering compaction
bug: #34764
pr: #34765
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-17 22:17:42 +08:00
Patrick Weizhi Xu
3cf2e1831a
feat: [cherry-pick][2.4] support partition key isolation ( #34336 ) ( #34613 )
...
issue: #34332
pr: #34336
---------
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit 104d0966b7
2024-07-17 14:03:43 +08:00
cai.zhang
19d3606e0d
fix: [cherry-pick] Fix the bug that caused small segment flush frequently ( #34727 )
...
issue: #34703
master pr: #34725
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-17 09:47:43 +08:00
Buqian Zheng
dfb41582cd
correctly set search params when using knowhere iterator ( #34732 )
...
issue: #34730
pr: #34731
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-07-16 21:17:41 +08:00
sre-ci-robot
cdb4979c29
[automated] Bump milvus version to v2.4.6 ( #34723 )
...
Bump milvus version to v2.4.6
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-07-16 20:05:38 +08:00
XuanYang-cn
edefc3cbb5
enhance: [skip e2e]Enable compaction it test ( #34526 ) ( #34720 )
...
pr: #34526
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-16 18:19:38 +08:00
SimFG
9cf1f03e89
fix: [2.4] the lint-fix error ( #34717 )
...
- issue: #31293
- pr: #34033
- related pr: #34699
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-16 17:37:39 +08:00
wayblink
419a4d22dd
fix: [cherry-pick] fix bug if datacoord restart when l0 compaction saveSegmentMeta ( #34714 )
...
issue: #34681
pr: #34679
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-16 17:27:38 +08:00
XuanYang-cn
5909a62ca6
fix: Fix accidentlly exit MixCompaction task loop ( #34689 )
...
See also: #33431 , #34460
pr: #34688
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-16 16:21:38 +08:00
cai.zhang
0c5aafd2d1
fix:[cherry-pick] Reset flushed row num after pack segment for clustering compaction ( #34704 )
...
issue: #34703
master pr: #34702
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-16 15:57:43 +08:00
SimFG
13f129aa9c
enhance: [2.4] format the sdk code ( #34699 )
...
- issue: #31293
- pr: #34033
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-16 14:15:38 +08:00
SimFG
d607a43ea6
fix: [2.4] metaCache cleanup issue when listPolicy failed ( #34697 )
...
- issue:#34667
- pr: #34449
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-16 14:13:45 +08:00
Xiaofan
66b5cbe27c
fix: CVE by upgrading some dependencies. ( #34693 )
...
fix #34434 and #34456
upgrade otelgrpc to fix CVE
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-07-16 14:05:37 +08:00
elstic
40f6febf9b
test: [cherry-pick] Fix index creation mismatch for sparse vector columns in multivector case ( #34620 )
...
pr: https://github.com/milvus-io/milvus/pull/34618
Signed-off-by: elstic <hao.wang@zilliz.com>
2024-07-16 09:39:38 +08:00
congqixia
274790b187
enhance: [2.4] Print reason when seal segments ( #34575 ) ( #34687 )
...
Cherry-pick from master
pr: #34575
See also #34574
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-15 22:49:39 +08:00
smellthemoon
0fdb288de7
enhance: upsert support autoid( #30342 ) ( #34633 )
...
pr: #30342
issue: #29258
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
2024-07-15 20:53:39 +08:00
wei liu
cf701a9bf0
enhance: Preserve fixed-size memory in delegator node for growing segment ( #34600 )
...
issue: #34595
pr: #34596
When consuming insert data on the delegator node, QueryCoord will move
out some sealed segments to manage its memory usage. After the growing
segment gets flushed, some sealed segments from other workers will be
moved back to the delegator node. To avoid the frequent movement of
segments, we estimate the maximum growing row count and preserve a
fixed-size memory in the delegator node.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-15 20:51:46 +08:00
congqixia
10c04f33c7
enhance: [2.4] Add param item for segmentFlushInterval ( #34629 ) ( #34663 )
...
Cherry-pick from master
pr: #34629
See also #28817
Add paramitem for segment flush interval
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-15 14:31:40 +08:00
