Min Tian
8834649650
enhance: [2.4] new knowhere param for range_search ( #34709 )
...
Cherry-pick from master
pr: #34686
issue: #34685
Signed-off-by: min.tian <min.tian.cn@gmail.com>
2024-07-19 19:45:41 +08:00
cai.zhang
323dee2fbc
fix: [cherry-pick] Fix the issue of concurrent packing of the same segment ( #34838 )
...
issue: #34703
master pr: #34840
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-19 18:27:40 +08:00
yihao.dai
07bc1b6717
enhance: Seal by total growing segments size ( #34692 ) ( #34779 )
...
Seals the largest growing segment if the total size of growing segments
of each shard exceeds the size threshold(default 4GB). Introducing this
policy can help keep the size of growing segments within a suitable
level, alleviating the pressure on the delegator.
issue: https://github.com/milvus-io/milvus/issues/34554
pr: https://github.com/milvus-io/milvus/pull/34692
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-19 18:25:50 +08:00
wayblink
8632582520
fix: [cherry-pick] fix unstable ut of compaction pickSlot ( #34829 )
...
bug: #34827
pr: #34828
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-19 17:53:48 +08:00
XuanYang-cn
ff4bd2c9d2
enhance: [cp24]Add ut for l0CompactionTask processExecuting ( #34801 )
...
See also: #34796
pr: #34800
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-19 16:31:41 +08:00
Jiquan Long
34873a7f76
feat: record the duration waiting in the proxy queue ( #34744 ) ( #34788 )
...
fix: https://github.com/milvus-io/milvus/issues/34743
pr: #34744
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-07-19 15:37:44 +08:00
foxspy
1f028f3f4b
enhance: [2.4] add scalar filtering and vector search latency metrics ( #34783 )
...
issue: #34780
pr: #34785
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-07-19 11:49:46 +08:00
Chun Han
e11aab3b37
fix: non-init seg_offset for growing raw-data when doing groupby( #34748 ) ( #34750 )
...
related: #34713
pr: https://github.com/milvus-io/milvus/pull/34748
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-19 10:21:44 +08:00
SimFG
0e226502e4
enhance: [2.4] pick default root password and log level pr ( #34777 )
...
default root password
- issue: #33058
- pr: #34752
set log level
- issue: #34756
- pr: #34757
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-18 13:45:43 +08:00
wayblink
a26e965e6a
enhance:[cherry-pick] Add compaction task slot usage logic ( #34625 )
...
issue: #34544
pr: #34581
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-18 09:55:43 +08:00
cai.zhang
d74a5bdc1d
fix: [cherry-pick] Fix bug where binlogs already flushed with new segment during pack ( #34760 )
...
issue: #34703
master pr: #34762
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-18 09:15:47 +08:00
wayblink
83fc26c31a
fix: [cherry-pick] compaction task not be cleaned correctly ( #34766 )
...
1.fix compaction task not be cleaned correctly
2.add a new parameter to control compaction gc loop interval
3.remove some useless configs of clustering compaction
bug: #34764
pr: #34765
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-17 22:17:42 +08:00
Patrick Weizhi Xu
3cf2e1831a
feat: [cherry-pick][2.4] support partition key isolation ( #34336 ) ( #34613 )
...
issue: #34332
pr: #34336
---------
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit 104d0966b7
2024-07-17 14:03:43 +08:00
cai.zhang
19d3606e0d
fix: [cherry-pick] Fix the bug that caused small segment flush frequently ( #34727 )
...
issue: #34703
master pr: #34725
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-17 09:47:43 +08:00
Buqian Zheng
dfb41582cd
correctly set search params when using knowhere iterator ( #34732 )
...
issue: #34730
pr: #34731
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-07-16 21:17:41 +08:00
sre-ci-robot
cdb4979c29
[automated] Bump milvus version to v2.4.6 ( #34723 )
...
Bump milvus version to v2.4.6
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-07-16 20:05:38 +08:00
XuanYang-cn
edefc3cbb5
enhance: [skip e2e]Enable compaction it test ( #34526 ) ( #34720 )
...
pr: #34526
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-16 18:19:38 +08:00
SimFG
9cf1f03e89
fix: [2.4] the lint-fix error ( #34717 )
...
- issue: #31293
- pr: #34033
- related pr: #34699
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-16 17:37:39 +08:00
wayblink
419a4d22dd
fix: [cherry-pick] fix bug if datacoord restart when l0 compaction saveSegmentMeta ( #34714 )
...
issue: #34681
pr: #34679
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-16 17:27:38 +08:00
XuanYang-cn
5909a62ca6
fix: Fix accidentlly exit MixCompaction task loop ( #34689 )
...
See also: #33431 , #34460
pr: #34688
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-16 16:21:38 +08:00
cai.zhang
0c5aafd2d1
fix:[cherry-pick] Reset flushed row num after pack segment for clustering compaction ( #34704 )
...
issue: #34703
master pr: #34702
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-16 15:57:43 +08:00
SimFG
13f129aa9c
enhance: [2.4] format the sdk code ( #34699 )
...
- issue: #31293
- pr: #34033
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-16 14:15:38 +08:00
SimFG
d607a43ea6
fix: [2.4] metaCache cleanup issue when listPolicy failed ( #34697 )
...
- issue:#34667
- pr: #34449
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-16 14:13:45 +08:00
Xiaofan
66b5cbe27c
fix: CVE by upgrading some dependencies. ( #34693 )
...
fix #34434 and #34456
upgrade otelgrpc to fix CVE
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-07-16 14:05:37 +08:00
elstic
40f6febf9b
test: [cherry-pick] Fix index creation mismatch for sparse vector columns in multivector case ( #34620 )
...
pr: https://github.com/milvus-io/milvus/pull/34618
Signed-off-by: elstic <hao.wang@zilliz.com>
2024-07-16 09:39:38 +08:00
congqixia
274790b187
enhance: [2.4] Print reason when seal segments ( #34575 ) ( #34687 )
...
Cherry-pick from master
pr: #34575
See also #34574
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-15 22:49:39 +08:00
smellthemoon
0fdb288de7
enhance: upsert support autoid( #30342 ) ( #34633 )
...
pr: #30342
issue: #29258
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
2024-07-15 20:53:39 +08:00
wei liu
cf701a9bf0
enhance: Preserve fixed-size memory in delegator node for growing segment ( #34600 )
...
issue: #34595
pr: #34596
When consuming insert data on the delegator node, QueryCoord will move
out some sealed segments to manage its memory usage. After the growing
segment gets flushed, some sealed segments from other workers will be
moved back to the delegator node. To avoid the frequent movement of
segments, we estimate the maximum growing row count and preserve a
fixed-size memory in the delegator node.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-15 20:51:46 +08:00
congqixia
10c04f33c7
enhance: [2.4] Add param item for segmentFlushInterval ( #34629 ) ( #34663 )
...
Cherry-pick from master
pr: #34629
See also #28817
Add paramitem for segment flush interval
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-15 14:31:40 +08:00
wayblink
92afcb7a39
fix: [cherry-pick] fix meta revert bug when clustering compaction task fail ( #34606 )
...
issue: #34603
pr: #34604
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-14 22:23:39 +08:00
zhagnlu
1cf0c67af6
fix: fix auto merge error ( #34662 )
...
pr: #34661
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-13 10:39:38 +08:00
cai.zhang
e1686d096f
fix: [cherry-pick] Remove flushed segment in segment manager generated through import ( #34651 )
...
issue: #34648
master pr: #34649
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-12 23:15:38 +08:00
congqixia
6a3a14affb
enhance: [2.4] Add lint rule to forbid gogo protobuf ( #34594 ) ( #34630 )
...
Cherry pick from master
pr: #34594
github.com/gogo/protobuf is deprecated and could be error prune after
upgrade protobuf message to v2.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-12 18:13:36 +08:00
zhagnlu
91952e860a
fix: fix bug that set incorrect info to columnbase ( #34621 )
...
pr: #34428
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-12 17:07:36 +08:00
SimFG
9bcab2ebee
enhance: [2.4] use the insert rate limit for the upsert request ( #34616 )
...
- issue: #34614
- pr: #34615
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-12 16:19:36 +08:00
zhagnlu
4e02e57044
enhance: mark duplicated pk as deleted ( #34619 )
...
pr: #34586
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-12 10:27:37 +08:00
wayblink
86b57b7827
enhance: [cherry-pick] Alter compactTo segments before compactFrom to avoid data loss if crash ( #34532 )
...
issue: #34512
pr: #34513
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-12 00:47:37 +08:00
jaime
1d2062a68a
fix: encountering orphan channel-cp meta after GC ( #34609 )
...
issue: #34545
pr: #34612
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-11 22:51:37 +08:00
XuanYang-cn
be41892c48
fix: Compaction task l0 state transfer wrong ( #34599 )
...
This bug caused failed L0 compaction tasks never end.
See also: #34460
pr: #34597
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-11 20:21:37 +08:00
wayblink
9dc98ac76d
enhance: [cherry-pick] Refine clustering compaction latency metrics ( #34539 )
...
issue: #34527
pr: #34528
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-11 17:43:43 +08:00
edward.zeng
3d29ebfffb
enhance: Refine default histogram metric bucket ( #34592 )
...
/kind improvement
pr: https://github.com/milvus-io/milvus/pull/34580
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
2024-07-11 16:51:35 +08:00
XuanYang-cn
7e2a9d6bca
fix: Milvus panic when compaction disabled and dropping a collection ( #34206 )
...
See also: #31059
pr: #34103
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-11 14:20:52 +08:00
aoiasd
d6fc6a9f41
fix: [Cherry-pick] Restful API use deprecate error code cause access log panic. ( #34579 )
...
Avoid panic when use deprecate error code and use merr code replace
deprecate common code.
relate: https://github.com/milvus-io/milvus/issues/34578
pr： https://github.com/milvus-io/milvus/pull/34576
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-07-11 11:58:51 +08:00
jaime
bfd386aad7
fix: unstable UT for level0 deletion ( #34525 )
...
issue: #34533
pr: #34524
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-11 10:02:56 +08:00
jaime
073a501fc7
enhance: skip orphan channel cp meta when checking cp lag ( #34556 )
...
issue: #34545
pr: #34555
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-11 09:36:56 +08:00
XuanYang-cn
2bef9c38c6
enhance: Add back compactionTaskNum metrics ( #34534 )
...
Fix L0 compaction task recover unable to set segment
not isCompacting
See also: milvus-io#34460
pr: milvus-io#34583
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-11 00:08:52 +08:00
yihao.dai
cbba5481f8
fix: Prevent dispatcher merging if curTs is 0 ( #34562 ) ( #34563 )
...
When the main dispatcher has not yet consumed data, curTs is 0. During
this time, merging dispatchers should not be allowed; otherwise, the
data of the solo dispatcher will be skipped.
issue: https://github.com/milvus-io/milvus/issues/34255
pr: https://github.com/milvus-io/milvus/pull/34562
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-10 22:46:51 +08:00
SimFG
15adb2feac
enhance: [2.4] add the seal segment when dispatch delete msgs ( #34566 )
...
/kind improvement
- pr: #34565
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-10 19:18:51 +08:00
wei liu
a7190b042d
fix: Indexnode stuck at stopping progress cause by wrong lifetime control ( #34559 )
...
issue: #34557
pr: #34558
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-10 15:14:13 +08:00
yihao.dai
1589d68413
fix: Fix no L0 compaction tasks are executed ( #34537 ) ( #34538 )
...
The nodeID for compaction task initialization is 0. This PR adjusts the
task reassignment conditions to allow new compaction tasks to be
reassigned and executed.
issue: https://github.com/milvus-io/milvus/issues/34460
pr: https://github.com/milvus-io/milvus/pull/34537
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-10 11:08:12 +08:00
