wei liu
a2c19af3ed
enhance: Solve channel unbalance on datanode ( #34984 ) ( #35033 )
...
issue: #33583
pr: #34984
the old policy permit datanode has at most 2 more channels than other
datanode. so if milvus has 2 datanode and 2 channels, both 2 channels
will be assign to 1 datanode, left another datanode empty.
This PR refine the balance policy to solve channel unbalance on datanode
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-29 11:33:50 +08:00
congqixia
d16320705e
enhance: [2.4] Add Segment Level in milvus segment info APIs ( #34763 ) ( #35023 )
...
Cherry-pick from master
pr: #34763
See also #34746
This PR add segment level field in response of
`GetPersistentSegmentInfo` and `GetQuerySegmentInfo`
---------
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-29 10:11:52 +08:00
shaoyue
0417463e90
enhance: Gin & restfulv1 handler use sonic json ( #35018 )
...
/cc @czs007 @congqixia
master pr: #35020
Signed-off-by: haorenfsa <haorenfsa@gmail.com>
2024-07-29 00:39:49 +08:00
zhagnlu
2e9a86ca80
enhance: redefine variable column block size ( #35041 )
...
pr: #35040
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-28 23:49:48 +08:00
congqixia
079276c6ff
fix: [2.4] Unify hook singleton implementation in proxy ( #34888 )
...
Cherry-pick from master
pr: #34887
Related to #34885
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-26 18:07:53 +08:00
wayblink
1e5c71d550
fix: [cherry-pick] fix dropped segment still visible after dropped by L2 single compaction ( #35006 )
...
bug: #35003
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-26 13:47:48 +08:00
congqixia
2a43f43916
fix: [2.4] Remove timeout in datanode watch ctx ( #35011 ) ( #35017 )
...
Cherry-pick from master
pr: #35011
See also #35008
Use tickle timeout logic instead of hardcode context timeout
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-26 11:59:46 +08:00
cai.zhang
9cd6dbcbc9
fix: [cherry-pick] Fix bug for block clustering compaction ( #35021 )
...
issue: #34703
master pr: #35019
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-26 11:33:40 +08:00
zhagnlu
6f0f3034c6
fix: fix query count(*) concurrently ( #35010 )
...
pr: #35007
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-26 11:11:46 +08:00
wei liu
b3bc7f3985
enhance: Limit collection's normal balance speed ( #34810 ) ( #34987 )
...
issue: #34798
pr: #34810
after we remove the task priority on query coord, to avoid load/release
segment blocked by too much balance task, we limit the balance task size
in each round. at same time, we reduce the balance interval to trigger
balance more frequently.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-26 10:13:46 +08:00
zhagnlu
99da94ee32
enhance: decrease variable column block size ( #35012 )
...
#35013
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-25 21:54:08 +08:00
zhagnlu
9eb42bbda0
fix: fix parse plan proto failed for search type ( #34957 )
...
pr: #34944
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-25 20:27:46 +08:00
wayblink
8c87ec0bc5
enhance: [cherry-pick] refine error handle in clustering compaction task ( #34958 )
...
related: #30633 #34792
pr: #34956
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-25 19:25:46 +08:00
Jiquan Long
2822d8729c
feat: record the number of times milvus turns into force-deny-writing states ( #34960 ) ( #34989 )
...
/kind improvement
fix : #34961
pr: #34960
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-07-25 15:43:09 +08:00
Chun Han
6283fd0b46
fix:nil part stats without l2 compaction ( #34977 )
...
related: #34923
pr: https://github.com/milvus-io/milvus/pull/34992
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-25 14:21:46 +08:00
congqixia
0dd9af8f13
fix: [2.4] Check legacy guarantee ts when skipping alloc ts ( #34982 )
...
Cherry-pick from master
pr: #34981
See also #34980
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-25 10:27:47 +08:00
jaime
77ae127a62
fix: check collection health(queryable) fail for releasing collection ( #34948 )
...
issue: #34946
pr: #34947
---------
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-25 10:25:57 +08:00
cai.zhang
12a24c3957
enhance: [cherry-pick] Refine code for GetRecoveryInfo ( #34974 )
...
issue: #34495
master pr: #34973
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-24 22:01:44 +08:00
jaime
597362e00c
fix: memory leak while parsing query plan ( #34932 )
...
issue: #34930
pr: #34931
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-24 14:47:43 +08:00
cai.zhang
74adedf750
enhance: Optimized the GC logic to ensure that memory is released in time ( #34950 )
...
issue: #34703
master pr: #34949
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-24 14:07:43 +08:00
Chun Han
ae1636c2be
fix: refine handling type for segment pruner( #34923 ) ( #34926 )
...
related: #34923
pr: https://github.com/milvus-io/milvus/pull/34925
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-24 12:05:44 +08:00
wayblink
8f3c126129
enhance: [cherry-pick] support l2 single compaction ( #34929 )
...
#34928
pr: #34935
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-24 11:47:50 +08:00
wei liu
c13c48d99a
fix: Failed to unmarshal field stats's bloom filter ( #34922 )
...
pr #34377 introduce this issue, which miss some new changes during the
cherry-pick
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-23 16:45:47 +08:00
wei liu
8c96026722
fix: Segment may bounce between delegator and worker ( #34904 )
...
issue: #34595
pr: #34830
pr#34596 to we add an overloaded factor to segment in delegator, which
cause same segment got different score in delegator and worker. which
may cause segment bounce between delegator and worker.
This PR use average score to compute the delegator overloaded factor, to
avoid segment bounce between delegator and worker.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-23 15:57:49 +08:00
chyezh
0f1bb9c697
enhance: recover stack info when non-cgo thread crash ( #34866 )
...
issue: #34864
pr: #34865
Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-23 14:21:46 +08:00
aoiasd
48063252b1
fix: [Cheryy-Pick] evict paramtable cache failed ( #34772 )
...
relate: https://github.com/milvus-io/milvus/issues/33461
pr: https://github.com/milvus-io/milvus/pull/34771
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-07-23 12:25:44 +08:00
wei liu
ebbccb870c
fix: Avoid segment lack caused by deduplicate segment task ( #34782 ) ( #34903 )
...
issue: #34781
pr: #34782
when balance segment hasn't finished yet, query coord may found 2 loaded
copy of segment, then it will generate task to deduplicate, which may
cancel the balance task. then the old copy has been released, and the
new copy hasn't be ready yet but canceled, then search failed by segment
lack.
this PR set deduplicate segment task's proirity to low, to avoid balance
segment task canceled by deduplicate task.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-23 11:06:15 +08:00
yellow-shine
0951fe4e64
feat: [skip e2e] effective way to use cache ( #34859 )
...
pr: https://github.com/milvus-io/milvus/pull/34774
issue: https://github.com/milvus-io/milvus/issues/34876
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-07-23 10:49:51 +08:00
cai.zhang
6986dfdd5b
enhance:[cherry-pick]Send flush signal when the water level reaches the high watermark ( #34908 )
...
issue: #30633
master pr: #34907
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-23 10:27:42 +08:00
cai.zhang
88da0251af
fix: [cherry-pick] Get current index version from knowhere ( #34902 )
...
issue: #34900
master pr: #34901
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-23 10:25:52 +08:00
jaime
176034d1d0
fix: collection meta is not removed after gc in DataCoord ( #34884 )
...
issue: #34847
pr: #34883
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-22 21:11:48 +08:00
cai.zhang
4ed62e9dbb
enhance: [cherry-pick] Add integration test for clustering compaction ( #34860 )
...
issue: #34792
master pr: #34881
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2024-07-22 17:49:42 +08:00
SimFG
4e95f3835a
fix: [2.4] execute method should return an error when result is a failure ( #34872 )
...
- issue: #34812
- pr: #34813
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-22 12:13:42 +08:00
congqixia
7540714c1b
enhance: [2.4] Add l0 segment entry num quota ( #34733 ) ( #34837 )
...
Cherry-pick from master
pr: #34733
See also #34670
This PR add quota configuration for l0 segment entry number per
collection. If l0 compaction cannot keep up the insertion/upsertion
rate, this feature could back press the related rate.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-22 12:05:43 +08:00
Patrick Weizhi Xu
606bdd7d41
feat: [restful v2][2.4] add partition key isolation prop ( #34869 )
...
issue: https://github.com/milvus-io/milvus/issues/34332
pr: https://github.com/milvus-io/milvus/pull/34710
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit 703feaf6b4a94d1747141cea073b5dcfa36eac73)
2024-07-22 11:43:41 +08:00
yihao.dai
ffee6b9c27
enhance: Optimize ConsistencyLevel and GuaranteeTimestamp log ( #34738 ) ( #34824 )
...
/kind improvement
pr: https://github.com/milvus-io/milvus/pull/34738
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-22 11:32:04 +08:00
wayblink
9b3bd650e6
fix: [cherry-pick] fix collection is compacting logic ( #34862 )
...
related: #34861 #34792
pr: #34855
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-22 11:09:45 +08:00
congqixia
c06a0ebef2
enhance: [2.4] Remove useless ops when there is no write ( #34767 ) ( #34839 )
...
Cherry pick from master
pr: #34767
Related to: #33235
THe querynode pipeline will make map & call ProcessInsert when there is
no write messages. So querynodes will have high CPU usage even when
there is no workload.
This PR check msg length before composing data struct and calling method
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-22 10:23:42 +08:00
wayblink
21973a600d
enhance: [cherry-pick] refine clustering compaction basic it ( #34794 )
...
issue: #34792
pr: #34793
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-21 20:05:41 +08:00
wayblink
c0c3c5f528
enhance: [cherry-pick] refine clustering compaction configs and logs ( #34818 )
...
issue: #30633
pr: #34784
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-21 19:27:41 +08:00
wayblink
33bbc614df
enhance: [cherry-pick] add ut for clustering_compactor ( #34817 )
...
issue: #34792
pr: #34852
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-21 19:25:41 +08:00
congqixia
d1e0d4ea4a
enhance: [2.4] Add dynamic cgo pool for proxy CGO call ( #34768 ) ( #34842 )
...
Cherry-pick from master
pr: #34768
Related to #34705
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-21 18:05:41 +08:00
Min Tian
8834649650
enhance: [2.4] new knowhere param for range_search ( #34709 )
...
Cherry-pick from master
pr: #34686
issue: #34685
Signed-off-by: min.tian <min.tian.cn@gmail.com>
2024-07-19 19:45:41 +08:00
cai.zhang
323dee2fbc
fix: [cherry-pick] Fix the issue of concurrent packing of the same segment ( #34838 )
...
issue: #34703
master pr: #34840
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-19 18:27:40 +08:00
yihao.dai
07bc1b6717
enhance: Seal by total growing segments size ( #34692 ) ( #34779 )
...
Seals the largest growing segment if the total size of growing segments
of each shard exceeds the size threshold(default 4GB). Introducing this
policy can help keep the size of growing segments within a suitable
level, alleviating the pressure on the delegator.
issue: https://github.com/milvus-io/milvus/issues/34554
pr: https://github.com/milvus-io/milvus/pull/34692
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-19 18:25:50 +08:00
wayblink
8632582520
fix: [cherry-pick] fix unstable ut of compaction pickSlot ( #34829 )
...
bug: #34827
pr: #34828
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-19 17:53:48 +08:00
XuanYang-cn
ff4bd2c9d2
enhance: [cp24]Add ut for l0CompactionTask processExecuting ( #34801 )
...
See also: #34796
pr: #34800
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-19 16:31:41 +08:00
Jiquan Long
34873a7f76
feat: record the duration waiting in the proxy queue ( #34744 ) ( #34788 )
...
fix: https://github.com/milvus-io/milvus/issues/34743
pr: #34744
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-07-19 15:37:44 +08:00
foxspy
1f028f3f4b
enhance: [2.4] add scalar filtering and vector search latency metrics ( #34783 )
...
issue: #34780
pr: #34785
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-07-19 11:49:46 +08:00
Chun Han
e11aab3b37
fix: non-init seg_offset for growing raw-data when doing groupby( #34748 ) ( #34750 )
...
related: #34713
pr: https://github.com/milvus-io/milvus/pull/34748
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-19 10:21:44 +08:00
