test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-11 09:46:26 +08:00
Code Issues Actions 16 Packages Projects Releases Wiki Activity
20,149 Commits 97 Branches 105 Tags 189 MiB
74adedf750
Commit Graph

20149 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
cai.zhang
74adedf750
enhance: Optimized the GC logic to ensure that memory is released in time (#34950) 
issue: #34703 

master pr: #34949

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-24 14:07:43 +08:00
Chun Han
ae1636c2be
fix: refine handling type for segment pruner(#34923) (#34926) 
related: #34923
pr: https://github.com/milvus-io/milvus/pull/34925

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-07-24 12:05:44 +08:00
wayblink
8f3c126129
enhance: [cherry-pick] support l2 single compaction (#34929) 
#34928
pr: #34935

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-24 11:47:50 +08:00
wei liu
c13c48d99a
fix: Failed to unmarshal field stats's bloom filter (#34922) 
pr #34377 introduce this issue, which miss some new changes during the
cherry-pick

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-23 16:45:47 +08:00
wei liu
8c96026722
fix: Segment may bounce between delegator and worker (#34904) 
issue: #34595
pr: #34830

pr#34596 to we add an overloaded factor to segment in delegator, which
cause same segment got different score in delegator and worker. which
may cause segment bounce between delegator and worker.

This PR use average score to compute the delegator overloaded factor, to
avoid segment bounce between delegator and worker.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-23 15:57:49 +08:00
chyezh
0f1bb9c697
enhance: recover stack info when non-cgo thread crash (#34866) 
issue: #34864
pr: #34865

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-07-23 14:21:46 +08:00
aoiasd
48063252b1
fix: [Cheryy-Pick] evict paramtable cache failed (#34772) 
relate: https://github.com/milvus-io/milvus/issues/33461
pr: https://github.com/milvus-io/milvus/pull/34771

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-07-23 12:25:44 +08:00
wei liu
ebbccb870c
fix: Avoid segment lack caused by deduplicate segment task (#34782) (#34903) 
issue: #34781
pr: #34782

when balance segment hasn't finished yet, query coord may found 2 loaded
copy of segment, then it will generate task to deduplicate, which may
cancel the balance task. then the old copy has been released, and the
new copy hasn't be ready yet but canceled, then search failed by segment
lack.

this PR set deduplicate segment task's proirity to low, to avoid balance
segment task canceled by deduplicate task.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-23 11:06:15 +08:00
yellow-shine
0951fe4e64
feat: [skip e2e] effective way to use cache (#34859) 
pr: https://github.com/milvus-io/milvus/pull/34774
issue: https://github.com/milvus-io/milvus/issues/34876

Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
 2024-07-23 10:49:51 +08:00
cai.zhang
6986dfdd5b
enhance:[cherry-pick]Send flush signal when the water level reaches the high watermark (#34908) 
issue: #30633 

master pr: #34907

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-23 10:27:42 +08:00
cai.zhang
88da0251af
fix: [cherry-pick] Get current index version from knowhere (#34902) 
issue: #34900 

master pr: #34901

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-23 10:25:52 +08:00
jaime
176034d1d0
fix: collection meta is not removed after gc in DataCoord (#34884) 
issue: #34847
pr: #34883

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-22 21:11:48 +08:00
cai.zhang
4ed62e9dbb
enhance: [cherry-pick] Add integration test for clustering compaction (#34860) 
issue: #34792 

master pr: #34881

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2024-07-22 17:49:42 +08:00
SimFG
4e95f3835a
fix: [2.4] execute method should return an error when result is a failure (#34872) 
- issue: #34812
- pr: #34813

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-07-22 12:13:42 +08:00
congqixia
7540714c1b
enhance: [2.4] Add l0 segment entry num quota (#34733) (#34837) 
Cherry-pick from master
pr: #34733
See also #34670

This PR add quota configuration for l0 segment entry number per
collection. If l0 compaction cannot keep up the insertion/upsertion
rate, this feature could back press the related rate.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-22 12:05:43 +08:00
Patrick Weizhi Xu
606bdd7d41
feat: [restful v2][2.4] add partition key isolation prop (#34869) 
issue: https://github.com/milvus-io/milvus/issues/34332
pr: https://github.com/milvus-io/milvus/pull/34710

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit 703feaf6b4a94d1747141cea073b5dcfa36eac73)
 2024-07-22 11:43:41 +08:00
yihao.dai
ffee6b9c27
enhance: Optimize ConsistencyLevel and GuaranteeTimestamp log (#34738) (#34824) 
/kind improvement

pr: https://github.com/milvus-io/milvus/pull/34738

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-22 11:32:04 +08:00
wayblink
9b3bd650e6
fix: [cherry-pick] fix collection is compacting logic (#34862) 
related: #34861 #34792
pr: #34855

---------

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-22 11:09:45 +08:00
congqixia
c06a0ebef2
enhance: [2.4] Remove useless ops when there is no write (#34767) (#34839) 
Cherry pick from master
pr: #34767
Related to: #33235

THe querynode pipeline will make map & call ProcessInsert when there is
no write messages. So querynodes will have high CPU usage even when
there is no workload.

This PR check msg length before composing data struct and calling method

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-22 10:23:42 +08:00
wayblink
21973a600d
enhance: [cherry-pick] refine clustering compaction basic it (#34794) 
issue: #34792
pr: #34793

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-21 20:05:41 +08:00
wayblink
c0c3c5f528
enhance: [cherry-pick] refine clustering compaction configs and logs (#34818) 
issue: #30633
pr: #34784

---------

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-21 19:27:41 +08:00
wayblink
33bbc614df
enhance: [cherry-pick] add ut for clustering_compactor (#34817) 
issue: #34792
pr: #34852

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-21 19:25:41 +08:00
congqixia
d1e0d4ea4a
enhance: [2.4] Add dynamic cgo pool for proxy CGO call (#34768) (#34842) 
Cherry-pick from master
pr: #34768
Related to #34705

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-21 18:05:41 +08:00
Min Tian
8834649650
enhance: [2.4] new knowhere param for range_search (#34709) 
Cherry-pick from master
pr: #34686
issue: #34685

Signed-off-by: min.tian <min.tian.cn@gmail.com>
 2024-07-19 19:45:41 +08:00
cai.zhang
323dee2fbc
fix: [cherry-pick] Fix the issue of concurrent packing of the same segment (#34838) 
issue: #34703 

master pr: #34840

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-19 18:27:40 +08:00
yihao.dai
07bc1b6717
enhance: Seal by total growing segments size (#34692) (#34779) 
Seals the largest growing segment if the total size of growing segments
of each shard exceeds the size threshold(default 4GB). Introducing this
policy can help keep the size of growing segments within a suitable
level, alleviating the pressure on the delegator.

issue: https://github.com/milvus-io/milvus/issues/34554

pr: https://github.com/milvus-io/milvus/pull/34692

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-19 18:25:50 +08:00
wayblink
8632582520
fix: [cherry-pick] fix unstable ut of compaction pickSlot (#34829) 
bug: #34827
pr: #34828

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-19 17:53:48 +08:00
XuanYang-cn
ff4bd2c9d2
enhance: [cp24]Add ut for l0CompactionTask processExecuting (#34801) 
See also: #34796
pr: #34800

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-07-19 16:31:41 +08:00
Jiquan Long
34873a7f76
feat: record the duration waiting in the proxy queue (#34744) (#34788) 
fix: https://github.com/milvus-io/milvus/issues/34743
pr: #34744

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-07-19 15:37:44 +08:00
foxspy
1f028f3f4b
enhance: [2.4] add scalar filtering and vector search latency metrics (#34783) 
issue: #34780 
pr: #34785

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
 2024-07-19 11:49:46 +08:00
Chun Han
e11aab3b37
fix: non-init seg_offset for growing raw-data when doing groupby(#34748) (#34750) 
related: #34713
pr: https://github.com/milvus-io/milvus/pull/34748

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-07-19 10:21:44 +08:00
SimFG
0e226502e4
enhance: [2.4] pick default root password and log level pr (#34777) 
default root password
- issue: #33058
- pr: #34752

set log level
- issue: #34756
- pr: #34757

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-07-18 13:45:43 +08:00
wayblink
a26e965e6a
enhance:[cherry-pick] Add compaction task slot usage logic (#34625) 
issue: #34544
pr: #34581

---------

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-18 09:55:43 +08:00
cai.zhang
d74a5bdc1d
fix: [cherry-pick] Fix bug where binlogs already flushed with new segment during pack (#34760) 
issue: #34703

master pr: #34762

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-18 09:15:47 +08:00
wayblink
83fc26c31a
fix: [cherry-pick] compaction task not be cleaned correctly (#34766) 
1.fix compaction task not be cleaned correctly
2.add a new parameter to control compaction gc loop interval
3.remove some useless configs of clustering compaction

bug: #34764
pr: #34765

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-17 22:17:42 +08:00
Patrick Weizhi Xu
3cf2e1831a
feat: [cherry-pick][2.4] support partition key isolation (#34336) (#34613) 
issue: #34332
pr: #34336

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit 104d0966b7)
 2024-07-17 14:03:43 +08:00
cai.zhang
19d3606e0d
fix: [cherry-pick] Fix the bug that caused small segment flush frequently (#34727) 
issue: #34703 

master pr: #34725

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-17 09:47:43 +08:00
Buqian Zheng
dfb41582cd
correctly set search params when using knowhere iterator (#34732) 
issue: #34730
pr: #34731

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-07-16 21:17:41 +08:00
sre-ci-robot
cdb4979c29
[automated] Bump milvus version to v2.4.6 (#34723) 
Bump milvus version to v2.4.6
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2024-07-16 20:05:38 +08:00
XuanYang-cn
edefc3cbb5
enhance: [skip e2e]Enable compaction it test (#34526) (#34720) 
pr: #34526

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-07-16 18:19:38 +08:00
SimFG
9cf1f03e89
fix: [2.4] the lint-fix error (#34717) 
- issue: #31293
- pr: #34033
- related pr: #34699

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-07-16 17:37:39 +08:00
wayblink
419a4d22dd
fix: [cherry-pick] fix bug if datacoord restart when l0 compaction saveSegmentMeta (#34714) 
issue: #34681
pr: #34679

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-16 17:27:38 +08:00
XuanYang-cn
5909a62ca6
fix: Fix accidentlly exit MixCompaction task loop (#34689) 
See also: #33431, #34460
pr: #34688

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-07-16 16:21:38 +08:00
cai.zhang
0c5aafd2d1
fix:[cherry-pick] Reset flushed row num after pack segment for clustering compaction (#34704) 
issue: #34703 
master pr: #34702

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-16 15:57:43 +08:00
SimFG
13f129aa9c
enhance: [2.4] format the sdk code (#34699) 
- issue: #31293
- pr: #34033

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-07-16 14:15:38 +08:00
SimFG
d607a43ea6
fix: [2.4] metaCache cleanup issue when listPolicy failed (#34697) 
- issue:#34667
- pr: #34449

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-07-16 14:13:45 +08:00
Xiaofan
66b5cbe27c
fix: CVE by upgrading some dependencies. (#34693) 
fix #34434 and #34456
upgrade otelgrpc to fix CVE

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
 2024-07-16 14:05:37 +08:00
elstic
40f6febf9b
test: [cherry-pick] Fix index creation mismatch for sparse vector columns in multivector case (#34620) 
pr: https://github.com/milvus-io/milvus/pull/34618

Signed-off-by: elstic <hao.wang@zilliz.com>
 2024-07-16 09:39:38 +08:00
congqixia
274790b187
enhance: [2.4] Print reason when seal segments (#34575) (#34687) 
Cherry-pick from master
pr: #34575
See also #34574

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-15 22:49:39 +08:00
smellthemoon
0fdb288de7
enhance: upsert support autoid(#30342) (#34633) 
pr: #30342 
issue: #29258

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
 2024-07-15 20:53:39 +08:00