test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-02 11:59:00 +08:00
Code Issues Actions 12 Packages Projects Releases Wiki Activity
20,468 Commits 94 Branches 105 Tags 184 MiB
58b198c785
Commit Graph

20468 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
cqy123456
58b198c785
enhance: [2.4]support sparse vector mmap in growing segment type (#36565) 
issue: https://github.com/milvus-io/milvus/issues/32984
related pr: https://github.com/milvus-io/milvus/pull/36566

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-10-11 17:23:21 +08:00
XuanYang-cn
e976b41f97
fix: Remove enableLevelZeroSegment config (#36507) 
See also: #36504
pr: #36535

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-10-11 16:41:21 +08:00
XuanYang-cn
28de6b86ba
fix: [24]fail to init fg clears flushTs so that slows flush (#36741) 
See also: #36709
pr: #36740

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-10-11 16:39:21 +08:00
yihao.dai
7ee46c582e
enhance: Support new RESTful URL to describe import job (#36753) (#36754) 
Support the new RESTful URL for retrieving/describing import progress:
`/v2/vectordb/jobs/import/describe`.
Deprecate the old URL: `/v2/vectordb/jobs/import/get_progress`.

issue: https://github.com/milvus-io/milvus/issues/36752

pr: https://github.com/milvus-io/milvus/pull/36753

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-11 16:03:22 +08:00
congqixia
bacbfae542
enhance: Bump milvus & proto version to v2.4.13 (#36758) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-11 16:01:22 +08:00
yihao.dai
a4ef93457d
enhance: Optimize import scheduling and add time cost metric (#36601) (#36684) 
1. Optimize import scheduling strategic:
a. Revise slot weights, calculating them based on the number of files
and segments for both import and pre-import tasks.
b. Ensure that the DN executes tasks in ascending order of task ID.
2. Add time cost metric and log.

issue: https://github.com/milvus-io/milvus/issues/36600,
https://github.com/milvus-io/milvus/issues/36518

pr: https://github.com/milvus-io/milvus/pull/36601

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-11 10:27:22 +08:00
congqixia
b24788b2c7
enhance: [2.4] Add balance report log for qc balancer (#36749) 
Cherry pick from master
pr: #36747 
Related to #36746

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-11 10:25:24 +08:00
SimFG
548f8e80c3
enhance: [2.4] the estimate method when loading the collection (#36728) 
- pr: #36307
- issue: #36530

Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
 2024-10-11 10:20:45 +08:00
XuanYang-cn
c69af44472
enhance: [24]Add more info in logs (#36732) 
pr: #36535

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-10-10 17:51:25 +08:00
yellow-shine
e8c40e94ad
enhance: use new implementation of e2e pipeline (#36687) 
https://github.com/milvus-io/milvus/issues/36348
https://github.com/milvus-io/milvus/pull/36347

Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
 2024-10-10 17:11:21 +08:00
aoiasd
eaa948752b
enhance: [Cherry-Pick] UpdateSegmentsInfo should update remaining segment info even if some one not exist (#36729) 
pr: https://github.com/milvus-io/milvus/pull/36726

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-10-10 15:11:20 +08:00
zhagnlu
02bd916fa4
fix: fix: fix json in [] expr bug (#36722) 
pr: #36721

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-10-09 21:53:20 +08:00
yihao.dai
9cb5396cf6
enhance: Use common gc config (#36668) (#36670) 
Use the GC config from `common` and remove the GC config from
`queryNode`.

issue: https://github.com/milvus-io/milvus/issues/36667

pr: https://github.com/milvus-io/milvus/pull/36668

related pr: https://github.com/milvus-io/milvus/pull/34949

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-09 19:49:20 +08:00
congqixia
3a80d1f602
enhance: [2.4] Add streaming forward policy switch for delegator (#36330) (#36712) 
Cherry pick from master
pr: #36330
Related to #35303

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-09 17:41:20 +08:00
XuanYang-cn
05f96f5298
fix: [24]raise l0 compaction memory ratio to 0.5 (#36691) 
5 percent of free memory is too less for l0 compaction. This pr will
raise it to 50 percent.

See also: #36614
pr: #36690

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-10-09 17:19:24 +08:00
yihao.dai
93a4574a35
Add buildIndex state for import job (#36705) 
issue: https://github.com/milvus-io/milvus/issues/36698

pr: https://github.com/milvus-io/milvus/pull/35868,
https://github.com/milvus-io/milvus/pull/36699

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-09 16:57:20 +08:00
XuanYang-cn
0ff8e13232
fix: [24]Remove neighbors if compactTo is unindexed (#36503) (#36694) 
See also: #36360
pr: #36503

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-10-09 15:43:20 +08:00
Zhen Ye
bcc661dbd0
fix: rocksmq consume too slow if the channel is full (#36618) 
issue: #36569
pr: #36617

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-10-09 11:59:31 +08:00
Patrick Weizhi Xu
6e93601b2c
enhance: [2.4] remove warnings when there is no partition key set (#36679) 
issue: #36673

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit db60f7cf2a56656b101067a2dbebd4dd32ed6ed2)
 2024-10-09 09:35:20 +08:00
cai.zhang
f7302a78fb
enhance: [cherry-pick]Convert unincode to ascii to improving expression parsing efficiency (#36676) 
issue: #36672 

master pr: #36675

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-10-09 09:23:24 +08:00
sre-ci-robot
a081bf394d
[automated] Update Pytest image changes (#36689) 
Update Pytest image changes
See changes:
f10806650a
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2024-10-09 08:51:19 +08:00
congqixia
1955738ab8
enhance: [2.4] Produce messages of multiple topics in parallel (#36344) (#36462) 
Cherry-pick from master
pr: #36344 
Related to #36343

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-08 18:31:20 +08:00
congqixia
9073f6281e
fix: [2.4] Add defer Unpin when error happens (#36620) (#36665) 
Cherry-pick from master
pr: #36620
Resolves: #36619

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-08 18:29:20 +08:00
Buqian Zheng
03fba95284
fix: [2.4] improve mmap related code in Column.h (#36521) 
pr: #36183
issue: #36182

---------

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-10-08 18:27:19 +08:00
nico
f10806650a
test: update pymilvus version and test case (#36590) 
pr: #36589

Signed-off-by: nico <cheng.yuan@zilliz.com>
 2024-10-08 17:25:24 +08:00
jaime
da2d3fb430
enhance: enable manual compaction for collections without indexes (#36581) 
issue: #36576
pr: #36577

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-10-08 14:51:20 +08:00
jaime
bf671786d9
enhance: decrease CPU overhead when calculating index file size (#36580) 
issue: #36578
pr: #36579

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-10-08 14:29:25 +08:00
yihao.dai
86004d852e
enhance: Reopen chunk cache cpp ut (#33622) (#33623) 
issue: https://github.com/milvus-io/milvus/issues/33210

pr: https://github.com/milvus-io/milvus/pull/33622

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-08 11:21:22 +08:00
yihao.dai
29c1d5042a
fix: Fix import segments leak in segment manager (#36602) (#36631) 
Directly add import segments from the meta, eliminating the dependency
on the segment manager.

issue: https://github.com/milvus-io/milvus/issues/34648

pr: https://github.com/milvus-io/milvus/pull/36602

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-08 10:15:22 +08:00
wei liu
2428adea3b
enhance: Enable balance on querynode with different mem capacity (#36466) (#36625) 
issue: #36464
pr: #36466
This PR enable balance on querynode with different mem capacity, for
query node which has more mem capactity will be assigned more records,
and query node with the largest difference between assignedScore and
currentScore will have a higher priority to carry the new segment.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-09-30 18:11:18 +08:00
congqixia
385bfc2639
enhance: [2.4] Unify InboundLabel case using metrics.Label (#36613) (#36616) 
Cherry pick from master
pr: #36613 
Previous label case broken by #36107, this PR make all inbound label
using label constants from metrics package.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-30 16:19:17 +08:00
wei liu
f8dfd0de53
fix: Skip unnecessary query node health check in proxy (#36491) (#36553) 
issue: #36490
pr: #36491
After the query node changes from a delegator to a worker, proxy should
skip this querynode's health check.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-09-30 14:27:16 +08:00
wei liu
4120320074
enhance: make TransferChannel/TransferSegment idempotent (#36489) (#36552) 
issue: #36488
pr: #36489
when call TransferChannel/TransferSegment, querycoord will generate and
submit balance task to scheduler, if segment/channel's task already
exist in scheduler, submit task will failed.

to make TransferChannel/TransferSegment idempotent, we should skip to
submit if task already exist in scheduler.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-09-30 14:25:26 +08:00
yihao.dai
c5088b8527
enhance: Add metrics to monitor import throughput and imported rows (#36519) (#36588) 
issue: https://github.com/milvus-io/milvus/issues/36518

pr: https://github.com/milvus-io/milvus/pull/36519

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-09-30 10:03:18 +08:00
SimFG
58a763c529
enhance: [2.4] avoid to create many timer object in the target (#36573) 
/kind improvement
- pr: #36570

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-09-29 19:27:16 +08:00
zhagnlu
9a07c1bca9
fix:fix term expr overflow bug (#36534) 
pr: #36525

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-09-28 17:43:16 +08:00
cai.zhang
6335c8bf69
fix: [cherry-pick] Record the nodeID before assigning tasks (#36493) 
issue: #33744 

master pr: #36371

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-28 17:29:21 +08:00
cai.zhang
132e4c3ba1
fix: [cherry-pick]Fix data race for cluerting compaction (#36499) 
issue: #36438 

master pr: #36440

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-28 17:19:21 +08:00
aoiasd
2231aeab4d
fix:[Cherry-Pick] Split delete task msg to MaxMessageSize (#36574) 
relate: https://github.com/milvus-io/milvus/issues/36089
pr: https://github.com/milvus-io/milvus/pull/36197
split delete task msg to MaxMessageSize to avoid mq message too large
error

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-09-27 18:15:19 +08:00
cai.zhang
e5a6c5b31d
fix: [cherry-pick]Check string array max length after type matching (#36497) 
issue: #36029 

master pr: #36449

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-27 11:15:14 +08:00
Zhen Ye
e34fa0461b
fix: port listen racing in mix or standalone mode (#36459) 
issue: #36441
pr: #36442

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-09-26 21:01:15 +08:00
wei liu
74af00ba8c
fix: Segment unbalance after many times load/release (#36537) (#36543) 
issue: #36536
pr: #36537
query coord use `segmentTaskDeleta/channelTaskDelta` to measure the
executing workload for querynode in scheduler, and we maintains the
`segmentTaskDeleta/channelTaskDelta` by `scheulder.Add(task)` and
`scheduler.remove(task)`, but `scheduler.remove(task)` has been called
in unexpected way, which cause a wrong
`segmentTaskDeleta/channelTaskDelta` value and affect the segment assign
logic, causes segment unbalance.

This PR moves to compute the `segmentTaskDeleta/channelTaskDelta` when
access, to avoid the wrong value affect.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-09-26 20:57:14 +08:00
SimFG
07e1bc8c08
enhance: [2.4] get msg type from the msg header to reduce the Unmarshal usage (#36454) 
/kind improvement
- pr: #36409

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-09-26 16:41:15 +08:00
SimFG
6604bbda8f
enhance: [2.4] update the expr version and format the expr http response (#36467) 
/kind improvement
- pr: #36406

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-09-26 14:27:20 +08:00
wei liu
ad5d24be65
enhance: Optimize workload based replica selection policy (#36181) (#36384) 
issue: #35859
pr: #36181

This PR introduce two new param: toleranceFactor and checkRequestNum,
after every checkRequestNum request has been assigned, try to compute
querynode's workload score.

if the diff is less than the toleranceFactor, replica selection policy
will fallback to round_robin, which reduce the average cost to about
500ns.

if the diff is larger than the toleranceFactor, replica selection policy
will compute querynode's score to select the target node with smallest
score in every assigment.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-09-26 11:19:14 +08:00
wei liu
975a9797a2
enhance: Enable dynamic update loaded collection's replica (#36417) 
issue: #35821
pr: #35822
After collection loaded, if we need to increase/decrease collection's
replica, we need to release and load it again.

milvus offers 4 solution to update loaded collection's replica, this PR
aims to dynamic change the replica number without release, and after
replica number changed, milvus will execute load replica or release
replica in async, and the replica loaded status can be checked by
getReplicas API.

Notice that if set too much replicas than querynode can afford，the new
replica won't be loaded successfully until enough querynode joins.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-09-26 10:43:15 +08:00
sre-ci-robot
720f591c04
[automated] Bump milvus version to v2.4.12 (#36494) 
Bump milvus version to v2.4.12
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2024-09-25 14:55:14 +08:00
wei liu
bdc59f3b15
fix: Fix cornor case that segment can't be move out from stopping node (#36431) (#36475) 
issue: #36426
pr: #36431
the old constriant requires only segment on current target can be
balanced, which is wrong, and caused that segment can't be move out from
stopping node, if it's only exist in next target.

by design, stopping balance need to move out all segment on it by
balance task, thus the unfair old constriant should be removed.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-09-25 10:13:18 +08:00
SimFG
a00523f0fd
fix: metric type error when the collection has two vec field (#36473) 
- issue: #36395

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-09-24 20:03:14 +08:00
congqixia
fa6354f6df
enhance: [skip e2e][2.4] Add unittest for reducing duplicated pk from multi segments (#36433) (#36460) 
Cherry-pick from master
pr: #36433
Related to #35505 #36362

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-24 18:09:14 +08:00