test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-04 04:49:08 +08:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity
20,468 Commits 96 Branches 105 Tags 185 MiB
58b198c785
Commit Graph

832 Commits

Author SHA1 Message Date
XuanYang-cn
e976b41f97
fix: Remove enableLevelZeroSegment config (#36507) 
See also: #36504
pr: #36535

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-10-11 16:41:21 +08:00
congqixia
bacbfae542
enhance: Bump milvus & proto version to v2.4.13 (#36758) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-11 16:01:22 +08:00
yihao.dai
a4ef93457d
enhance: Optimize import scheduling and add time cost metric (#36601) (#36684) 
1. Optimize import scheduling strategic:
a. Revise slot weights, calculating them based on the number of files
and segments for both import and pre-import tasks.
b. Ensure that the DN executes tasks in ascending order of task ID.
2. Add time cost metric and log.

issue: https://github.com/milvus-io/milvus/issues/36600,
https://github.com/milvus-io/milvus/issues/36518

pr: https://github.com/milvus-io/milvus/pull/36601

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-11 10:27:22 +08:00
SimFG
548f8e80c3
enhance: [2.4] the estimate method when loading the collection (#36728) 
- pr: #36307
- issue: #36530

Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
 2024-10-11 10:20:45 +08:00
yihao.dai
9cb5396cf6
enhance: Use common gc config (#36668) (#36670) 
Use the GC config from `common` and remove the GC config from
`queryNode`.

issue: https://github.com/milvus-io/milvus/issues/36667

pr: https://github.com/milvus-io/milvus/pull/36668

related pr: https://github.com/milvus-io/milvus/pull/34949

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-09 19:49:20 +08:00
congqixia
3a80d1f602
enhance: [2.4] Add streaming forward policy switch for delegator (#36330) (#36712) 
Cherry pick from master
pr: #36330
Related to #35303

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-09 17:41:20 +08:00
XuanYang-cn
05f96f5298
fix: [24]raise l0 compaction memory ratio to 0.5 (#36691) 
5 percent of free memory is too less for l0 compaction. This pr will
raise it to 50 percent.

See also: #36614
pr: #36690

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-10-09 17:19:24 +08:00
Zhen Ye
bcc661dbd0
fix: rocksmq consume too slow if the channel is full (#36618) 
issue: #36569
pr: #36617

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-10-09 11:59:31 +08:00
congqixia
1955738ab8
enhance: [2.4] Produce messages of multiple topics in parallel (#36344) (#36462) 
Cherry-pick from master
pr: #36344 
Related to #36343

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-08 18:31:20 +08:00
yihao.dai
c5088b8527
enhance: Add metrics to monitor import throughput and imported rows (#36519) (#36588) 
issue: https://github.com/milvus-io/milvus/issues/36518

pr: https://github.com/milvus-io/milvus/pull/36519

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-09-30 10:03:18 +08:00
SimFG
58a763c529
enhance: [2.4] avoid to create many timer object in the target (#36573) 
/kind improvement
- pr: #36570

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-09-29 19:27:16 +08:00
aoiasd
2231aeab4d
fix:[Cherry-Pick] Split delete task msg to MaxMessageSize (#36574) 
relate: https://github.com/milvus-io/milvus/issues/36089
pr: https://github.com/milvus-io/milvus/pull/36197
split delete task msg to MaxMessageSize to avoid mq message too large
error

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-09-27 18:15:19 +08:00
Zhen Ye
e34fa0461b
fix: port listen racing in mix or standalone mode (#36459) 
issue: #36441
pr: #36442

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-09-26 21:01:15 +08:00
SimFG
07e1bc8c08
enhance: [2.4] get msg type from the msg header to reduce the Unmarshal usage (#36454) 
/kind improvement
- pr: #36409

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-09-26 16:41:15 +08:00
SimFG
6604bbda8f
enhance: [2.4] update the expr version and format the expr http response (#36467) 
/kind improvement
- pr: #36406

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-09-26 14:27:20 +08:00
wei liu
ad5d24be65
enhance: Optimize workload based replica selection policy (#36181) (#36384) 
issue: #35859
pr: #36181

This PR introduce two new param: toleranceFactor and checkRequestNum,
after every checkRequestNum request has been assigned, try to compute
querynode's workload score.

if the diff is less than the toleranceFactor, replica selection policy
will fallback to round_robin, which reduce the average cost to about
500ns.

if the diff is larger than the toleranceFactor, replica selection policy
will compute querynode's score to select the target node with smallest
score in every assigment.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-09-26 11:19:14 +08:00
wei liu
975a9797a2
enhance: Enable dynamic update loaded collection's replica (#36417) 
issue: #35821
pr: #35822
After collection loaded, if we need to increase/decrease collection's
replica, we need to release and load it again.

milvus offers 4 solution to update loaded collection's replica, this PR
aims to dynamic change the replica number without release, and after
replica number changed, milvus will execute load replica or release
replica in async, and the replica loaded status can be checked by
getReplicas API.

Notice that if set too much replicas than querynode can afford，the new
replica won't be loaded successfully until enough querynode joins.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-09-26 10:43:15 +08:00
jaime
b92daa1532
fix: iaccurate size estimation for encoded array data (#36379) 
issue: #36029
pr: #36373

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-09-23 21:17:13 +08:00
SimFG
a35d99eabf
fix: [2.4] long buffering causes mq to be unable to receive messages. (#36425) 
- issue: #36397
- pr: #36420

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-09-23 16:33:17 +08:00
congqixia
2b796b180b
enhance: Bump milvus & proto version to v2.4.12 (#36376) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-23 14:51:13 +08:00
cai.zhang
eb47150f66
enhance: [cherry-pick]Disallow the keywords as a field name or dynamic field name (#36108) 
issue: #35873

master pr: #36101

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-15 15:19:14 +08:00
XuanYang-cn
c1dab50fed
enhance: [cp]Add metrics for Delete entries num of L0seg (#36227) 
- Add metrics *DataCoordL0DeleteEntriesNum*
- Remove metrics *DataCoordRateStoredL0Segment*

See also: #36147
pr: #36175

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-09-14 10:37:08 +08:00
congqixia
13d443eb2e
enhance: [2.4] Add L0 forward policy to support remote load (#36189) (#36208) 
Cherry-pick from master
pr: #36189
Related to #35303

This PR add a param item to support change l0 forward behavior from bf
filtering and forward to remote load.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-12 19:09:08 +08:00
Buqian Zheng
089790a459
enhance: [2.4]Allow empty sparse row (#36061) 
pr: #34700

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-09-12 10:13:09 +08:00
congqixia
1cd8d1bd80
enhance: [2.4] Use stats Handler to record request/response size metrics (#36107) (#36118) 
Cherry-pick from master
pr: #36107 
Related to #36102

This PR use newly added `grpcSizeStatsHandler` to reduce calling
`proto.Size` since the request & response size info is recorded by grpc
framework.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-10 17:13:08 +08:00
XuanYang-cn
835c9d5c65
fix: Change l0SegmentsRowCount limits to a reasonable value (#36015) 
pr: #36014
See also: #36028

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-09-08 16:55:05 +08:00
congqixia
9d0378ae84
enhance: Bump milvus & proto version to v2.4.11 (#36069) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-06 23:39:06 +08:00
congqixia
55b33cd3cf
fix: [2.4] Fix tracing config update logic (#35928) (#35998) 
Cherry-pick from master
pr: #35928 
Related to #35927

There are serveral issue this PR addresses:
- Use `ResetTraceConfig` method instead init one in update event handler
- Implement dynamic stats.Handler to receive tracing config update event
- Update `enable_trace` flag when `ResetTraceConfig` is invoked
- Change `enable_trace` to `std::atomic<bool>` in case of data race

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-06 11:19:05 +08:00
Ted Xu
45b2049d5d
fix: fallback params may be overridden (#35972) (#36006) 
See #35756

---------

pr: #35972

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-09-05 19:05:05 +08:00
wei liu
ceca666e2a
fix: Fix privilege group hasn't been register for validate (#35938) 
issue: #35471
pr: #35937

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-09-05 18:09:11 +08:00
congqixia
da0bc22a5f
enhance: [2.4] Add delete buffer related quota logic (#35918) (#35997) 
Cherry pick from master
pr: #35128 #35918
See also #35303

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: aoiasd <45024769+aoiasd@users.noreply.github.com>
 2024-09-05 16:43:06 +08:00
jaime
2c1fa50412
enhance: remove cooling off in rate limiter for read requests (#35936) 
issue: #35934
pr: #35935

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-09-04 14:39:10 +08:00
congqixia
268c60c301
fix: [2.4] Check clustering key skip load behavior (#35865) (#35899) 
Cherry-pick from master
pr: #35865
feature issue: #35415
See also #35861

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-02 16:03:04 +08:00
SimFG
8b706122a8
enhance: [2.4] support to drop the role which is related the privilege list (#35863) 
- issue: #35545
- pr: #35727

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-08-31 21:57:02 +08:00
congqixia
fedf3a7fe1
enhance: Bump milvus & proto version to v2.4.10 (#35798) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-29 14:55:01 +08:00
jaime
2fc111eb10
enhance: set database properties to restrict read access (#35754) 
issue: #35744
pr: #35745

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-08-29 11:41:01 +08:00
SimFG
fc324b4254
feat: [2.4] add the rbac msg and send them to the replicate channel (#35562) 
- issue: #35391
- pr: #35392

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-08-27 14:45:00 +08:00
congqixia
ab261d0f8b
feat: [2.4] Support field partial load collection (#35416) (#35696) 
Cherry-pick from master
pr: #35416
Related to #35415

---------

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-27 14:07:00 +08:00
Gao
1687d64c46
enhance: [2.4] add hit segment num metrics for queryHook (#35619) 
issue: #35576 
pr: #35577

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-08-23 12:49:02 +08:00
SimFG
5b5119a51f
feat: [2.4] provide more general configuration to control mmap behavior (#35609) 
- issue: #35273
- pr: #35359

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-08-23 12:35:02 +08:00
wei liu
e014ad9280
fix: fix dynamic update config doesn't works for some param (#35572) (#35637) 
issue: #35570
pr: #35572
milvus support config cache to spped up config access, but only evict
param's cache when param has been updated. but milvus's param may rely
on other param's value, let's say ParamsA relys on paramsB, when paramsB
updated, it will evict paramB's cache, but the paramA's cache still keep
the old value.

This PR evict all config cache to solve the above issue, cause dynamic
update config won't be much frequetly.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-22 16:00:58 +08:00
wei liu
e2542a1bf5
enhance: Update protobuf-go to protobuf-go v2 (#34394) (#35555) 
issue: #34252
pr: #34394 #35072 #35084

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-21 18:50:58 +08:00
Chun Han
cf8494ef45
enhance: support httpv1/v2 throttle and add it for httpV2(#35350) (#35504) 
related: #35350
pr: https://github.com/milvus-io/milvus/pull/35470

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-08-20 16:32:56 +08:00
wei liu
14ec3dc357
enhance: Enable ReadOnly/ReadWrite/Admin Privilege Group to simplify RBAC grant progress (#35472) (#35543) 
issue: #35471
pr: #35472 #35515

---------

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-19 16:24:54 +08:00
wei liu
248a6ea401
enhance: Add BackupRBAC/RestoreRBAC API to enable rbac backup (#35444) (#35513) 
issue: #35443
pr: #35444

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-18 13:42:55 +08:00
congqixia
4e098f1b23
enhance: Bump milvus & proto to v2.4.9 (#35516) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-16 16:24:54 +08:00
congqixia
c8926aee23
enhance: [2.4] Support dynamic field in SchemaHelper (#35461) (#35469) 
Cherry-pick from master
pr: #35461
Related to #35415

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-15 10:02:54 +08:00
Ted Xu
57d4bcbf15
enhance: adding the msgchannel section in generated yaml (#35466) 
See #32168

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-08-14 19:03:11 +08:00
congqixia
e44c63883b
enhance: Bump milvus & proto to v2.4.8 (#35460) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-14 12:52:52 +08:00
Ted Xu
ce53e79f12
fix: enable milvus.yaml check (#34567) (#35446) 
See #32168

pr: #34567 #35152

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-08-13 19:00:23 +08:00