Commit Graph

146 Commits

Author SHA1 Message Date
chyezh
2586c2f1b3
enhance: use WalkWithPrefix api for oss, enable piplined file gc (#31740)
issue: #19095,#29655,#31718

- Change `ListWithPrefix` to `WalkWithPrefix` of OOS into a pipeline
mode.

- File garbage collection is performed in other goroutine.

- Segment Index Recycle clean index file too.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-25 20:41:27 +08:00
chyezh
a2502bde75
enhance: replica manager enhancement (#31496)
issue: #30647 

- ReplicaManager manage read only node now, and always do persistent of
node distribution of replica.

- All segment/channel checker using ReplicaManager to get read-only node
or read-write node, but not ResourceManager.

- ReplicaManager promise that only apply unique querynode to one replica
in same collection now (replicas in same collection never hold same
querynode at same time).

- ReplicaManager promise that fairly node count assignment policy if
multi replicas of collection is assigned to one resource group.

- Move some parameters check into ReplicaManager to avoid data race.

- Allow transfer replica to resource group that already load replica of
same collection

- Allow transfer node between resource groups that load replica of same
collection

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-05 04:57:16 +08:00
SimFG
8f3e0b6b41
enhance: the return result of list db api (#31544)
issue: #31543

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-03-28 07:13:10 +08:00
SimFG
b1a1cca10b
feat: add more operation detail info for better allocation (#30438)
issue: #30436

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-03-28 06:33:11 +08:00
congqixia
8e5865f630
enhance: Save collection targets by batches (#31616)
See also #28491 #31240

When colleciton number is large, querycoord saves collection target one
by one, which is slow and may block querycoord exits.

In local run, 500 collections scenario may lead to about 40 seconds
saving collection targets.

This PR changes the `SaveCollectionTarget` interface into batch one and
organizes the collection in 16 per bundle batches to accelerate this
procedure.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-27 00:09:08 +08:00
Bingyi Sun
8e661f791a
fix: lazy load index data in cache (#31094)
issue: https://github.com/milvus-io/milvus/issues/31571

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-25 15:43:07 +08:00
wei liu
e6d50def4f
enhance: Enable properites in database meta (#31394)
issue: #30040

This PR add properties in database meta, so we can store some database
level info.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-19 19:21:06 +08:00
wei liu
d79aa58b37
enhance: Speed up target recovery after query coord restart (#31240)
issue: #28491

after querycoord restart, it will pull a new target, which include
channel and segment list. when segments loaded on querynode has reached
the target, the collection could provide search/query. but if segment
list changes by time, ater querycoord pull a new target, it will takes a
few minutes to catch up the target's segment distribution. and before
that, query/search will fail due to lack of segments.

This PR save the current loaded target to meta storein querycoord's stop
progress, and recover it when query coord starts, to speed up the target
recovery time.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-15 14:19:03 +08:00
Bingyi Sun
5c0bb40549
fix: merge index params when creating index (#31127)
issue: https://github.com/milvus-io/milvus/issues/31102

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-11 17:31:03 +08:00
chyezh
6a9418d200
fix: lost dbname when only passing collection id to describeCollection (#31167)
issue: #30931

Signed-off-by: chyezh <chyezh@outlook.com>
2024-03-11 17:19:02 +08:00
yihao.dai
c411cb4a49
enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941)
This PR includes the following adjustments:
1. To prevent channelCP update task backlog, only one task with the same
vchannel is retained in the updater. Additionally, the lastUpdateTime is
refreshed after the flowgraph submits the update task, rather than in
the callBack function.
2. Batch updates of multiple vchannel checkpoints are performed in the
UpdateChannelCheckpoint RPC (default batch size is 128). Additionally,
the lock for channelCPs in DataCoord meta has been switched from key
lock to global lock.
3. The concurrency of UpdateChannelCheckpoint RPCs in the datanode has
been reduced from 1000 to 10.

issue: https://github.com/milvus-io/milvus/issues/30004

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: congqixia <congqi.xia@zilliz.com>
2024-03-07 20:39:02 +08:00
yihao.dai
a434d33e75
feat: Add import scheduler and manager (#29367)
This PR introduces novel managerial roles for importv2:
1. ImportMeta: To manage all the import tasks;
2. ImportScheduler: To process tasks and modify their states;
3. ImportChecker: To ascertain the completion of all tasks and instigate
relevant operations.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-01 18:31:02 +08:00
smellthemoon
da50fbfbf4
fix: not store timestampTo and timestampFrom (#30450)
#30451

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-02-04 14:23:05 +08:00
congqixia
e3b8c3f60b
fix: collection properties not saved for alter collection (#30145)
Resolves: #30144

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-19 23:06:54 +08:00
smellthemoon
e52ce370b6
enhance:don't store logPath in meta to reduce memory (#28873)
don't store logPath in meta to reduce memory, when service get
segmentinfo, generate logpath from logid.
#28885

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-18 22:06:31 +08:00
smellthemoon
073de855b3
enhance:remove some unused code (#29827)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-14 10:38:51 +08:00
wei liu
9fc5f1176c
fix: Drop segment meta info with prefix (#29856)
If segment has more than 128 log fils, drop segment will exceed etcd txn
ops limit, which will failed the drop segment request
This PR drop segment meta info with prefix, to avoid drop segment meta
failed

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-11 12:22:51 +08:00
wayblink
635a7f777c
feat: add clustering key in create/describe collection (#29506)
#28410
/kind feature

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-01-07 19:56:48 +08:00
SimFG
5a7fb97e07
fix: missing the grant info when using the SelectGrant api with the default db (#29173)
issue: #29172

Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-12-14 16:20:43 +08:00
Bingyi Sun
ad866d2889
feat: integrate storagev2 into index build process (#28995)
issue: https://github.com/milvus-io/milvus/issues/28994

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-12-13 17:24:38 +08:00
Jiquan Long
e5f155612a
fix: txn control not applied to snapshot.Remove (#29124)
issue: #29123

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-12-12 14:48:38 +08:00
yihao.dai
7f819ac594
enhance: Remove partition load infos with prefix (#28969)
Removing partition load infos with prefix instead of removing them one
by one. This enhance can save transactions.

issue: https://github.com/milvus-io/milvus/issues/28343

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-12-05 12:52:34 +08:00
wayblink
7f46f4c628
fix: failed to release collection with more than 128 partitions (#28446)
#28343

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-11-19 18:28:21 +08:00
aoiasd
1d4be0d257
Adjust datacoord for L0 Delta (#28021)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-06 15:26:16 +08:00
Xiaofan
2ea7579dbb
Reduce rpc size for GetRecoveryInfoV2 (#27483)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-10-23 21:44:09 +08:00
Jiquan Long
babf4e37e4
Fix partition's gc (#27816)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-10-20 16:30:09 +08:00
yah01
15746b5b5f
Refine KV errors (#27588)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-13 17:15:34 +08:00
Jiquan Long
370fdaf50d
Record engine version for segment index (#27384)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-09-28 18:03:28 +08:00
yah01
6539a5ae2c
Refine DataCoord status (#27262)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-26 17:15:27 +08:00
foxspy
5db4a0489e
dynamic index version control (#27335)
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-09-25 21:39:27 +08:00
foxspy
370b6fde58
milvus support multi index engine (#27178)
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-09-22 09:59:26 +08:00
SimFG
26f06dd732
Format the code (#27275)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
cai.zhang
a362bb1457
Support array datatype (#26369)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-09-19 14:23:23 +08:00
congqixia
cc9974979f
Add staticcheck linter and fix existing problems (#27174)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-19 10:05:22 +08:00
Enwei Jiao
fb0705df1b
Decouple basetable and componentparam (#26725)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-05 10:31:48 +08:00
congqixia
1a8cf5c415
Organize all mockery generation commands in Makefile (#26826)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-04 21:19:48 +08:00
XuanYang-cn
b2e7cbdf4b
Remove TimeTravel in compactor (#26785)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-09-04 17:41:48 +08:00
yah01
3349db4aa7
Refine errors to remove changes breaking design (#26521)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-04 09:57:09 +08:00
yah01
941a383019
Fix failed to load collection with more than 128 partitions (#26763)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-02 00:09:01 +08:00
jaime
d001133bd2
[skip e2e] Refine collection not exists error (#26707)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-08-31 17:29:03 +08:00
yah01
213db490bd
Use pointer receiver for large struct (#26668)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-30 10:24:29 +08:00
jaime
9f61dba3f9
Support rename db name of the collection (#26543)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-08-29 14:54:26 +08:00
jaime
c603f1c244
Remove mysql metastore (#26633)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-08-29 14:36:26 +08:00
Xiaofan
f9c47bbc1e
Fix drop index with large txn exceed etcd limit (#25622)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-08-17 21:04:21 +08:00
SimFG
d2649b63db
Delete the user-role mapping info when deleting the user (#25988)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-08-04 18:37:08 +08:00
Bingyi Sun
a3e22786ed
Move meta store to kv catalog (#25915)
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-07-31 13:57:04 +08:00
Filip Haltmayer
7bd9eceb15
Remove unused KV functions (#25972)
Remove RevertAlterSegmentsAndAddNewSegment



Remove AlterIndex



Remove AlterSegmentsAndAddNewSegment



Remove AlterSegmentIndex



Remove IndexCoordCatalog

Signed-off-by: Filip Haltmayer <filip.haltmayer@zilliz.com>
2023-07-28 10:13:03 +08:00
Jiquan Long
bccdef1ad7
Fix segment meta only record last flushed binlog (#25707)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-07-19 15:28:57 +08:00
Enwei Jiao
66fdc71479
Refactor logs in DataCoord & DataNode (#25574)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-07-14 15:56:31 +08:00
Bingyi Sun
b22d6d2a5d
Avoid update all binlogs every a segment is updated (#25526)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-07-13 14:08:29 +08:00