milvus/internal/metastore
yihao.dai c411cb4a49
enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941)
This PR includes the following adjustments:
1. To prevent channelCP update task backlog, only one task with the same
vchannel is retained in the updater. Additionally, the lastUpdateTime is
refreshed after the flowgraph submits the update task, rather than in
the callBack function.
2. Batch updates of multiple vchannel checkpoints are performed in the
UpdateChannelCheckpoint RPC (default batch size is 128). Additionally,
the lock for channelCPs in DataCoord meta has been switched from key
lock to global lock.
3. The concurrency of UpdateChannelCheckpoint RPCs in the datanode has
been reduced from 1000 to 10.

issue: https://github.com/milvus-io/milvus/issues/30004

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: congqixia <congqi.xia@zilliz.com>
2024-03-07 20:39:02 +08:00
..
kv enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
mocks enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
model feat: add clustering key in create/describe collection (#29506) 2024-01-07 19:56:48 +08:00
catalog_test.go Refactor RootCoord (#18930) 2022-09-05 13:29:11 +08:00
catalog.go enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00