milvus/internal
yihao.dai c411cb4a49
enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941)
This PR includes the following adjustments:
1. To prevent channelCP update task backlog, only one task with the same
vchannel is retained in the updater. Additionally, the lastUpdateTime is
refreshed after the flowgraph submits the update task, rather than in
the callBack function.
2. Batch updates of multiple vchannel checkpoints are performed in the
UpdateChannelCheckpoint RPC (default batch size is 128). Additionally,
the lock for channelCPs in DataCoord meta has been switched from key
lock to global lock.
3. The concurrency of UpdateChannelCheckpoint RPCs in the datanode has
been reduced from 1000 to 10.

issue: https://github.com/milvus-io/milvus/issues/30004

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: congqixia <congqi.xia@zilliz.com>
2024-03-07 20:39:02 +08:00
..
allocator enhance:change some logs (#29579) 2024-01-05 16:12:48 +08:00
core [automated] Update Knowhere Commit (#31089) 2024-03-07 12:05:02 +08:00
datacoord enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
datanode enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
distributed enhance: Add ListIndexes API from datacoord (#31104) 2024-03-07 17:37:01 +08:00
http
indexnode enhance: add graceful stop timeout to avoid node stop hang under extreme cases (#30317) 2024-02-29 17:01:50 +08:00
kv enhance: disallow tikv/rawkv usages (#30027) 2024-02-06 10:41:41 +08:00
metastore enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
mocks enhance: Add ListIndexes API from datacoord (#31104) 2024-03-07 17:37:01 +08:00
mq fix: close rocksmq and natsmq when shutdown (#30706) 2024-02-26 11:38:55 +08:00
parser/planparserv2 enhance: Avoid create schema helper for each read task (#30981) 2024-03-04 19:39:00 +08:00
proto enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
proxy fix: set CollectionID field for AlterCollection request (#30814) 2024-03-07 14:49:02 +08:00
querycoordv2 fix: Dirty sealed segment won't release after channel balance (#31095) 2024-03-07 16:23:01 +08:00
querynodev2 fix: check collection lazy load prop using schema (#30992) 2024-03-06 16:19:01 +08:00
registry
rootcoord enhance: add graceful stop timeout to avoid node stop hang under extreme cases (#30317) 2024-02-29 17:01:50 +08:00
storage feat: Define FieldValue, FieldStats and PartitionStats (#30286) 2024-03-06 20:42:37 -08:00
tso
types feat: Revise the RESTful bulk insert API (#29698) 2024-03-05 15:03:00 +08:00
util enhance: support stream call for grpc client (#30013) 2024-03-07 17:45:01 +08:00