milvus/pkg
yihao.dai c411cb4a49
enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941)
This PR includes the following adjustments:
1. To prevent channelCP update task backlog, only one task with the same
vchannel is retained in the updater. Additionally, the lastUpdateTime is
refreshed after the flowgraph submits the update task, rather than in
the callBack function.
2. Batch updates of multiple vchannel checkpoints are performed in the
UpdateChannelCheckpoint RPC (default batch size is 128). Additionally,
the lock for channelCPs in DataCoord meta has been switched from key
lock to global lock.
3. The concurrency of UpdateChannelCheckpoint RPCs in the datanode has
been reduced from 1000 to 10.

issue: https://github.com/milvus-io/milvus/issues/30004

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: congqixia <congqi.xia@zilliz.com>
2024-03-07 20:39:02 +08:00
..
common fix: check collection lazy load prop using schema (#30992) 2024-03-06 16:19:01 +08:00
config enhance: support clean paramtable config event in test (#30534) 2024-02-20 14:16:51 +08:00
eventlog enhance:don't store logPath in meta to reduce memory (#28873) 2024-01-18 22:06:31 +08:00
log fix: log can not reduce level (#28750) 2023-11-28 10:02:27 +08:00
metrics feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
mq fix: Support TLS for kafka connection (#30468) 2024-02-28 18:43:07 +08:00
tracer enhance: Make compaction log has traceID (#30338) 2024-01-30 10:09:03 +08:00
util enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
go.mod enhance: use latest milvus proto, add unimplemented server (#30889) 2024-02-29 10:13:00 +08:00
go.sum enhance: use latest milvus proto, add unimplemented server (#30889) 2024-02-29 10:13:00 +08:00
Makefile Add the mock factory of msg stream (#27834) 2023-10-20 20:14:08 +08:00
rules.go Add back gocritic linter and fix related issues (#27289) 2023-09-22 10:05:26 +08:00