test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-04 12:59:23 +08:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity
0a2655dba0
milvus/internal/datanode
History
yihao.dai c411cb4a49
enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 
This PR includes the following adjustments:
1. To prevent channelCP update task backlog, only one task with the same
vchannel is retained in the updater. Additionally, the lastUpdateTime is
refreshed after the flowgraph submits the update task, rather than in
the callBack function.
2. Batch updates of multiple vchannel checkpoints are performed in the
UpdateChannelCheckpoint RPC (default batch size is 128). Additionally,
the lock for channelCPs in DataCoord meta has been switched from key
lock to global lock.
3. The concurrency of UpdateChannelCheckpoint RPCs in the datanode has
been reduced from 1000 to 10.

issue: https://github.com/milvus-io/milvus/issues/30004

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: congqixia <congqi.xia@zilliz.com>
2024-03-07 20:39:02 +08:00
..
allocator
broker enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
importv2 feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
io enhance: Reduce unnessary log in binlog_io (#30625) 2024-02-18 16:50:51 +08:00
iterators enhance: Speed up L0 compaction (#30410) 2024-02-04 10:49:05 +08:00
metacache enhance: Use RWMutex and change WLock to RLock (#30557) 2024-02-06 17:13:56 +08:00
syncmgr test: support multiple data/querynodes in integration test (#30618) 2024-02-21 11:54:53 +08:00
writebuffer fix: Decoupling importing segment from flush process (#30402) 2024-02-03 13:01:12 +08:00
binlog_io_test.go enhance: replace binlogIO with io.BinlogIO in datanode (#29725) 2024-02-20 14:38:51 +08:00
binlog_io.go fix: Fix binlog_io metric name conflict (#30689) 2024-03-01 18:13:02 +08:00
cache_test.go
cache.go
channel_checkpoint_updater_test.go enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
channel_checkpoint_updater.go enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
channel_manager_test.go enhance: Change DN channelmanger into interface (#29307) 2023-12-27 16:00:48 +08:00
channel_manager.go enhance: Change DN channelmanger into interface (#29307) 2023-12-27 16:00:48 +08:00
compaction_executor_test.go fix: Separate schedule and check results loop (#28692) 2023-11-29 10:50:29 +08:00
compaction_executor.go fix: Clear DN unkown compaction tasks (#30850) 2024-03-01 11:31:00 +08:00
compactor_test.go enhance: replace binlogIO with io.BinlogIO in datanode (#29725) 2024-02-20 14:38:51 +08:00
compactor.go enhance: replace binlogIO with io.BinlogIO in datanode (#29725) 2024-02-20 14:38:51 +08:00
data_node_test.go enhance: Add FlowgraphManager interface (#28852) 2023-11-30 18:42:32 +08:00
data_node.go enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
data_sync_service_test.go enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
data_sync_service.go test: support multiple data/querynodes in integration test (#30618) 2024-02-21 11:54:53 +08:00
event_manager_test.go enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
event_manager.go test: support multiple data/querynodes in integration test (#30618) 2024-02-21 11:54:53 +08:00
flow_graph_dd_node_test.go
flow_graph_dd_node.go fix: Add more throughput in related metrics (#30038) 2024-01-19 11:34:54 +08:00
flow_graph_dmstream_input_node_test.go
flow_graph_dmstream_input_node.go fix: add back existing datanode metrics (#29360) 2023-12-22 14:20:43 +08:00
flow_graph_manager_test.go enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
flow_graph_manager.go enhance: Remove debug log (#30955) 2024-03-01 10:02:59 +08:00
flow_graph_message_test.go
flow_graph_message.go
flow_graph_node.go
flow_graph_time_tick_node.go enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
flow_graph_time_ticker.go enhance: create goroutine only once (#28594) 2023-11-22 10:26:28 +08:00
flow_graph_write_node.go enhance: Remove commented code and fix naming issue (#28450) 2023-11-16 00:22:20 +08:00
flush_task_counter_test.go
flush_task_counter.go
io_pool_test.go
io_pool.go Fix getting incorrect CPU num (#28146) 2023-11-06 06:02:16 +08:00
l0_compactor_test.go enhance: Speed up L0 compaction (#30410) 2024-02-04 10:49:05 +08:00
l0_compactor.go enhance: Add param item for datanode l0 batch/linear mode memory ratio (#30523) 2024-02-18 13:02:50 +08:00
meta_service_test.go
meta_service.go
meta_util.go
metrics_info.go enhance: Add FlowgraphManager interface (#28852) 2023-11-30 18:42:32 +08:00
mock_channelmanager.go enhance: Change DN channelmanger into interface (#29307) 2023-12-27 16:00:48 +08:00
mock_fgmanager.go enhance: separate serializer logic from sync task (#29413) 2023-12-26 10:40:47 +08:00
mock_test.go test: support multiple data/querynodes in integration test (#30618) 2024-02-21 11:54:53 +08:00
OWNERS
rate_collector_test.go
rate_collector.go
README.md
services_test.go enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
services.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
stats_updater_test.go Use writebuffer, sync manager refactory in datanode (#28320) 2023-11-15 15:24:18 +08:00
stats_updater.go test: support multiple data/querynodes in integration test (#30618) 2024-02-21 11:54:53 +08:00
timetick_sender_test.go enhance: Reduce the memory usage of the timeTickSender (#30968) 2024-03-02 10:13:01 +08:00
timetick_sender.go enhance: Reduce the memory usage of the timeTickSender (#30968) 2024-03-02 10:13:01 +08:00
util.go

README.md

Data Node

DataNode is the component to write insert and delete messages into persistent blob storage, for example MinIO or S3.

Dependency

  • KV store: a kv store that persists messages into blob storage.
  • Message stream: receive messages and publish imformation
  • Root Coordinator: get the latest unique IDs.
  • Data Coordinator: get the flush information and which message stream to subscribe.