test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-02 11:59:00 +08:00
Code Issues Actions 12 Packages Projects Releases Wiki Activity
a68b32134a
milvus/internal/datanode
History
congqixia a68b32134a
fix: Verify sync task target segment and retry if not match (#30500) 
See also #27675 #30469

For a sync task, the segment could be compacted during sync task. In
previous implementation, this sync task will hold only the old segment
id as KeyLock, in which case compaction on compacted to segment may run
in parallel with delta sync of this sync task.

This PR introduces sync target segment verification logic. It shall
check target segment lock it's holding beforing actually syncing logic.
If this check failed, sync task shall return`errTargetSegementNotMatch`
error and make manager re-fetch the current target segment id.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-05 11:33:43 +08:00
..
allocator Replace manual composed grpc call with Broker methods (#27676) 2023-10-13 09:55:34 +08:00
broker enhance:don't store logPath in meta to reduce memory (#28873) 2024-01-18 22:06:31 +08:00
importv2 enhance: Extend support for varchar autoID to BulkInsertV2 (#30477) 2024-02-04 16:57:05 +08:00
io enhance: Speed up L0 compaction (#30410) 2024-02-04 10:49:05 +08:00
iterators enhance: Speed up L0 compaction (#30410) 2024-02-04 10:49:05 +08:00
metacache fix: Decoupling importing segment from flush process (#30402) 2024-02-03 13:01:12 +08:00
syncmgr fix: Verify sync task target segment and retry if not match (#30500) 2024-02-05 11:33:43 +08:00
writebuffer fix: Decoupling importing segment from flush process (#30402) 2024-02-03 13:01:12 +08:00
binlog_io_test.go enhance: make compactor use actual buffer size to decide when to sync (#29945) 2024-01-13 01:32:52 +08:00
binlog_io.go enhance: Add open telemetry tracing for compaction (#30168) 2024-01-23 10:37:00 +08:00
cache_test.go Make duplicate flush recalls success and other logic update (#16755) 2022-05-06 17:49:51 +08:00
cache.go Use typeutil.ConcurrentMap instead of sync.Map (#25846) 2023-07-24 10:23:01 +08:00
channel_checkpoint_updater.go feat: Introduce channelCheckpointUpdater to reduce goroutine use in ttNode (#28570) 2023-12-12 13:48:42 +08:00
channel_manager_test.go enhance: Change DN channelmanger into interface (#29307) 2023-12-27 16:00:48 +08:00
channel_manager.go enhance: Change DN channelmanger into interface (#29307) 2023-12-27 16:00:48 +08:00
compaction_executor_test.go fix: Separate schedule and check results loop (#28692) 2023-11-29 10:50:29 +08:00
compaction_executor.go fix: call injectDone after compaction failed (#30277) 2024-02-01 14:25:04 +08:00
compactor_test.go fix: call injectDone after compaction failed (#30277) 2024-02-01 14:25:04 +08:00
compactor.go fix: call injectDone after compaction failed (#30277) 2024-02-01 14:25:04 +08:00
data_node_test.go enhance: Add FlowgraphManager interface (#28852) 2023-11-30 18:42:32 +08:00
data_node.go feat: Add import executor and manager for datanode (#29438) 2024-01-31 20:45:04 +08:00
data_sync_service_test.go fix: Use channel cp as the dml&start position for import segments (#30107) 2024-01-22 14:36:55 +08:00
data_sync_service.go enhance:don't store logPath in meta to reduce memory (#28873) 2024-01-18 22:06:31 +08:00
event_manager_test.go enhance: Add FlowgraphManager interface (#28852) 2023-11-30 18:42:32 +08:00
event_manager.go fix: panic with datanode negetive wait group counter (#30135) 2024-01-30 18:15:04 +08:00
flow_graph_dd_node_test.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
flow_graph_dd_node.go fix: Add more throughput in related metrics (#30038) 2024-01-19 11:34:54 +08:00
flow_graph_dmstream_input_node_test.go Support to replicate the mq message (#27240) 2023-10-20 14:26:09 +08:00
flow_graph_dmstream_input_node.go fix: add back existing datanode metrics (#29360) 2023-12-22 14:20:43 +08:00
flow_graph_manager_test.go enhance: Add FlowgraphManager interface (#28852) 2023-11-30 18:42:32 +08:00
flow_graph_manager.go enhance: Add FlowgraphManager interface (#28852) 2023-11-30 18:42:32 +08:00
flow_graph_message_test.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
flow_graph_message.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
flow_graph_node.go Make flow graph retry longer (#24355) 2023-05-25 09:53:30 +08:00
flow_graph_time_tick_node.go fix: Skip updating checkpoint after dropcollection (#29220) 2023-12-15 16:04:45 +08:00
flow_graph_time_ticker.go enhance: create goroutine only once (#28594) 2023-11-22 10:26:28 +08:00
flow_graph_write_node.go enhance: Remove commented code and fix naming issue (#28450) 2023-11-16 00:22:20 +08:00
flush_task_counter_test.go Fix flushManager.isFull is too slow (#28141) 2023-11-03 14:42:17 +08:00
flush_task_counter.go Fix flushManager.isFull is too slow (#28141) 2023-11-03 14:42:17 +08:00
io_pool_test.go Fix copylocks linter errors (#26217) 2023-08-08 20:41:15 +08:00
io_pool.go Fix getting incorrect CPU num (#28146) 2023-11-06 06:02:16 +08:00
l0_compactor_test.go enhance: Speed up L0 compaction (#30410) 2024-02-04 10:49:05 +08:00
l0_compactor.go enhance: Speed up L0 compaction (#30410) 2024-02-04 10:49:05 +08:00
meta_service_test.go Replace manual composed grpc call with Broker methods (#27676) 2023-10-13 09:55:34 +08:00
meta_service.go Replace manual composed grpc call with Broker methods (#27676) 2023-10-13 09:55:34 +08:00
meta_util.go Use go-api/v2 for milvus-proto (#24770) 2023-06-09 01:28:37 +08:00
metrics_info.go enhance: Add FlowgraphManager interface (#28852) 2023-11-30 18:42:32 +08:00
mock_channelmanager.go enhance: Change DN channelmanger into interface (#29307) 2023-12-27 16:00:48 +08:00
mock_fgmanager.go enhance: separate serializer logic from sync task (#29413) 2023-12-26 10:40:47 +08:00
mock_test.go enhance: make compactor use actual buffer size to decide when to sync (#29945) 2024-01-13 01:32:52 +08:00
OWNERS [skip ci]Update OWNERS files (#11898) 2021-11-16 15:41:11 +08:00
rate_collector_test.go Move some modules from internal to public package (#22572) 2023-04-06 19:14:32 +08:00
rate_collector.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
README.md [skip ci]Change uppercase of MinIO (#9539) 2021-10-09 11:22:58 +08:00
services_test.go fix: Use channel cp as the dml&start position for import segments (#30107) 2024-01-22 14:36:55 +08:00
services.go fix: Decoupling importing segment from flush process (#30402) 2024-02-03 13:01:12 +08:00
stats_updater_test.go Use writebuffer, sync manager refactory in datanode (#28320) 2023-11-15 15:24:18 +08:00
stats_updater.go enhance:Remove msgbase unnecessary assignments (#28511) 2023-11-24 15:02:39 +08:00
timetick_sender_test.go Refine datanode Timetick Sender (#28393) 2023-11-13 21:46:20 +08:00
timetick_sender.go Refine datanode Timetick Sender (#28393) 2023-11-13 21:46:20 +08:00
util.go Format the code (#27275) 2023-09-21 09:45:27 +08:00

README.md

Data Node

DataNode is the component to write insert and delete messages into persistent blob storage, for example MinIO or S3.

Dependency

  • KV store: a kv store that persists messages into blob storage.
  • Message stream: receive messages and publish imformation
  • Root Coordinator: get the latest unique IDs.
  • Data Coordinator: get the flush information and which message stream to subscribe.