test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-02 03:48:37 +08:00
Code Issues Actions 12 Packages Projects Releases Wiki Activity
57a5e44b88
milvus/internal/datanode
History
yihao.dai b5c67948b7
enhance: Enhance and modify the return content of ImportV2 (#31192) 
1. The Import APIs now provide detailed progress information for each
imported file, including details such as file name, file size, progress,
and more.
2. The APIs now return the collection name and the completion time.
3. Other modifications include changing jobID to jobId and other similar
adjustments.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-13 19:51:03 +08:00
..
allocator Replace manual composed grpc call with Broker methods (#27676) 2023-10-13 09:55:34 +08:00
broker enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
importv2 enhance: Enhance and modify the return content of ImportV2 (#31192) 2024-03-13 19:51:03 +08:00
io enhance: Reduce unnessary log in binlog_io (#30625) 2024-02-18 16:50:51 +08:00
iterators enhance: Speed up L0 compaction (#30410) 2024-02-04 10:49:05 +08:00
metacache enhance: Add segment id short cut for WithSegmentID filter (#31144) 2024-03-11 10:55:02 +08:00
syncmgr fix: TestBlock case use different segment id in testcase (#31173) 2024-03-11 17:51:03 +08:00
writebuffer fix: Decoupling importing segment from flush process (#30402) 2024-02-03 13:01:12 +08:00
binlog_io_test.go enhance: replace binlogIO with io.BinlogIO in datanode (#29725) 2024-02-20 14:38:51 +08:00
binlog_io.go enhance: Enable binlog deserialize reader in datanode compaction (#31036) 2024-03-08 18:25:02 +08:00
cache_test.go Make duplicate flush recalls success and other logic update (#16755) 2022-05-06 17:49:51 +08:00
cache.go Use typeutil.ConcurrentMap instead of sync.Map (#25846) 2023-07-24 10:23:01 +08:00
channel_checkpoint_updater_test.go enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
channel_checkpoint_updater.go enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
channel_manager_test.go enhance: Change DN channelmanger into interface (#29307) 2023-12-27 16:00:48 +08:00
channel_manager.go enhance: Change DN channelmanger into interface (#29307) 2023-12-27 16:00:48 +08:00
compaction_executor_test.go fix: Separate schedule and check results loop (#28692) 2023-11-29 10:50:29 +08:00
compaction_executor.go fix: Clear DN unkown compaction tasks (#30850) 2024-03-01 11:31:00 +08:00
compactor_test.go enhance: Add segment id short cut for WithSegmentID filter (#31144) 2024-03-11 10:55:02 +08:00
compactor.go enhance: Enable binlog deserialize reader in datanode compaction (#31036) 2024-03-08 18:25:02 +08:00
data_node_test.go enhance: Add FlowgraphManager interface (#28852) 2023-11-30 18:42:32 +08:00
data_node.go enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
data_sync_service_test.go enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
data_sync_service.go test: support multiple data/querynodes in integration test (#30618) 2024-02-21 11:54:53 +08:00
event_manager_test.go enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
event_manager.go test: support multiple data/querynodes in integration test (#30618) 2024-02-21 11:54:53 +08:00
flow_graph_dd_node_test.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
flow_graph_dd_node.go fix: Add more throughput in related metrics (#30038) 2024-01-19 11:34:54 +08:00
flow_graph_dmstream_input_node_test.go Support to replicate the mq message (#27240) 2023-10-20 14:26:09 +08:00
flow_graph_dmstream_input_node.go fix: add back existing datanode metrics (#29360) 2023-12-22 14:20:43 +08:00
flow_graph_manager_test.go enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
flow_graph_manager.go enhance: Remove debug log (#30955) 2024-03-01 10:02:59 +08:00
flow_graph_message_test.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
flow_graph_message.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
flow_graph_node.go Make flow graph retry longer (#24355) 2023-05-25 09:53:30 +08:00
flow_graph_time_tick_node.go enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
flow_graph_time_ticker.go enhance: create goroutine only once (#28594) 2023-11-22 10:26:28 +08:00
flow_graph_write_node.go enhance: Remove commented code and fix naming issue (#28450) 2023-11-16 00:22:20 +08:00
flush_task_counter_test.go Fix flushManager.isFull is too slow (#28141) 2023-11-03 14:42:17 +08:00
flush_task_counter.go Fix flushManager.isFull is too slow (#28141) 2023-11-03 14:42:17 +08:00
io_pool_test.go Fix copylocks linter errors (#26217) 2023-08-08 20:41:15 +08:00
io_pool.go Fix getting incorrect CPU num (#28146) 2023-11-06 06:02:16 +08:00
l0_compactor_test.go enhance: Speed up L0 compaction (#30410) 2024-02-04 10:49:05 +08:00
l0_compactor.go enhance: Add param item for datanode l0 batch/linear mode memory ratio (#30523) 2024-02-18 13:02:50 +08:00
meta_service_test.go Replace manual composed grpc call with Broker methods (#27676) 2023-10-13 09:55:34 +08:00
meta_service.go Replace manual composed grpc call with Broker methods (#27676) 2023-10-13 09:55:34 +08:00
meta_util.go Use go-api/v2 for milvus-proto (#24770) 2023-06-09 01:28:37 +08:00
metrics_info.go enhance: Add FlowgraphManager interface (#28852) 2023-11-30 18:42:32 +08:00
mock_channelmanager.go enhance: Change DN channelmanger into interface (#29307) 2023-12-27 16:00:48 +08:00
mock_fgmanager.go enhance: separate serializer logic from sync task (#29413) 2023-12-26 10:40:47 +08:00
mock_test.go test: support multiple data/querynodes in integration test (#30618) 2024-02-21 11:54:53 +08:00
OWNERS [skip ci]Update OWNERS files (#11898) 2021-11-16 15:41:11 +08:00
rate_collector_test.go Move some modules from internal to public package (#22572) 2023-04-06 19:14:32 +08:00
rate_collector.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
README.md [skip ci]Change uppercase of MinIO (#9539) 2021-10-09 11:22:58 +08:00
services_test.go enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
services.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
stats_updater_test.go Use writebuffer, sync manager refactory in datanode (#28320) 2023-11-15 15:24:18 +08:00
stats_updater.go test: support multiple data/querynodes in integration test (#30618) 2024-02-21 11:54:53 +08:00
timetick_sender_test.go enhance: Reduce the memory usage of the timeTickSender (#30968) 2024-03-02 10:13:01 +08:00
timetick_sender.go enhance: Reduce the memory usage of the timeTickSender (#30968) 2024-03-02 10:13:01 +08:00
util.go Format the code (#27275) 2023-09-21 09:45:27 +08:00

README.md

Data Node

DataNode is the component to write insert and delete messages into persistent blob storage, for example MinIO or S3.

Dependency

  • KV store: a kv store that persists messages into blob storage.
  • Message stream: receive messages and publish imformation
  • Root Coordinator: get the latest unique IDs.
  • Data Coordinator: get the flush information and which message stream to subscribe.