test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-04 12:59:23 +08:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity
0a2655dba0
milvus/internal/datacoord
History
yihao.dai c411cb4a49
enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 
This PR includes the following adjustments:
1. To prevent channelCP update task backlog, only one task with the same
vchannel is retained in the updater. Additionally, the lastUpdateTime is
refreshed after the flowgraph submits the update task, rather than in
the callBack function.
2. Batch updates of multiple vchannel checkpoints are performed in the
UpdateChannelCheckpoint RPC (default batch size is 128). Additionally,
the lock for channelCPs in DataCoord meta has been switched from key
lock to global lock.
3. The concurrency of UpdateChannelCheckpoint RPCs in the datanode has
been reduced from 1000 to 10.

issue: https://github.com/milvus-io/milvus/issues/30004

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: congqixia <congqi.xia@zilliz.com>
2024-03-07 20:39:02 +08:00
..
broker enhance: index meta use independent rather than global meta lock (#30869) 2024-03-04 16:56:59 +08:00
allocator_test.go
allocator.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
build_index_policy.go
channel_checker_test.go
channel_checker.go
channel_manager_factory.go enhance: Remove not in use policies (#29448) 2023-12-28 10:38:46 +08:00
channel_manager_test.go enhance: Change ChannelManager to interface (#29300) 2023-12-25 19:24:46 +08:00
channel_manager.go enhance:change some logs (#29579) 2024-01-05 16:12:48 +08:00
channel_store_test.go Change channel to Interface (#27839) 2023-11-13 11:16:18 +08:00
channel_store.go Change channel to Interface (#27839) 2023-11-13 11:16:18 +08:00
channel.go Change channel to Interface (#27839) 2023-11-13 11:16:18 +08:00
cluster_test.go enhance: Change ChannelManager to interface (#29300) 2023-12-25 19:24:46 +08:00
cluster.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
compaction_l0_view_test.go enhance: Add force trigger (#30641) 2024-02-19 18:40:50 +08:00
compaction_l0_view.go fix: Trigger l0 compaction when l0 views don't change (#30729) 2024-03-05 16:37:00 +08:00
compaction_scheduler_test.go enhance: Add CompactionTaskNum metrics (#29518) 2023-12-28 15:46:55 +08:00
compaction_scheduler.go fix: Trigger l0 compaction when l0 views don't change (#30729) 2024-03-05 16:37:00 +08:00
compaction_test.go fix: Skip invalid compaction plan (#31045) 2024-03-06 21:36:59 +08:00
compaction_trigger_test.go enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
compaction_trigger_v2_test.go fix: Trigger l0 compaction when l0 views don't change (#30729) 2024-03-05 16:37:00 +08:00
compaction_trigger_v2.go fix: Trigger l0 compaction when l0 views don't change (#30729) 2024-03-05 16:37:00 +08:00
compaction_trigger.go enhance: Check channel cp lag before generate compaction task (#30997) 2024-03-05 13:39:01 +08:00
compaction_view_manager_test.go fix: Trigger l0 compaction when l0 views don't change (#30729) 2024-03-05 16:37:00 +08:00
compaction_view_manager.go fix: Trigger l0 compaction when l0 views don't change (#30729) 2024-03-05 16:37:00 +08:00
compaction_view.go fix: Trigger l0 compaction when l0 views don't change (#30729) 2024-03-05 16:37:00 +08:00
compaction.go fix: Skip invalid compaction plan (#31045) 2024-03-06 21:36:59 +08:00
const.go enhance: pass partition key scalar info if enabled when build vector index (#29931) 2024-01-24 00:04:55 +08:00
errors_test.go
errors.go
garbage_collector_test.go enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
garbage_collector.go enhance: index meta use independent rather than global meta lock (#30869) 2024-03-04 16:56:59 +08:00
handler.go enhance: Explicitly pass LevelZero segment ids in vchan info (#29612) 2024-01-04 16:46:45 +08:00
import_checker_test.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
import_checker.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
import_job.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
import_meta_test.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
import_meta.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
import_scheduler_test.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
import_scheduler.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
import_task.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
import_util_test.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
import_util.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
index_builder_test.go enhance: index meta use independent rather than global meta lock (#30869) 2024-03-04 16:56:59 +08:00
index_builder.go enhance: index meta use independent rather than global meta lock (#30869) 2024-03-04 16:56:59 +08:00
index_engine_version_manager_test.go Optimize the codec code of session (#27360) 2023-10-01 10:33:30 +08:00
index_engine_version_manager.go feat: trigger compaction to handle index version (#28442) 2023-11-21 09:26:22 +08:00
index_meta_test.go enhance: index meta use independent rather than global meta lock (#30869) 2024-03-04 16:56:59 +08:00
index_meta.go fix: filter mmap key when checking index params (#31030) 2024-03-06 16:03:00 +08:00
index_service_test.go enhance: Add ListIndexes API from datacoord (#31104) 2024-03-07 17:37:01 +08:00
index_service.go enhance: Add ListIndexes API from datacoord (#31104) 2024-03-07 17:37:01 +08:00
indexnode_manager_test.go Refine state check (#27541) 2023-10-11 21:01:35 +08:00
indexnode_manager.go
meta_test.go enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
meta_util.go
meta.go enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
metrics_info_test.go Refine state check (#27541) 2023-10-11 21:01:35 +08:00
metrics_info.go disable auto balance when old node exists (#28191) 2023-11-07 14:02:20 +08:00
mock_allocator_test.go enhance: index meta use independent rather than global meta lock (#30869) 2024-03-04 16:56:59 +08:00
mock_channel_store.go Change channel to Interface (#27839) 2023-11-13 11:16:18 +08:00
mock_channelmanager.go enhance: Add CompactionTaskNum metrics (#29518) 2023-12-28 15:46:55 +08:00
mock_cluster.go enhance: index meta use independent rather than global meta lock (#30869) 2024-03-04 16:56:59 +08:00
mock_compaction_meta.go enhance: Refine compaction interfaces to support major compaction (#30632) 2024-02-19 20:52:50 +08:00
mock_compaction_plan_context.go fix: Separate schedule and check results loop (#28692) 2023-11-29 10:50:29 +08:00
mock_handler.go Change channel to Interface (#27839) 2023-11-13 11:16:18 +08:00
mock_index_engine_version_manager.go feat: trigger compaction to handle index version (#28442) 2023-11-21 09:26:22 +08:00
mock_scheduler.go enhance: Add CompactionTaskNum metrics (#29518) 2023-12-28 15:46:55 +08:00
mock_segment_manager.go enhance: index meta use independent rather than global meta lock (#30869) 2024-03-04 16:56:59 +08:00
mock_session_manager.go enhance: index meta use independent rather than global meta lock (#30869) 2024-03-04 16:56:59 +08:00
mock_test.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
mock_trigger_manager.go feat: Add Compaction views and triggers (#27906) 2023-11-23 17:30:25 +08:00
OWNERS
policy_test.go enhance: Remove not in use policies (#29448) 2023-12-28 10:38:46 +08:00
policy.go enhance: Remove not in use policies (#29448) 2023-12-28 10:38:46 +08:00
README.md
segment_allocation_policy_test.go Adjust datacoord for L0 Delta (#28021) 2023-11-06 15:26:16 +08:00
segment_allocation_policy.go fix: Decoupling importing segment from flush process (#30402) 2024-02-03 13:01:12 +08:00
segment_info_test.go fix: starve lock caused by slow GetCompactionTo method when too much segments (#30963) 2024-03-05 10:04:59 +08:00
segment_info.go fix: starve lock caused by slow GetCompactionTo method when too much segments (#30963) 2024-03-05 10:04:59 +08:00
segment_manager_test.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
segment_manager.go fix: add GetSegments optimization to avoid meta mutex competition (#31025) 2024-03-05 14:47:00 +08:00
segment_operator_test.go fix: Prevent clone when selecting segments from meta (#30928) 2024-03-01 13:59:02 +08:00
segment_operator.go fix: Prevent clone when selecting segments from meta (#30928) 2024-03-01 13:59:02 +08:00
server_test.go enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
server.go fix: Trigger l0 compaction when l0 views don't change (#30729) 2024-03-05 16:37:00 +08:00
services_test.go enhance: index meta use independent rather than global meta lock (#30869) 2024-03-04 16:56:59 +08:00
services.go enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
session_manager_test.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
session_manager.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
session.go
util_test.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
util.go enhance: index meta use independent rather than global meta lock (#30869) 2024-03-04 16:56:59 +08:00

README.md

Data Coordinator

Data cooridnator(datacoord for short) is the component to organize DataNodes and segments allocations.

Dependency

  • KV store: a kv store has all the meta info datacoord needs to operate. (etcd)
  • Message stream: a message stream to communicate statistics information with data nodes. (Pulsar)
  • Root Coordinator: timestamp, id and meta source.
  • Data Node(s): could be an instance or a cluster, actual worker group handles data modification operations.