test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-03 12:29:36 +08:00
Code Issues Actions 13 Packages Projects Releases Wiki Activity
4e264003bf
milvus/internal/datacoord
History
yihao.dai 4e264003bf
enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629) 
Feature Introduced:
1. Ensure ImportV2 waits for the index to be built

Enhancements Introduced:
1. Utilization of local time for timeout ts instead of allocating ts
from rootcoord.
3. Enhanced input file length check for binlog import.
4. Removal of duplicated manager in datanode.
5. Renaming of executor to scheduler in datanode.
6. Utilization of a thread pool in the scheduler in datanode.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-04-01 20:09:13 +08:00
..
broker enhance: index meta use independent rather than global meta lock (#30869) 2024-03-04 16:56:59 +08:00
allocator_test.go Decoupling client and server API in types interface (#27186) 2023-09-26 09:57:25 +08:00
allocator.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
build_index_policy.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
channel_checker_test.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
channel_checker.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
channel_manager_factory.go enhance: Remove not in use policies (#29448) 2023-12-28 10:38:46 +08:00
channel_manager_test.go enhance: Change ChannelManager to interface (#29300) 2023-12-25 19:24:46 +08:00
channel_manager.go enhance:change some logs (#29579) 2024-01-05 16:12:48 +08:00
channel_store_test.go Change channel to Interface (#27839) 2023-11-13 11:16:18 +08:00
channel_store.go Change channel to Interface (#27839) 2023-11-13 11:16:18 +08:00
channel.go Change channel to Interface (#27839) 2023-11-13 11:16:18 +08:00
cluster_test.go enhance: Remove import v1 (#31403) 2024-03-22 15:29:09 +08:00
cluster.go enhance: Remove import v1 (#31403) 2024-03-22 15:29:09 +08:00
compaction_l0_view_test.go enhance: Add force trigger (#30641) 2024-02-19 18:40:50 +08:00
compaction_l0_view.go fix: Trigger l0 compaction when l0 views don't change (#30729) 2024-03-05 16:37:00 +08:00
compaction_scheduler_test.go enhance: Add CompactionTaskNum metrics (#29518) 2023-12-28 15:46:55 +08:00
compaction_scheduler.go fix: Trigger l0 compaction when l0 views don't change (#30729) 2024-03-05 16:37:00 +08:00
compaction_test.go enhance: Skip submit empty l0 tasks in DC (#31280) 2024-03-19 10:13:14 +08:00
compaction_trigger_test.go enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
compaction_trigger_v2_test.go fix: Skip to submit l0 tasks when scheduler full (#31270) 2024-03-15 10:21:12 +08:00
compaction_trigger_v2.go fix: Skip to submit l0 tasks when scheduler full (#31270) 2024-03-15 10:21:12 +08:00
compaction_trigger.go feat: Add global mmap enable configuration (#31267) 2024-03-18 15:17:10 +08:00
compaction_view_manager_test.go fix: Correct the last empty l0 views (#31198) 2024-03-14 10:31:04 +08:00
compaction_view_manager.go fix: Correct the last empty l0 views (#31198) 2024-03-14 10:31:04 +08:00
compaction_view.go fix: Trigger l0 compaction when l0 views don't change (#30729) 2024-03-05 16:37:00 +08:00
compaction.go enhance: Rename update segment operator (#31121) 2024-03-20 17:53:14 +08:00
const.go enhance: pass partition key scalar info if enabled when build vector index (#29931) 2024-01-24 00:04:55 +08:00
errors_test.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
errors.go Refine DataCoord status (#27262) 2023-09-26 17:15:27 +08:00
garbage_collector_test.go enhance: Remove import v1 (#31403) 2024-03-22 15:29:09 +08:00
garbage_collector.go fix: Use correct ticker for gc work defer (#31456) 2024-03-20 22:17:06 +08:00
handler.go enhance: Add AllPartitionsID const to replace InvalidPartitionID (#31438) 2024-03-20 19:01:05 +08:00
import_checker_test.go enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629) 2024-04-01 20:09:13 +08:00
import_checker.go enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629) 2024-04-01 20:09:13 +08:00
import_job.go enhance: Return more fields in import progress response (#31539) 2024-03-24 21:57:06 +08:00
import_meta_test.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
import_meta.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
import_scheduler_test.go enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629) 2024-04-01 20:09:13 +08:00
import_scheduler.go enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629) 2024-04-01 20:09:13 +08:00
import_task.go enhance: Enhance and modify the return content of ImportV2 (#31192) 2024-03-13 19:51:03 +08:00
import_util_test.go enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629) 2024-04-01 20:09:13 +08:00
import_util.go enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629) 2024-04-01 20:09:13 +08:00
index_builder_test.go enhance: add materialized view search info (#30888) 2024-03-21 11:19:07 +08:00
index_builder.go feat: add more operation detail info for better allocation (#30438) 2024-03-28 06:33:11 +08:00
index_engine_version_manager_test.go Optimize the codec code of session (#27360) 2023-10-01 10:33:30 +08:00
index_engine_version_manager.go feat: trigger compaction to handle index version (#28442) 2023-11-21 09:26:22 +08:00
index_meta_test.go enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629) 2024-04-01 20:09:13 +08:00
index_meta.go enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629) 2024-04-01 20:09:13 +08:00
index_service_test.go enhance: Add ListIndexes API from datacoord (#31104) 2024-03-07 17:37:01 +08:00
index_service.go feat: Add global mmap enable configuration (#31267) 2024-03-18 15:17:10 +08:00
indexnode_manager_test.go Refine state check (#27541) 2023-10-11 21:01:35 +08:00
indexnode_manager.go Decoupling client and server API in types interface (#27186) 2023-09-26 09:57:25 +08:00
meta_test.go enhance: Remove import v1 (#31403) 2024-03-22 15:29:09 +08:00
meta_util.go Make VChannelInfo read compatible with 2.0.2 (#17995) 2022-07-07 21:58:22 +08:00
meta.go enhance: Remove import v1 (#31403) 2024-03-22 15:29:09 +08:00
metrics_info_test.go Refine state check (#27541) 2023-10-11 21:01:35 +08:00
metrics_info.go feat: add more operation detail info for better allocation (#30438) 2024-03-28 06:33:11 +08:00
mock_allocator_test.go enhance: index meta use independent rather than global meta lock (#30869) 2024-03-04 16:56:59 +08:00
mock_channel_store.go Change channel to Interface (#27839) 2023-11-13 11:16:18 +08:00
mock_channelmanager.go enhance: Add CompactionTaskNum metrics (#29518) 2023-12-28 15:46:55 +08:00
mock_cluster.go fix: add lazy load field to mark segment load type (#31591) 2024-03-28 11:23:10 +08:00
mock_compaction_meta.go enhance: Refine compaction interfaces to support major compaction (#30632) 2024-02-19 20:52:50 +08:00
mock_compaction_plan_context.go fix: Separate schedule and check results loop (#28692) 2023-11-29 10:50:29 +08:00
mock_handler.go Change channel to Interface (#27839) 2023-11-13 11:16:18 +08:00
mock_index_engine_version_manager.go feat: trigger compaction to handle index version (#28442) 2023-11-21 09:26:22 +08:00
mock_scheduler.go enhance: Add CompactionTaskNum metrics (#29518) 2023-12-28 15:46:55 +08:00
mock_segment_manager.go enhance: index meta use independent rather than global meta lock (#30869) 2024-03-04 16:56:59 +08:00
mock_session_manager.go fix: add lazy load field to mark segment load type (#31591) 2024-03-28 11:23:10 +08:00
mock_test.go enhance: Remove import v1 (#31403) 2024-03-22 15:29:09 +08:00
mock_trigger_manager.go feat: Add Compaction views and triggers (#27906) 2023-11-23 17:30:25 +08:00
OWNERS [skip ci]Update OWNERS files (#11898) 2021-11-16 15:41:11 +08:00
policy_test.go enhance: Remove not in use policies (#29448) 2023-12-28 10:38:46 +08:00
policy.go enhance: Remove not in use policies (#29448) 2023-12-28 10:38:46 +08:00
README.md [skip ci]Change etcd to lowercase (#9983) 2021-10-15 18:58:37 +08:00
segment_allocation_policy_test.go Adjust datacoord for L0 Delta (#28021) 2023-11-06 15:26:16 +08:00
segment_allocation_policy.go fix: Decoupling importing segment from flush process (#30402) 2024-02-03 13:01:12 +08:00
segment_info_test.go fix: starve lock caused by slow GetCompactionTo method when too much segments (#30963) 2024-03-05 10:04:59 +08:00
segment_info.go enhance: Remove import v1 (#31403) 2024-03-22 15:29:09 +08:00
segment_manager_test.go feat: add more operation detail info for better allocation (#30438) 2024-03-28 06:33:11 +08:00
segment_manager.go enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629) 2024-04-01 20:09:13 +08:00
segment_operator_test.go fix: Prevent clone when selecting segments from meta (#30928) 2024-03-01 13:59:02 +08:00
segment_operator.go fix: Prevent clone when selecting segments from meta (#30928) 2024-03-01 13:59:02 +08:00
server_test.go feat: add more operation detail info for better allocation (#30438) 2024-03-28 06:33:11 +08:00
server.go enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629) 2024-04-01 20:09:13 +08:00
services_test.go enhance: Remove import v1 (#31403) 2024-03-22 15:29:09 +08:00
services.go enhance: Ensure ImportV2 waits for the index to be built and refine some logic (#31629) 2024-04-01 20:09:13 +08:00
session_manager_test.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
session_manager.go feat: add more operation detail info for better allocation (#30438) 2024-03-28 06:33:11 +08:00
session.go Decoupling client and server API in types interface (#27186) 2023-09-26 09:57:25 +08:00
util_test.go feat: Add import scheduler and manager (#29367) 2024-03-01 18:31:02 +08:00
util.go fix: Avoid acquire index meta's lock for each segment (#31723) 2024-04-01 15:49:13 +08:00

README.md

Data Coordinator

Data cooridnator(datacoord for short) is the component to organize DataNodes and segments allocations.

Dependency

  • KV store: a kv store has all the meta info datacoord needs to operate. (etcd)
  • Message stream: a message stream to communicate statistics information with data nodes. (Pulsar)
  • Root Coordinator: timestamp, id and meta source.
  • Data Node(s): could be an instance or a cluster, actual worker group handles data modification operations.