test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-03 12:29:36 +08:00
Code Issues Actions 13 Packages Projects Releases Wiki Activity
033eae9e73
milvus/internal/datacoord
History
xige-16 033eae9e73
enhance: Set segment.maxSize param to 1024M (#30139) 
issue: #25639 
/kind improvement

When the number of vector columns increases, the number of rows per
segment will decrease. In order to reduce the impact on vector indexing
performance, it is necessary to increase the segment max limit.

If a collection has multiple vector fields with memory and disk indices
on different vector fields, the size limit after segment compaction is
the minimum of segment.maxSize and segment.diskSegmentMaxSize.

Signed-off-by: xige-16 <xi.ge@zilliz.com>

---------

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-01-29 10:17:02 +08:00
..
broker enhance: Move datacoord broker into separate package (#28876) 2023-12-01 10:22:34 +08:00
allocator_test.go Decoupling client and server API in types interface (#27186) 2023-09-26 09:57:25 +08:00
allocator.go enhance:Remove msgbase unnecessary assignments (#28511) 2023-11-24 15:02:39 +08:00
build_index_policy.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
channel_checker_test.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
channel_checker.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
channel_manager_factory.go enhance: Remove not in use policies (#29448) 2023-12-28 10:38:46 +08:00
channel_manager_test.go enhance: Change ChannelManager to interface (#29300) 2023-12-25 19:24:46 +08:00
channel_manager.go enhance:change some logs (#29579) 2024-01-05 16:12:48 +08:00
channel_store_test.go Change channel to Interface (#27839) 2023-11-13 11:16:18 +08:00
channel_store.go Change channel to Interface (#27839) 2023-11-13 11:16:18 +08:00
channel.go Change channel to Interface (#27839) 2023-11-13 11:16:18 +08:00
cluster_test.go enhance: Change ChannelManager to interface (#29300) 2023-12-25 19:24:46 +08:00
cluster.go enhance: Change ChannelManager to interface (#29300) 2023-12-25 19:24:46 +08:00
compaction_l0_view_test.go enhance:change some logs (#29579) 2024-01-05 16:12:48 +08:00
compaction_l0_view.go fix: Fix some left comments (#29027) 2023-12-13 18:48:39 +08:00
compaction_scheduler_test.go enhance: Add CompactionTaskNum metrics (#29518) 2023-12-28 15:46:55 +08:00
compaction_scheduler.go enhance: Add CompactionTaskNum metrics (#29518) 2023-12-28 15:46:55 +08:00
compaction_test.go fix: compact operation on datacoord meta should preform as a transcation (#29775) 2024-01-26 16:59:00 +08:00
compaction_trigger_test.go enhance: Set segment.maxSize param to 1024M (#30139) 2024-01-29 10:17:02 +08:00
compaction_trigger_v2_test.go fix: Fix L0 compaction in datacoord (#28814) 2023-12-05 18:44:37 +08:00
compaction_trigger_v2.go fix: Fill in info in CompactionSegmentBinlogs (#30279) 2024-01-26 10:51:02 +08:00
compaction_trigger.go enhance: Set segment.maxSize param to 1024M (#30139) 2024-01-29 10:17:02 +08:00
compaction_view_manager_test.go fix: Fix L0 compaction in datacoord (#28814) 2023-12-05 18:44:37 +08:00
compaction_view_manager.go fix: Fix L0 compaction in datacoord (#28814) 2023-12-05 18:44:37 +08:00
compaction_view.go fix: Fix some left comments (#29027) 2023-12-13 18:48:39 +08:00
compaction.go fix: compact operation on datacoord meta should preform as a transcation (#29775) 2024-01-26 16:59:00 +08:00
const.go enhance: pass partition key scalar info if enabled when build vector index (#29931) 2024-01-24 00:04:55 +08:00
errors_test.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
errors.go Refine DataCoord status (#27262) 2023-09-26 17:15:27 +08:00
garbage_collector_test.go enhance:don't store logPath in meta to reduce memory (#28873) 2024-01-18 22:06:31 +08:00
garbage_collector.go enhance:don't store logPath in meta to reduce memory (#28873) 2024-01-18 22:06:31 +08:00
handler.go enhance: Explicitly pass LevelZero segment ids in vchan info (#29612) 2024-01-04 16:46:45 +08:00
index_builder_test.go enhance: pass partition key scalar info if enabled when build vector index (#29931) 2024-01-24 00:04:55 +08:00
index_builder.go enhance: pass partition key scalar info if enabled when build vector index (#29931) 2024-01-24 00:04:55 +08:00
index_engine_version_manager_test.go Optimize the codec code of session (#27360) 2023-10-01 10:33:30 +08:00
index_engine_version_manager.go feat: trigger compaction to handle index version (#28442) 2023-11-21 09:26:22 +08:00
index_meta_test.go feat: Support multiple vector indexes in a collection (#27700) 2023-12-29 11:44:45 +08:00
index_meta.go feat: Support multiple vector indexes in a collection (#27700) 2023-12-29 11:44:45 +08:00
index_service_test.go fix: report error if the altering index doesn't support mmap (#29832) 2024-01-17 16:40:54 +08:00
index_service.go enhance: pass partition key scalar info if enabled when build vector index (#29931) 2024-01-24 00:04:55 +08:00
indexnode_manager_test.go Refine state check (#27541) 2023-10-11 21:01:35 +08:00
indexnode_manager.go Decoupling client and server API in types interface (#27186) 2023-09-26 09:57:25 +08:00
meta_test.go fix: compact operation on datacoord meta should preform as a transcation (#29775) 2024-01-26 16:59:00 +08:00
meta_util.go Make VChannelInfo read compatible with 2.0.2 (#17995) 2022-07-07 21:58:22 +08:00
meta.go fix: compact operation on datacoord meta should preform as a transcation (#29775) 2024-01-26 16:59:00 +08:00
metrics_info_test.go Refine state check (#27541) 2023-10-11 21:01:35 +08:00
metrics_info.go disable auto balance when old node exists (#28191) 2023-11-07 14:02:20 +08:00
mock_allocator_test.go Organize all mockery generation commands in Makefile (#26826) 2023-09-04 21:19:48 +08:00
mock_channel_store.go Change channel to Interface (#27839) 2023-11-13 11:16:18 +08:00
mock_channelmanager.go enhance: Add CompactionTaskNum metrics (#29518) 2023-12-28 15:46:55 +08:00
mock_cluster.go enhance: pack datacoord Cluster and SessionManager with interface and mock them (#28869) 2023-12-11 17:52:37 +08:00
mock_compaction_meta.go fix: compact operation on datacoord meta should preform as a transcation (#29775) 2024-01-26 16:59:00 +08:00
mock_compaction_plan_context.go fix: Separate schedule and check results loop (#28692) 2023-11-29 10:50:29 +08:00
mock_handler.go Change channel to Interface (#27839) 2023-11-13 11:16:18 +08:00
mock_index_engine_version_manager.go feat: trigger compaction to handle index version (#28442) 2023-11-21 09:26:22 +08:00
mock_scheduler.go enhance: Add CompactionTaskNum metrics (#29518) 2023-12-28 15:46:55 +08:00
mock_session_manager.go enhance: Add open telemetry tracing for compaction (#30168) 2024-01-23 10:37:00 +08:00
mock_test.go feat: Implement DescribeAlias and ListAliases interfaces (#29641) 2024-01-11 19:12:51 +08:00
mock_trigger_manager.go feat: Add Compaction views and triggers (#27906) 2023-11-23 17:30:25 +08:00
OWNERS [skip ci]Update OWNERS files (#11898) 2021-11-16 15:41:11 +08:00
policy_test.go enhance: Remove not in use policies (#29448) 2023-12-28 10:38:46 +08:00
policy.go enhance: Remove not in use policies (#29448) 2023-12-28 10:38:46 +08:00
README.md [skip ci]Change etcd to lowercase (#9983) 2021-10-15 18:58:37 +08:00
segment_allocation_policy_test.go Adjust datacoord for L0 Delta (#28021) 2023-11-06 15:26:16 +08:00
segment_allocation_policy.go Adjust datacoord for L0 Delta (#28021) 2023-11-06 15:26:16 +08:00
segment_info.go enhance:don't store logPath in meta to reduce memory (#28873) 2024-01-18 22:06:31 +08:00
segment_manager_test.go Adjust datacoord for L0 Delta (#28021) 2023-11-06 15:26:16 +08:00
segment_manager.go enhance:change some logs (#29579) 2024-01-05 16:12:48 +08:00
server_test.go fix: Use channel cp as the dml&start position for import segments (#30107) 2024-01-22 14:36:55 +08:00
server.go fix: unhealthy datacoord started with unhealthy channel manager (#29848) 2024-01-14 10:36:52 +08:00
services_test.go enhance:don't store logPath in meta to reduce memory (#28873) 2024-01-18 22:06:31 +08:00
services.go fix: Use channel cp as the dml&start position for import segments (#30107) 2024-01-22 14:36:55 +08:00
session_manager_test.go enhance: Change ChannelManager to interface (#29300) 2023-12-25 19:24:46 +08:00
session_manager.go enhance: Add open telemetry tracing for compaction (#30168) 2024-01-23 10:37:00 +08:00
session.go Decoupling client and server API in types interface (#27186) 2023-09-26 09:57:25 +08:00
util_test.go enhance: add l0 metric and fix datacoord no need drop l0 segment when flush (#28373) 2023-11-24 15:58:24 +08:00
util.go fix: Use size bucket for compacted segment size metric (#30028) 2024-01-26 10:53:02 +08:00

README.md

Data Coordinator

Data cooridnator(datacoord for short) is the component to organize DataNodes and segments allocations.

Dependency

  • KV store: a kv store has all the meta info datacoord needs to operate. (etcd)
  • Message stream: a message stream to communicate statistics information with data nodes. (Pulsar)
  • Root Coordinator: timestamp, id and meta source.
  • Data Node(s): could be an instance or a cluster, actual worker group handles data modification operations.