Adding a collection id to the index node log allows you to associate an
index building task with a specific collection.
If the host CPU usage is too high due to index build, you can use the
collection id to quickly locate a specific collection, improving fault
locating efficiency.
Signed-off-by: dengxiaohai <rolkdengxiaohai@didiglobal.com>
Co-authored-by: dengxiaohai <rolkdengxiaohai@didiglobal.com>
related to #32165
1. for all the manager, support collection level index
2. remove collection level filter to avoid extra cpu usage when
collection number increases
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
issue: #19095,#29655,#31718
- Change `ListWithPrefix` to `WalkWithPrefix` of OOS into a pipeline
mode.
- File garbage collection is performed in other goroutine.
- Segment Index Recycle clean index file too.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #30647
- Remove error report if there's no query node serve. It's hard for
programer to use it to do resource management.
- Change resource group `transferNode` logic to keep compatible with old
version sdk.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
See also #31468
Balance test suite may assert segment number based on test setup.
However the compaction may reduce the number and cause test cases
unstable.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #29419
added helper functions to parse JSON representation of sparse float
vectors, will be used by both the restful server and the import utils.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
See also #32165
Add segment dist and leader view filter criterion struct to store
frequent filter conditions.
Add collection/channel filter results for these two meta
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
`applyDelete` used to be serial for delete entries on each segments.
This PR make it work in parallel with errgroup to improve performance
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
See also #32165
Change `SegmentDistFilter` to interface in order to provde node index
when filter segment dist.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
See also #32403
`fg_buffer_size` was decreased after metrics removed in flowgraph
ddnode, which make metrics value negative.
This PR move remove metrics logic into `dataSyncService.Close`
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
See also #32165
Change `LeaderViewFilter` to interface to provided map key to avoid
iterating all key-values in LeaderViewManager
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
See also #32440
- Add loadTask in collection observer
- For load collection/partitions, load task shall timeout as a whole
- Change related constructor to load jobs
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to https://github.com/milvus-io/milvus/issues/32165
1. nodeid based channel store access should use map access instead of
iteration.
2. The join-ish functions calls are slow when # collections/segments
increases (e.g. 10k).
e.g.
getNumRowsOfCollectionUnsafe is O(num_segments); GetAllCollectionNumRows
is of O(num_collections*num_segments).
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
