Related to #27675
Use `struct{}` instead `error` for sync task future result type to
reduce result size and preventing logci error.
Also change some unused parameter to `_` to suppress lint warning
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
1. Make the dynamic field file optional during numpy import
2. Add integration importing test with dynamic
3. Disallow file of pk when autoID=true during numpy import
issue: https://github.com/milvus-io/milvus/issues/32542
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Found lots of `failed to updateTimeTick` with error `skip
ChannelTimeTickMsg from un-recognized session 1` The reason was etcd
client became singleton and used last root path in multiple cases are
run in one suite.
This PR add close singleton client invocation to fix this problem.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
related issue: #32653
1. reduce test params in search.py
2. skip all the xfail test to save running time
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
Related to #32598
Use `WithBlock` may fail fast when create etcd client to some invalid
etcd endpoints and make it easier to check problem.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Adding a collection id to the index node log allows you to associate an
index building task with a specific collection.
If the host CPU usage is too high due to index build, you can use the
collection id to quickly locate a specific collection, improving fault
locating efficiency.
Signed-off-by: dengxiaohai <rolkdengxiaohai@didiglobal.com>
Co-authored-by: dengxiaohai <rolkdengxiaohai@didiglobal.com>
related to #32165
1. for all the manager, support collection level index
2. remove collection level filter to avoid extra cpu usage when
collection number increases
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
issue: #19095,#29655,#31718
- Change `ListWithPrefix` to `WalkWithPrefix` of OOS into a pipeline
mode.
- File garbage collection is performed in other goroutine.
- Segment Index Recycle clean index file too.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #30647
- Remove error report if there's no query node serve. It's hard for
programer to use it to do resource management.
- Change resource group `transferNode` logic to keep compatible with old
version sdk.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
See also #31468
Balance test suite may assert segment number based on test setup.
However the compaction may reduce the number and cause test cases
unstable.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #29419
added helper functions to parse JSON representation of sparse float
vectors, will be used by both the restful server and the import utils.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
See also #32165
Add segment dist and leader view filter criterion struct to store
frequent filter conditions.
Add collection/channel filter results for these two meta
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
`applyDelete` used to be serial for delete entries on each segments.
This PR make it work in parallel with errgroup to improve performance
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
See also #32165
Change `SegmentDistFilter` to interface in order to provde node index
when filter segment dist.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
See also #32403
`fg_buffer_size` was decreased after metrics removed in flowgraph
ddnode, which make metrics value negative.
This PR move remove metrics logic into `dataSyncService.Close`
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
