Related to #32598
Use `WithBlock` may fail fast when create etcd client to some invalid
etcd endpoints and make it easier to check problem.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
related to #32165
1. for all the manager, support collection level index
2. remove collection level filter to avoid extra cpu usage when
collection number increases
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
issue: #19095,#29655,#31718
- Change `ListWithPrefix` to `WalkWithPrefix` of OOS into a pipeline
mode.
- File garbage collection is performed in other goroutine.
- Segment Index Recycle clean index file too.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #29419
added helper functions to parse JSON representation of sparse float
vectors, will be used by both the restful server and the import utils.
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
https://github.com/milvus-io/milvus/issues/32321
Issue Description:
Tracing is an important means of identifying bottleneck points in a
system and is crucial for debugging production issues. Milvus(or any DB)
is generally the most downstream system for an user call -- a user call
can originate from UI and pass through multiple components, in
micro-services architecture, before reaching Milvus.
So, when an user experiences a glitch, one would debug the call trace
via logs using a common trace id. As of now, Milvus generates a new
trace id for every call and this request is to make sure client can pass
the trace id which will be used for all the logs across the Milvus
sub-components so that one can fetch logs for a user call across the
components -- including Milvus.
Signed-off-by: Shreesha Srinath Madogaran <smadogaran@salesforce.com>
Co-authored-by: Shreesha Srinath Madogaran <smadogaran@salesforce.com>
issue: #30647
- Add declarative resource group api
- Add config for resource group management
- Resource group recovery enhancement
---------
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #29892
This PR:
1. Move the process of gathering materialized search info to when the
search plan is created, before it goes to each segment, to avoid
repeated work and access the plan node under multi-threaded
circumstances.
2. Enforce the supported MV type to `VARCHAR`
3. Add integration test
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
fix https://github.com/milvus-io/milvus/issues/32059
this pr fix two issues:
offset is not handled correctly without specify a limit
reduceStopForBest doesn't guarantee to return limit result even if there
are more result when there is small segment
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
Use an individual buffer size parameter for imports and set buffer size
to 64MB.
issue: https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
See #31944
There is deadlock in concurrent invocation on `cache.DoWait()`. Suppose
2 callers are calling `DoWait` concurrently, the notification for cache
wait queue may be limited to just 1 caller's occupation instead of them
combined.
To fix this issue, this patch is trying to notify all waiters in queue.
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
issue: #30647
- ReplicaManager manage read only node now, and always do persistent of
node distribution of replica.
- All segment/channel checker using ReplicaManager to get read-only node
or read-write node, but not ResourceManager.
- ReplicaManager promise that only apply unique querynode to one replica
in same collection now (replicas in same collection never hold same
querynode at same time).
- ReplicaManager promise that fairly node count assignment policy if
multi replicas of collection is assigned to one resource group.
- Move some parameters check into ReplicaManager to avoid data race.
- Allow transfer replica to resource group that already load replica of
same collection
- Allow transfer node between resource groups that load replica of same
collection
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Feature Introduced:
1. Ensure ImportV2 waits for the index to be built
Enhancements Introduced:
1. Utilization of local time for timeout ts instead of allocating ts
from rootcoord.
3. Enhanced input file length check for binlog import.
4. Removal of duplicated manager in datanode.
5. Renaming of executor to scheduler in datanode.
6. Utilization of a thread pool in the scheduler in datanode.
issue: https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Fix etcd config source didn't respect auth enabled
Also removed pulsar recoverable error when pulsar return ConsumerBusy.
It could happen that pulsar didn't find the original consumer is dead
and recover takes some time.
fix#31631
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
