See also #32440
- Add loadTask in collection observer
- For load collection/partitions, load task shall timeout as a whole
- Change related constructor to load jobs
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Related to https://github.com/milvus-io/milvus/issues/32165
1. nodeid based channel store access should use map access instead of
iteration.
2. The join-ish functions calls are slow when # collections/segments
increases (e.g. 10k).
e.g.
getNumRowsOfCollectionUnsafe is O(num_segments); GetAllCollectionNumRows
is of O(num_collections*num_segments).
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
See also #31632
Release conc.Pool after usage to clean worker and stop background purge
and ticktock.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #32225#31978#32360
1. v1 can only accept one vector, but v2 accept list of vectors #32225
2. cannot get dbName from AliasReq #31978
3. enhance: [restful v2]support config DB-Name in the http header #32360
---------
Signed-off-by: PowderLi <min.li@zilliz.com>
issue: #30647
- Same dst and src resource group should not be allowed in
`TransferReplica` and `TransferNode`.
- Remove redundant parameter check.
Signed-off-by: chyezh <chyezh@outlook.com>
Related to https://github.com/milvus-io/milvus/issues/32165
Avoid iterating through all replicas/collections if possible. Iteration
is expensive when there are large number of replicas/collections.
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
See also #32165
`ChannelManager.Match` is a frequent operation for datacoord. When the
collection number is large, iteration over all channels will cost lots
of CPU time and time consuming.
This PR change the data structure storing datanode-channel info to map
avoiding this iteration when checking channel existence.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #30361
- Delete may be lost when segment is not data-loaded status in lru
cache. skip filtering to fix it.
- `stats_` and `variable_fields_avg_size_` should be reset when
`ReleaseData`
- Remove repeat load delta log operation in lru.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #32029
lack of logic to clean collection info in datacoord's meta, This PR
clean collection info after drop channel, to avoid collection info leak
in datacoord
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #32165
Add collection to partitionIDs mapping to avoid interation on all
partitions loaded when trying to get all partitions with collection id
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #30647
- Add declarative resource group api
- Add config for resource group management
- Resource group recovery enhancement
---------
Signed-off-by: chyezh <chyezh@outlook.com>
issue: #31176
1. cannot get dbName correctly while describe alias #31978
2. return a valid json string even if the user doesn't have the whole
privileges to describe collection #31635
3. rename IndexParam.IndexConfig to IndexParam.Params
4. FieldSchema.ElementTypeParams, IndexParam.Params can not only accept
string
Signed-off-by: PowderLi <min.li@zilliz.com>
Tidy the following test codes
- Remove channel in newTestServer
- Remove newTestServerWithMeta
- Remove newTestServer2
- Remove testDataCoordBase
- Use the same func for handleTTmsg and handleRPCTTmsg
See also: #31620
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
See also #27675#26177
Make memory check evict memory buffer until memory water level is safe.
Also make `EvictBuffer` wait until sync task done.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Issue: #32129
This PR adds a dynamic SVE detection for ARM CPU families for the bitset
code.
Also, allows the code to be compiled if the compiler does not support
NEON (arm-v7).
Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
