See also #32748
This PR:
- Add `metautil.Channel` utiltiy which convert virtual name to physical
channel name, collectionID and shard idx
- Add channel mapper interface & implementation to convert limited
physical channel name into int index
- Apply `metautil.Channel` filter in querynode segment manager logic
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
issue: #31479#31797
milvus will add released segment to excluded info, and filter out it's
stream data in filter_node. but for data buffered in insert_node's
channel, if it belongs to growing segment which already be released,
then it will all the growing segment back again.
This PR maintain `excluded segments` in delegator, and check excluded
segment before new growing segment.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Related to #27349
See also #30816
Level zero is not allowed to balance among delegators, they shall always
serve current delegator. This PR releases all level zero segments after
channel is unsubscribed and preventing level zero segment blocking
graceful stop.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
add sparse float vector support to different milvus components,
including proxy, data node to receive and write sparse float vectors to
binlog, query node to handle search requests, index node to build index
for sparse float column, etc.
https://github.com/milvus-io/milvus/issues/29419
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
issue: #31109
This PR remove duplicate target node id check, due to server id has
already been checked in rpc's interceptor
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
When the TimeTravel functionality was previously removed, it
inadvertently affected the MVCC functionality within the system. This PR
aims to reintroduce the internal MVCC functionality as follows:
1. Add MvccTimestamp to the requests of Search/Query and the results of
Search internally.
2. When the delegator receives a Query/Search request and there is no
MVCC timestamp set in the request, set the delegator's current tsafe as
the MVCC timestamp of the request. If the request already has an MVCC
timestamp, do not modify it.
3. When the Proxy handles Search and triggers the second phase ReQuery,
divide the ReQuery into different shards and pass the MVCC timestamp to
the corresponding Query requests.
issue: #29656
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
See also #27515
When Delegator processes delete data, it forwards delete data with only
segment id specified. When two segments has same segment id but one is
growing and the other is sealed, the delete will be applied to both
segments which causes delete data out of order when concurrent load
segment occurs.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
We have implemented the chunkcache (in cpp) to retrieve vectors, hence
rendering the vectorchunkcache (in golang) obsolete.
issue: https://github.com/milvus-io/milvus/issues/28568
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
issue: #27678
when interimIndex = true, memory predict should be update with the
memory usage of binlog index build process.
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
See also #28022#28034
The load segment may reaches before watch dml channel, so the index meta
may be empty as well
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Since Milvus in lower version (< 2.3.0), there is no load info in set action
which may corrupt data integrity and cause panicking
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Fix logic error from #26926
function `waitSegmentLoadDone` shall return error when context is done
Make delegator control concurrency for each same segment
Related to #26908
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
