milvus/internal
Buqian Zheng 070dfc77bf
feat: [Sparse Float Vector] segcore basics and index building (#30357)
This commit adds sparse float vector support to segcore with the
following:

1. data type enum declarations
2. Adds corresponding data structures for handling sparse float vectors
in various scenarios, including:
* FieldData as a bridge between the binlog and the in memory data
structures
* mmap::Column as the in memory representation of a sparse float vector
column of a sealed segment;
* ConcurrentVector as the in memory representation of a sparse float
vector of a growing segment which supports inserts.
3. Adds logic in payload reader/writer to serialize/deserialize from/to
binlog
4. Adds the ability to allow the index node to build sparse float vector
index
5. Adds the ability to allow the query node to build growing index for
growing segment and temp index for sealed segment without index built

This commit also includes some code cleanness, comment improvement, and
some unit tests for sparse vector.

https://github.com/milvus-io/milvus/issues/29419

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-03-11 14:45:02 +08:00
..
allocator enhance:change some logs (#29579) 2024-01-05 16:12:48 +08:00
core feat: [Sparse Float Vector] segcore basics and index building (#30357) 2024-03-11 14:45:02 +08:00
datacoord fix: alter index request's index name can not be empty (#31128) 2024-03-11 13:05:02 +08:00
datanode enhance: Add segment id short cut for WithSegmentID filter (#31144) 2024-03-11 10:55:02 +08:00
distributed enhance: Add ListIndexes API from datacoord (#31104) 2024-03-07 17:37:01 +08:00
http enhance: Support to get the param value in the runtime (#29297) 2023-12-22 18:36:44 +08:00
indexnode enhance: add graceful stop timeout to avoid node stop hang under extreme cases (#30317) 2024-02-29 17:01:50 +08:00
kv enhance: disallow tikv/rawkv usages (#30027) 2024-02-06 10:41:41 +08:00
metastore enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
mocks enhance: Add ListIndexes API from datacoord (#31104) 2024-03-07 17:37:01 +08:00
mq fix: close rocksmq and natsmq when shutdown (#30706) 2024-02-26 11:38:55 +08:00
parser/planparserv2 enhance: Avoid create schema helper for each read task (#30981) 2024-03-04 19:39:00 +08:00
proto enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941) 2024-03-07 20:39:02 +08:00
proxy feat: Replacing the current import API with the v2 implementation (#31046) 2024-03-10 12:23:02 +08:00
querycoordv2 enhance: change frequency log to rated level (#31084) 2024-03-08 16:39:02 +08:00
querynodev2 enhance: add metrics for load segment progress (#31005) 2024-03-08 15:39:02 +08:00
registry Add querynode client wrapper and avoid grpc in standalone mode (#27781) 2023-10-19 11:10:07 +08:00
rootcoord enhance: add graceful stop timeout to avoid node stop hang under extreme cases (#30317) 2024-02-29 17:01:50 +08:00
storage enhance: Enable binlog deserialize reader in datanode compaction (#31036) 2024-03-08 18:25:02 +08:00
tso Format the code (#27275) 2023-09-21 09:45:27 +08:00
types feat: Revise the RESTful bulk insert API (#29698) 2024-03-05 15:03:00 +08:00
util enhance: support stream call for grpc client (#30013) 2024-03-07 17:45:01 +08:00