In Milvus, a collection has multiple fields, mainly there are two kinds of fields: vector field and scalar field. We call a row as an entity, one entity encapsulates multiple vectors and scalar values.
QueryNode subscribes to the insert channel and will determine whether to use the data extracted from the insert channel or data processed by DataNode to provide services according to the status of a segment.
When the DataNode processes each inserted entity, it updates the bloomfilter of the Segment to which the entity belongs. If it does not exist, it creates a bloomfilter in memory and updates it.
Once DataNode receives a Flush command from DataCoord, it sorts the data in the segment in ascending order of primary key, records the maximum and minimum values of a primary key, and writes the segment, statistics and bloomfilter to the storage system.
After receiving the Get request from the client, the Proxy sends the request to the `search` channel and waits for the result returned from the `searchResult` channel.
Two new types of Binlog are added: BFBinlog and StatsBinlog.
BFBinlog Payload: Refer to https://github.com/milvus-io/milvus/blob/1.1/core/src/segment/SegmentWriter.h for storage methods
StatsBinlog Payload: Json format string, currently only contains the keys `max`, `min`.
## Impact
### API
- A new Get API
- DropCollection / ReleaseCollection / DropPartition / ReleasePartition requests need to clear the corresponding statistics files and bloomfilter files in the memory
### Storage
- The name of the binlog file has been changed from `${log_idx}` to `_${log_idx}`
- Each binlog adds a stats file
- Each binlog adds a bloomfilter file
## Test Plan
### Testcase 1
In the newly created collection, insert an entity with a primary key of 107, call the Get interface to query the entity with a primary key of 107, and each field of the retrieved entity is exactly the same as the inserted entity.
### Testcase 2
In the newly created collection, insert a record with a primary key of 107, call the Get interface to query the record with a primary key of 106, and the retrieved record is empty.
In the newly created collection, insert the records with the primary keys of 105, 106, 107, call the Get interface to query the records with the primary keys of 101, 102, 103, 104, 105, 106, 107, the retrieved result only contains the records with primary keys of 105, 106, 107.
In the newly created collection, insert a record with a primary key of 107, call the Flush interface, and check whether there are stats and bloomfilter files on MinIO.