milvus/internal/storage
congqixia e677af19b0
enhance: Add PrimaryKeys interface to reduce memory usage (#30405)
See also #30404

`PrimaryKey` is used to hold pk values for both int64 & varchar data
type. Since it is an interface it may occupies more memory than pure
slices when holding a group of pks.

This PR add `PrimaryKeys` interface when some other module need to hold
lots of PrimaryKeys.
By using this interface, it could reduce the memory of pk slice to half
when using Int64 Pk data type and reduce interface cost for each row of
varchar as well.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-01 09:57:11 +08:00
..
aliyun Identify service providers based on addresses (#27907) 2023-10-25 17:28:10 +08:00
gcp Format the code (#27275) 2023-09-21 09:45:27 +08:00
tencent feat: Support tencent cloud object storage for milvus (#30163) 2024-01-23 11:28:56 +08:00
azure_object_storage_test.go fix: Fix bug for read data from azure (#30007) 2024-01-22 15:44:54 +08:00
azure_object_storage.go fix: Fix bug for read data from azure (#30007) 2024-01-22 15:44:54 +08:00
binlog_iterator_test.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
binlog_iterator.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
binlog_reader.go Move some modules from internal to public package (#22572) 2023-04-06 19:14:32 +08:00
binlog_test.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
binlog_util_test.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
binlog_util.go Move some modules from internal to public package (#22572) 2023-04-06 19:14:32 +08:00
binlog_writer_test.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
binlog_writer.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
data_codec_test.go Add float16 approve for multi-type part (#28427) 2024-01-11 15:48:51 +08:00
data_codec.go Add float16 approve for multi-type part (#28427) 2024-01-11 15:48:51 +08:00
data_sorter_test.go Add float16 approve for multi-type part (#28427) 2024-01-11 15:48:51 +08:00
data_sorter.go Add float16 approve for multi-type part (#28427) 2024-01-11 15:48:51 +08:00
event_data.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
event_header.go Move some modules from internal to public package (#22572) 2023-04-06 19:14:32 +08:00
event_reader.go Use go-api/v2 for milvus-proto (#24770) 2023-06-09 01:28:37 +08:00
event_test.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
event_writer_test.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
event_writer.go Add go payload writer (#24656) (#24762) 2023-06-09 13:52:39 +08:00
factory.go Use OpenDAL to access object store (#25642) 2023-11-01 09:00:14 +08:00
index_data_codec_test.go Check error by Error() and NoError() for better report message (#24736) 2023-06-08 15:36:36 +08:00
index_data_codec.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
insert_data_test.go Add float16 approve for multi-type part (#28427) 2024-01-11 15:48:51 +08:00
insert_data.go feat: Add import executor and manager for datanode (#29438) 2024-01-31 20:45:04 +08:00
local_chunk_manager_test.go enhance: Remove vector chunk manager (#28569) 2023-11-30 18:00:33 +08:00
local_chunk_manager.go Refine chunk manager errors (#27590) 2023-10-31 12:18:15 +08:00
minio_chunk_manager_test.go enhance: use random root path for minio unit tests (#29753) 2024-01-08 15:58:48 +08:00
minio_chunk_manager.go fix: Fix minio latency monitoring for get operation (#28510) 2023-11-28 10:00:27 +08:00
minio_object_storage_test.go fix: Align minio object storage ut to new minio server behavior (#29014) 2023-12-06 15:42:43 +08:00
minio_object_storage.go feat: Support tencent cloud object storage for milvus (#30163) 2024-01-23 11:28:56 +08:00
options.go Add chunk manager request timeout (#27692) 2023-10-23 20:08:08 +08:00
OWNERS [skip ci]Update OWNERS files (#11898) 2021-11-16 15:41:11 +08:00
payload_reader_test.go Update arrow version to v12 (#28425) 2023-11-15 10:36:19 +08:00
payload_reader.go Add float16 approve for multi-type part (#28427) 2024-01-11 15:48:51 +08:00
payload_test.go Add float16 approve for multi-type part (#28427) 2024-01-11 15:48:51 +08:00
payload_writer.go Add float16 approve for multi-type part (#28427) 2024-01-11 15:48:51 +08:00
payload.go Add float16 approve for multi-type part (#28427) 2024-01-11 15:48:51 +08:00
pk_statistics.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
primary_key_test.go Use go-api/v2 for milvus-proto (#24770) 2023-06-09 01:28:37 +08:00
primary_key.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
primary_keys_test.go enhance: Add PrimaryKeys interface to reduce memory usage (#30405) 2024-02-01 09:57:11 +08:00
primary_keys.go enhance: Add PrimaryKeys interface to reduce memory usage (#30405) 2024-02-01 09:57:11 +08:00
print_binlog_test.go Remove deprecated io/ioutil usage (#27747) 2023-10-17 20:32:09 +08:00
print_binlog.go Format the code (#27275) 2023-09-21 09:45:27 +08:00
remote_chunk_manager_test.go fix: Fix bug for read data from azure (#30007) 2024-01-22 15:44:54 +08:00
remote_chunk_manager.go feat: Support tencent cloud object storage for milvus (#30163) 2024-01-23 11:28:56 +08:00
stats_test.go Add retry time when lazy load BF (#25096) 2023-06-25 11:32:43 +08:00
stats.go enhance: add param for bloomfilter(#29388) (#29490) 2023-12-28 18:10:46 +08:00
storage_test.go enhance: Remove vector chunk manager (#28569) 2023-11-30 18:00:33 +08:00
types.go enhance: Support importing data with parquet file (#28608) 2023-11-29 20:52:27 +08:00
unsafe_test.go [skip e2e]Update license for storage unsafe (#14452) 2021-12-28 20:03:56 +08:00
unsafe.go [skip e2e]Update license for storage unsafe (#14452) 2021-12-28 20:03:56 +08:00
utils_test.go Add float16 approve for multi-type part (#28427) 2024-01-11 15:48:51 +08:00
utils.go Add float16 approve for multi-type part (#28427) 2024-01-11 15:48:51 +08:00