mirror of
https://gitee.com/milvus-io/milvus.git
synced 2024-12-04 04:49:08 +08:00
548c82eca5
Benchmark Milvus with https://github.com/qdrant/vector-db-benchmark and specify the datasets as 'deep-image-96-angular'. Meanwhile, do perf profiling during 'upload + index' stage of vector-db-benchmark and see the following hot spots. 39.59%--github.com/milvus-io/milvus/internal/storage.MergeInsertData | |--21.43%--github.com/milvus-io/milvus/internal/storage.MergeFieldData | | | |--17.22%--runtime.memmove | | | |--1.53%--asm_exc_page_fault | ...... | |--18.16%--runtime.memmove | |--1.66%--asm_exc_page_fault ...... The hot code path is in storage.MergeInsertData() which updates buffer.buffer by creating a new 'InsertData' instance and merging both the old buffer.buffer and addedBuffer into it. When it calls golang runtime.memmove to move buffer.buffer which is with big size (>1M), the hot spots appear. To avoid the above overhead, update storage.MergeInsertData() by appending addedBuffer to buffer.buffer, instead of moving buffer.buffer and addedBuffer to a new 'InsertData'. This change removes the hot spots 'runtime.memmove' from perf profiling output. Additionally, the 'upload + index' time, which is one performance metric of vector-db-benchmark, is reduced around 60% with this change. Signed-off-by: Cathy Zhang <cathy.zhang@intel.com> |
||
|---|---|---|
| .. | ||
| aliyun | ||
| gcp | ||
| binlog_iterator_test.go | ||
| binlog_iterator.go | ||
| binlog_reader.go | ||
| binlog_test.go | ||
| binlog_util_test.go | ||
| binlog_util.go | ||
| binlog_writer_test.go | ||
| binlog_writer.go | ||
| data_codec_test.go | ||
| data_codec.go | ||
| data_sorter_test.go | ||
| data_sorter.go | ||
| event_data.go | ||
| event_header.go | ||
| event_reader.go | ||
| event_test.go | ||
| event_writer_test.go | ||
| event_writer.go | ||
| factory.go | ||
| index_data_codec_test.go | ||
| index_data_codec.go | ||
| local_chunk_manager_test.go | ||
| local_chunk_manager.go | ||
| minio_chunk_manager_test.go | ||
| minio_chunk_manager.go | ||
| options.go | ||
| OWNERS | ||
| payload_reader_test.go | ||
| payload_reader.go | ||
| payload_test.go | ||
| payload_writer.go | ||
| payload.go | ||
| pk_statistics.go | ||
| primary_key_test.go | ||
| primary_key.go | ||
| print_binlog_test.go | ||
| print_binlog.go | ||
| stats_test.go | ||
| stats.go | ||
| types.go | ||
| unsafe_test.go | ||
| unsafe.go | ||
| utils_test.go | ||
| utils.go | ||
| vector_chunk_manager_test.go | ||
| vector_chunk_manager.go | ||