test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-11-30 02:48:45 +08:00
Code Issues Actions 11 Packages Projects Releases Wiki Activity
17,905 Commits 94 Branches 105 Tags 184 MiB
548c82eca5
Commit Graph

17905 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
bjzhjing
548c82eca5
Refactor storage.MergeInsertData() to optimize the merging process (#26839) 
Benchmark Milvus with https://github.com/qdrant/vector-db-benchmark and
specify the datasets as 'deep-image-96-angular'. Meanwhile, do perf
profiling during 'upload + index' stage of vector-db-benchmark and see
the following hot spots.

39.59%--github.com/milvus-io/milvus/internal/storage.MergeInsertData
        |
        |--21.43%--github.com/milvus-io/milvus/internal/storage.MergeFieldData
        |          |
        |          |--17.22%--runtime.memmove
        |                     |
        |                     |--1.53%--asm_exc_page_fault
        |                     ......
        |
        |--18.16%--runtime.memmove
                   |
                   |--1.66%--asm_exc_page_fault
                   ......

The hot code path is in storage.MergeInsertData() which updates
buffer.buffer by creating a new 'InsertData' instance and merging both
the old buffer.buffer and addedBuffer into it. When it calls golang
runtime.memmove to move buffer.buffer which is with big size (>1M), the
hot spots appear.

To avoid the above overhead, update storage.MergeInsertData() by
appending addedBuffer to buffer.buffer, instead of moving buffer.buffer
and addedBuffer to a new 'InsertData'. This change removes the hot spots
'runtime.memmove' from perf profiling output. Additionally, the 'upload
+ index' time, which is one performance metric of vector-db-benchmark,
is reduced around 60% with this change.

Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
 2023-09-05 21:41:48 +08:00
congqixia
9ed5841730
Use specified mockery & golangci-lint version (#26845) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-05 17:39:48 +08:00
Enwei Jiao
73f4347ac6
Fix code checker error (#26844) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-05 15:51:48 +08:00
zhuwenxing
1a80380a6a
[skip e2e]Add wait time when verifying the chaos is deleted (#26847) 
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2023-09-05 14:33:49 +08:00
congqixia
fe7f7ea237
Add cpp-build and setenv for querynode mockery regen command (#26843) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-05 14:13:49 +08:00
congqixia
901e98a868
Update CONTRIBUTING.md for mockery generation (#26842) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-05 14:11:54 +08:00
Enwei Jiao
fb0705df1b
Decouple basetable and componentparam (#26725) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-05 10:31:48 +08:00
nico
f0c911afc2
add case about diskann and binary index (#26804) 
Signed-off-by: nico <cheng.yuan@zilliz.com>
 2023-09-05 10:23:48 +08:00
zhuwenxing
68a2940b66
[test]Refine chaos apply (#26823) 
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2023-09-05 10:17:48 +08:00
congqixia
4b58c71908
Add ctx parameter for organizeTask and GetWorker method (#26835) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-05 10:05:48 +08:00
sre-ci-robot
c132c53b1a
Update knowhere commit (#26840) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-09-05 09:05:49 +08:00
congqixia
1a8cf5c415
Organize all mockery generation commands in Makefile (#26826) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-04 21:19:48 +08:00
wei liu
1097776477
stop heartbeat if reach heartbeat limit (#26728) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-04 17:51:48 +08:00
cai.zhang
622077f9ad
Refine error message for json length exceed max length (#26676) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-09-04 17:47:48 +08:00
XuanYang-cn
b2e7cbdf4b
Remove TimeTravel in compactor (#26785) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-04 17:41:48 +08:00
cai.zhang
d7cd1f2a6d
Fix bug for get index state with compaction segment (#26822) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-09-04 17:29:48 +08:00
zhagnlu
7056e9c0f7
Increase minio log level to avoid unnecessary log (#26776) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-09-04 17:25:48 +08:00
yihao.dai
4340cbfba2
Merge syncCPLagTooBehind policy into syncPeriodically policy (#26713) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-04 14:37:47 +08:00
chyezh
0530fd80c9
[Fixup] remove nats from default (#26791) 
Signed-off-by: chyezh <ye.zhen@zilliz.com>
 2023-09-04 10:01:04 +08:00
yihao.dai
7624c2b949
Improve rated log in insertBufferNode (#26788) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-04 09:59:10 +08:00
yah01
3349db4aa7
Refine errors to remove changes breaking design (#26521) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-04 09:57:09 +08:00
yihao.dai
c6024a32f5
Add rate limit ratio monitoring (#26709) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-03 20:47:02 +08:00
MrPresent-Han
a34a9d606c
fix panic due to empty traceID(#26754) (#26808) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-09-02 16:13:01 +08:00
yah01
941a383019
Fix failed to load collection with more than 128 partitions (#26763) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-02 00:09:01 +08:00
yihao.dai
64cf5eab18
Accelerate flush in flushAll (#26769) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-01 15:23:01 +08:00
congqixia
e8f1b1736e
Remove log.Error(err.error())-style log (#26783) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-01 13:09:01 +08:00
wei liu
5602b22531
refine checker code style (#26759) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-01 11:57:01 +08:00
wei liu
949c320185
remove pull target from qc recover (#26775) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-01 11:17:01 +08:00
smellthemoon
e56b0018e4
Remove merr mask logic to make it ok to compare resp.status with origin error (#26773) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-09-01 11:09:00 +08:00
XuanYang-cn
ef75784715
Fix LoadSegmentLatency metric p99 (#26761) 
See also: #26743

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-01 11:07:07 +08:00
zhuwenxing
b3de99e336
[test]Add method to analyze chaos test result (#26724) 
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2023-09-01 10:31:01 +08:00
yah01
09218bfd3d
Optimize loading by reduce 1x copy while reading data (#26746) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-01 10:15:01 +08:00
sre-ci-robot
b47da91f3c
Update knowhere commit (#26792) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-09-01 09:23:01 +08:00
Enwei Jiao
d206ebf220
Change default config files (#26780) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-01 09:19:00 +08:00
sre-ci-robot
b8d472e479 Update all contributors 
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
 2023-09-01 00:01:22 +00:00
Cai Yudong
8dc16b599b
Add binary metric types SUBSTRUCTURE/SUPERSTRUCTURE back (#26766) 
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
 2023-08-31 20:07:00 +08:00
nico
aad3d47a06
update test cases (#26771) 
Signed-off-by: nico <cheng.yuan@zilliz.com>
 2023-08-31 18:55:00 +08:00
XuanYang-cn
86e9f1818f
Change default compaction timeout to 15mins (#26757) 
See also: #26566

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-31 18:07:00 +08:00
cai.zhang
82b0e39abe
Fix bug for concurrent parsing expr with strings (#26721) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-08-31 17:39:02 +08:00
XuanYang-cn
8d54509e54
Fix CompactionLatency metrics (#26747) 
- Refine compactor logs

See also: #26743

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-31 17:35:03 +08:00
yah01
9004601817
Enable vector index mmap (#26750) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-31 17:31:11 +08:00
jaime
d001133bd2
[skip e2e] Refine collection not exists error (#26707) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-08-31 17:29:03 +08:00
congqixia
4bf26dd4ed
Refine standalone components stop order (#26742) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-31 16:51:02 +08:00
sre-ci-robot
3752ebb4c9
Update Pytest image changes (#26730) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-08-31 13:57:01 +08:00
MrPresent-Han
8330c18dc9
add log for loading segment(#26564) (#26640) 
/kind improvement

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-31 12:03:00 +08:00
Xu Tong
5f08e3a72b
Fix field_name spell error (#26718) 
Signed-off-by: Writer-X <1256866856@qq.com>
 2023-08-31 11:39:02 +08:00
sre-ci-robot
f468a64019
Update Builder image changes (#26710) 
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2023-08-31 11:21:01 +08:00
Jiquan Long
36e29ef859
Auto generate parser code (#26716) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-08-31 11:13:01 +08:00
Bennu
f60a335334
Update milvus gpu image tag (#26740) 
Signed-off-by: Bennu-Li <yunmei.li@zilliz.com>
 2023-08-31 10:31:01 +08:00
Gao
7e36f819b4
Make autoindex config clean (#26732) 
Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-08-31 10:25:01 +08:00