test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-04 12:59:23 +08:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity
17,929 Commits 96 Branches 105 Tags 185 MiB
0901b76732
Commit Graph

779 Commits

Author SHA1 Message Date
yiwangdr
337edc321b
tikv integration (#26246) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-09-07 07:25:14 +08:00
bjzhjing
548c82eca5
Refactor storage.MergeInsertData() to optimize the merging process (#26839) 
Benchmark Milvus with https://github.com/qdrant/vector-db-benchmark and
specify the datasets as 'deep-image-96-angular'. Meanwhile, do perf
profiling during 'upload + index' stage of vector-db-benchmark and see
the following hot spots.

39.59%--github.com/milvus-io/milvus/internal/storage.MergeInsertData
        |
        |--21.43%--github.com/milvus-io/milvus/internal/storage.MergeFieldData
        |          |
        |          |--17.22%--runtime.memmove
        |                     |
        |                     |--1.53%--asm_exc_page_fault
        |                     ......
        |
        |--18.16%--runtime.memmove
                   |
                   |--1.66%--asm_exc_page_fault
                   ......

The hot code path is in storage.MergeInsertData() which updates
buffer.buffer by creating a new 'InsertData' instance and merging both
the old buffer.buffer and addedBuffer into it. When it calls golang
runtime.memmove to move buffer.buffer which is with big size (>1M), the
hot spots appear.

To avoid the above overhead, update storage.MergeInsertData() by
appending addedBuffer to buffer.buffer, instead of moving buffer.buffer
and addedBuffer to a new 'InsertData'. This change removes the hot spots
'runtime.memmove' from perf profiling output. Additionally, the 'upload
+ index' time, which is one performance metric of vector-db-benchmark,
is reduced around 60% with this change.

Signed-off-by: Cathy Zhang <cathy.zhang@intel.com>
 2023-09-05 21:41:48 +08:00
congqixia
fe7f7ea237
Add cpp-build and setenv for querynode mockery regen command (#26843) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-05 14:13:49 +08:00
Enwei Jiao
fb0705df1b
Decouple basetable and componentparam (#26725) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-09-05 10:31:48 +08:00
XuanYang-cn
b2e7cbdf4b
Remove TimeTravel in compactor (#26785) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-09-04 17:41:48 +08:00
yihao.dai
4340cbfba2
Merge syncCPLagTooBehind policy into syncPeriodically policy (#26713) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-04 14:37:47 +08:00
yihao.dai
7624c2b949
Improve rated log in insertBufferNode (#26788) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-04 09:59:10 +08:00
yah01
3349db4aa7
Refine errors to remove changes breaking design (#26521) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-04 09:57:09 +08:00
congqixia
e8f1b1736e
Remove log.Error(err.error())-style log (#26783) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-01 13:09:01 +08:00
XuanYang-cn
8d54509e54
Fix CompactionLatency metrics (#26747) 
- Refine compactor logs

See also: #26743

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-31 17:35:03 +08:00
smellthemoon
6069a7d42b
Log out memory in MB (#26262) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-08-30 17:13:00 +08:00
congqixia
cc584551fb
Remove unused hash value setting in DDNode (#26575) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-28 11:30:26 +08:00
SimFG
b9bc6681ae
Use atomic.bool for the isWatchFailed param in the tickler (#26558) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-08-23 13:22:27 +08:00
XuanYang-cn
27691e843c
Fix channel checkpoint stuck (#26534) 
See also: #23621

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-22 18:10:22 +08:00
XuanYang-cn
08fd28b30b
Only do gracefully stop when DN Stop (#26399) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-21 11:16:20 +08:00
SimFG
3be4ac4022
Fix datanode/datacoord continuous restart (#26470) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-08-20 21:20:24 +08:00
Enwei Jiao
533f0ddf6d
Add amazonlinux image, prepare to remove Centos7 image, remove openblas image (#26368) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-08-18 18:32:19 +08:00
congqixia
fbb5d32cb6
Make write binlog in parallel (#26325) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-17 15:44:18 +08:00
congqixia
f371ec0316
Add data sync service close log for watch stuck debugging (#26332) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-16 10:57:32 +08:00
Enwei Jiao
78bc688d16
Remove QueryMsgStream in MqFactory interface (#26374) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-08-16 10:29:33 +08:00
congqixia
f8bcf60e55
Adjust some confusing Warning log to INFO (#26356) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-15 17:49:33 +08:00
wayblink
21eeb37ce9
Save binlog timestampFrom, timestampTo meta when compact (#26203) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-08-08 21:17:21 +08:00
congqixia
b9850ce5c0
Fix copylocks linter errors (#26217) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-08 20:41:15 +08:00
MrPresent-Han
b84f5c560a
refine log out for datanode(#25763) (#26195) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-08 16:49:08 +08:00
MrPresent-Han
fb933fe64d
fix sync memory policy lose effect when inserting too many partitions(#25763) (#26154) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-07 09:59:07 +08:00
XuanYang-cn
830f06783a
Reduce log volumes of DC&DN (#26060) 
See also: #26057

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-02 17:13:11 +08:00
XuanYang-cn
84253f255e
Fix datanode graceful stop panic (#25932) 
See also: #25925

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-07-28 10:11:08 +08:00
groot
a6808e6484
Fix bulkinsert row count error (#25869) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-07-25 11:29:00 +08:00
cai.zhang
779d677eb2
Remove lock for compaction sync segments (#25711) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-07-24 17:25:00 +08:00
jaime
8e5d6193f9
Add a timeout config for bulkinsert request (#25789) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-07-24 14:23:00 +08:00
congqixia
3c503afe7c
Use typeutil.ConcurrentMap instead of sync.Map (#25846) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-24 10:23:01 +08:00
congqixia
31b40b3ccc
Use typeutil.ConcurrentMap instead of sync.Map in fg manager (#25835) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-24 10:21:07 +08:00
Enwei Jiao
66fdc71479
Refactor logs in DataCoord & DataNode (#25574) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-07-14 15:56:31 +08:00
yiwangdr
b9189b9f41
Organize mocks from types.go (#25466) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-07-14 10:12:31 +08:00
wayblink
53b4b70316
Reduce log frequency in timetick_sender (#25532) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-07-13 14:38:29 +08:00
wei liu
012fd1152a
fix datacoord consume datanode tt metrics (#25524) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-13 14:26:29 +08:00
congqixia
dbfade77c5
Retry on SyncSegment failure (#25540) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-13 14:12:29 +08:00
wayblink
ed3e4b0bc5
Fix unstable ut in timetick_sender_test (#25539) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-07-13 10:04:29 +08:00
MrPresent-Han
ea3817fbbc
refinement towards sync_cp_lag_too_behind_policy to avoid submit syncTasks too frequently(#25441) (#25442) (#25500) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-07-12 20:22:29 +08:00
SimFG
f9e2d00f91
Prevent exclusive consumer exception in pulsar (#25376) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-07-12 17:26:30 +08:00
wayblink
fc12d3997c
Rename newTimeTickManager to newTimeTickSender (#25415) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-07-12 14:56:28 +08:00
groot
96c987ed62
Bulkinsert supports partition keys (#25284) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-07-11 15:18:28 +08:00
wei liu
71d99d4ec2
fix unexpected metrics value cause by wrong time unit (#25240) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-06 08:48:27 +08:00
Xiaofan
63b6a4a639
use single instance ppol (#25159) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-28 14:54:45 +08:00
xige-16
33c2012675
Add more metrics (#25081) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-06-26 17:52:44 +08:00
yiwangdr
c7b851f870
add interface for non-watch metakv (#25092) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-06-26 09:20:44 +08:00
congqixia
736916222b
Stop retry lazy load stats after segment released (#25054) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-25 19:20:43 +08:00
jaime
18df2ba6fd
[Cherry-Pick] Support Database (#24769) 
Support Database(#23742)
Fix db nonexists error for FlushAll (#24222)
Fix check collection limits fails (#24235)
backward compatibility with empty DB name (#24317)
Fix GetFlushAllState with DB (#24347)
Remove db from global meta cache after drop database (#24474)
Fix db name is empty for describe collection response (#24603)
Add RBAC for Database API (#24653)
Fix miss load the same name collection during recover stage (#24941)

RBAC supports Database validation (#23609)
Fix to list grant with db return empty (#23922)
Optimize PrivilegeAll permission check (#23972)
Add the default db value for the rbac request (#24307)

Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-06-25 17:20:43 +08:00
Xiaofan
e8911ebda7
Add retry time when lazy load BF (#25096) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-25 11:32:43 +08:00
Xiaofan
72c5e2a41a
Fix channel reassigned to other datanodes (#25015) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-21 21:26:42 +08:00