test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-04 12:59:23 +08:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity
18,297 Commits 96 Branches 105 Tags 185 MiB
38aa53f523
Commit Graph

824 Commits

Author SHA1 Message Date
yihao.dai
4340cbfba2
Merge syncCPLagTooBehind policy into syncPeriodically policy (#26713) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-04 14:37:47 +08:00
yihao.dai
7624c2b949
Improve rated log in insertBufferNode (#26788) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-04 09:59:10 +08:00
yah01
3349db4aa7
Refine errors to remove changes breaking design (#26521) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-04 09:57:09 +08:00
congqixia
e8f1b1736e
Remove log.Error(err.error())-style log (#26783) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-01 13:09:01 +08:00
XuanYang-cn
8d54509e54
Fix CompactionLatency metrics (#26747) 
- Refine compactor logs

See also: #26743

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-31 17:35:03 +08:00
smellthemoon
6069a7d42b
Log out memory in MB (#26262) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-08-30 17:13:00 +08:00
congqixia
cc584551fb
Remove unused hash value setting in DDNode (#26575) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-28 11:30:26 +08:00
SimFG
b9bc6681ae
Use atomic.bool for the isWatchFailed param in the tickler (#26558) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-08-23 13:22:27 +08:00
XuanYang-cn
27691e843c
Fix channel checkpoint stuck (#26534) 
See also: #23621

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-22 18:10:22 +08:00
XuanYang-cn
08fd28b30b
Only do gracefully stop when DN Stop (#26399) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-21 11:16:20 +08:00
SimFG
3be4ac4022
Fix datanode/datacoord continuous restart (#26470) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-08-20 21:20:24 +08:00
Enwei Jiao
533f0ddf6d
Add amazonlinux image, prepare to remove Centos7 image, remove openblas image (#26368) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-08-18 18:32:19 +08:00
congqixia
fbb5d32cb6
Make write binlog in parallel (#26325) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-17 15:44:18 +08:00
congqixia
f371ec0316
Add data sync service close log for watch stuck debugging (#26332) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-16 10:57:32 +08:00
Enwei Jiao
78bc688d16
Remove QueryMsgStream in MqFactory interface (#26374) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-08-16 10:29:33 +08:00
congqixia
f8bcf60e55
Adjust some confusing Warning log to INFO (#26356) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-15 17:49:33 +08:00
wayblink
21eeb37ce9
Save binlog timestampFrom, timestampTo meta when compact (#26203) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-08-08 21:17:21 +08:00
congqixia
b9850ce5c0
Fix copylocks linter errors (#26217) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-08 20:41:15 +08:00
MrPresent-Han
b84f5c560a
refine log out for datanode(#25763) (#26195) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-08 16:49:08 +08:00
MrPresent-Han
fb933fe64d
fix sync memory policy lose effect when inserting too many partitions(#25763) (#26154) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-07 09:59:07 +08:00
XuanYang-cn
830f06783a
Reduce log volumes of DC&DN (#26060) 
See also: #26057

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-02 17:13:11 +08:00
XuanYang-cn
84253f255e
Fix datanode graceful stop panic (#25932) 
See also: #25925

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-07-28 10:11:08 +08:00
groot
a6808e6484
Fix bulkinsert row count error (#25869) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-07-25 11:29:00 +08:00
cai.zhang
779d677eb2
Remove lock for compaction sync segments (#25711) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-07-24 17:25:00 +08:00
jaime
8e5d6193f9
Add a timeout config for bulkinsert request (#25789) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-07-24 14:23:00 +08:00
congqixia
3c503afe7c
Use typeutil.ConcurrentMap instead of sync.Map (#25846) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-24 10:23:01 +08:00
congqixia
31b40b3ccc
Use typeutil.ConcurrentMap instead of sync.Map in fg manager (#25835) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-24 10:21:07 +08:00
Enwei Jiao
66fdc71479
Refactor logs in DataCoord & DataNode (#25574) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-07-14 15:56:31 +08:00
yiwangdr
b9189b9f41
Organize mocks from types.go (#25466) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-07-14 10:12:31 +08:00
wayblink
53b4b70316
Reduce log frequency in timetick_sender (#25532) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-07-13 14:38:29 +08:00
wei liu
012fd1152a
fix datacoord consume datanode tt metrics (#25524) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-13 14:26:29 +08:00
congqixia
dbfade77c5
Retry on SyncSegment failure (#25540) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-13 14:12:29 +08:00
wayblink
ed3e4b0bc5
Fix unstable ut in timetick_sender_test (#25539) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-07-13 10:04:29 +08:00
MrPresent-Han
ea3817fbbc
refinement towards sync_cp_lag_too_behind_policy to avoid submit syncTasks too frequently(#25441) (#25442) (#25500) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-07-12 20:22:29 +08:00
SimFG
f9e2d00f91
Prevent exclusive consumer exception in pulsar (#25376) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-07-12 17:26:30 +08:00
wayblink
fc12d3997c
Rename newTimeTickManager to newTimeTickSender (#25415) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-07-12 14:56:28 +08:00
groot
96c987ed62
Bulkinsert supports partition keys (#25284) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-07-11 15:18:28 +08:00
wei liu
71d99d4ec2
fix unexpected metrics value cause by wrong time unit (#25240) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-06 08:48:27 +08:00
Xiaofan
63b6a4a639
use single instance ppol (#25159) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-28 14:54:45 +08:00
xige-16
33c2012675
Add more metrics (#25081) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-06-26 17:52:44 +08:00
yiwangdr
c7b851f870
add interface for non-watch metakv (#25092) 
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2023-06-26 09:20:44 +08:00
congqixia
736916222b
Stop retry lazy load stats after segment released (#25054) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-25 19:20:43 +08:00
jaime
18df2ba6fd
[Cherry-Pick] Support Database (#24769) 
Support Database(#23742)
Fix db nonexists error for FlushAll (#24222)
Fix check collection limits fails (#24235)
backward compatibility with empty DB name (#24317)
Fix GetFlushAllState with DB (#24347)
Remove db from global meta cache after drop database (#24474)
Fix db name is empty for describe collection response (#24603)
Add RBAC for Database API (#24653)
Fix miss load the same name collection during recover stage (#24941)

RBAC supports Database validation (#23609)
Fix to list grant with db return empty (#23922)
Optimize PrivilegeAll permission check (#23972)
Add the default db value for the rbac request (#24307)

Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-06-25 17:20:43 +08:00
Xiaofan
e8911ebda7
Add retry time when lazy load BF (#25096) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-25 11:32:43 +08:00
Xiaofan
72c5e2a41a
Fix channel reassigned to other datanodes (#25015) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-21 21:26:42 +08:00
aoiasd
7f5de9aab3
add read lock for getSegment (#25036) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-06-21 14:06:42 +08:00
PowderLi
3f4356df10
fix the spelling of field (#25008) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-06-21 14:00:42 +08:00
yihao.dai
6168a55c04
Lower the log frequency when sync tasks are skipped (#25006) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-06-20 17:56:41 +08:00
jaime
b75dcf90c0
Fix Flush hang after SyncSegments timeout (#24953) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-06-20 10:20:41 +08:00
Xiaofan
af1d84e5e1
Support sync when cp is lagged (#24989) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-20 10:00:41 +08:00
Xiaofan
f57fe6d70b
Add compaction log (#24976) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-19 14:18:41 +08:00
yihao.dai
c73219a54d
Limit the number of concurrent sync tasks and allow only one sync task for the same segment (#24881) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-06-16 14:14:39 +08:00
congqixia
d186f35895
Fix TestChannelMetaMockSkipBFLoad/transient_error not stable (#24928) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-16 09:50:44 +08:00
wayblink
bfae6b49af
Remove datanode timetick mq, use rpc to report instead (#23156) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-06-14 14:16:38 +08:00
groot
5c3b744b0c
Fix potential crash bug of bulkinsert (#24763) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-06-13 10:22:38 +08:00
congqixia
41af0a98fa
Use go-api/v2 for milvus-proto (#24770) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-09 01:28:37 +08:00
yah01
ebd0279d3f
Check error by Error() and NoError() for better report message (#24736) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-06-08 15:36:36 +08:00
congqixia
31880ab427
Use conc.PoolOption instead of ants.Option (#24585) 
- Add conc.PoolOption to setup conc.Pool
- Change panic default behavior
- Make future has error when job panicks

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-01 15:52:31 +08:00
wayblink
5958b42941
Add retry on getChannelLatestMsgID (#24518) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-05-30 20:03:28 +08:00
aoiasd
c84bdcea49
merge stats log when segment flushing or compacting (#23570) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-05-29 10:21:28 +08:00
Xiaofan
49f75e5e11
Make flow graph retry longer (#24355) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-05-25 09:53:30 +08:00
congqixia
03297a7ef7
Fix event manager not closed after datanode stop (#24206) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-18 11:43:23 +08:00
yah01
99cc24b283
Fix compaction doesn't support JSON data (#24151) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-05-17 19:09:22 +08:00
congqixia
73a181d226
Fix get vector it timeout and improve some string const usage (#24141) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-16 17:41:22 +08:00
congqixia
084424d636
Make data node start only once (#24031) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-11 13:51:20 +08:00
congqixia
345855c984
Make datanode load statslog lazy if SkipBFStatsLog is true (#23877) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-05-06 15:12:39 +08:00
MrPresent-Han
b517bc9e6a
refine balance mechanism including:(#23454) (#23763) (#23791) 
1. balance granuity to replica to avoid influence unrelated replicas
2. avoid balance back and forth

Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
 2023-05-04 12:22:40 +08:00
congqixia
796f9355ee
Fix global rateCollector init multiple times (#23703) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-26 10:16:36 +08:00
wei liu
4fb8919a97
support collection level rate limit (#22767) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-04-25 15:54:35 +08:00
zhenshan.cao
03ec804e68
Change default shard number to 1 (#23586) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-04-21 07:08:32 +08:00
congqixia
4a6c2b78a7
Fix multiple transferNewSegment and removeSegment causes panic (#23564) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-20 13:48:31 +08:00
Enwei Jiao
967a97b9bd
Support json & array types (#23408) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
 2023-04-20 11:32:31 +08:00
congqixia
4fe363c4b2
Add KNN cgo pool (#23526) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-19 23:56:31 +08:00
congqixia
4508786819
Fix test_watch_channel unit test unstable (#23507) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-19 10:00:31 +08:00
wei liu
cbfe7a45ef
fix pull target (#23491) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-04-18 18:30:32 +08:00
wayblink
b23723535c
Add log in AddImportSegment.getChannelLatestMsgID (#23480) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-04-18 17:50:31 +08:00
zhenshan.cao
4a32b842e8
Improve the check logic of channel remove (#23473) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-04-18 02:58:30 +08:00
aoiasd
695cc769df
Fix watch event timeout when IO Pool busy (#23293) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-04-17 11:02:30 +08:00
aoiasd
f603456b72
remove segment from datanode meta when segment dropped (#23398) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-04-17 10:58:30 +08:00
congqixia
ba84f52119
Fix watcher loop quit and channel shouldDrop logic (#23402) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-04-14 09:54:28 +08:00
cai.zhang
43a9e175a3
Exit component process when session key is deleted (#21658) (#22164) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-04-12 20:12:28 +08:00
XuanYang-cn
b90fa5f459
Refine codes of datanode buffer (#23168) 
This PR refines deltabuffer of datanode:
- Add last sync time for compacted segment, see also: #23210
- Ensure all deltabuffermanager handles all delete related operations
- Change usedMemory to atomic.Int64
- Remove allocator in delete buffer

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-04-10 18:42:30 +08:00
jaime
c9d0c157ec
Move some modules from internal to public package (#22572) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-04-06 19:14:32 +08:00
aoiasd
1a485044a9
fix NumConsumers metric not work (#23167) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-04-05 08:18:28 +08:00
smellthemoon
ce91f61e5a
Pre-create topics (#21907) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-04-03 16:44:23 +08:00
XuanYang-cn
be0827417a
Fix buffer calculate memorysize negative (#23152) 
Along with the following small patches:
- And fix logging dup segmentIDs for delete buffer status
- Add String method for PriorityQueue to debug

See also: #23105

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-03-31 18:30:23 +08:00
Jiquan Long
0f2c3ac832
Optimize log (#23121) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-03-30 10:44:22 +08:00
Enwei Jiao
50dab2d394
Add tracing for retrieve (#23033) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-03-29 18:10:02 +08:00
XuanYang-cn
9691315c21
Fix flush deltabuf with segmentID=0 (#23059) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-03-28 19:14:01 +08:00
aoiasd
5d172d0f4f
Refine errors of datanode (#22852) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-03-28 19:04:00 +08:00
zhenshan.cao
e3c3c949c4
Fix datarace of flow_graph_manager_test.go in package datanode (#23043) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-03-28 10:42:00 +08:00
yihao.dai
b21f03682a
Fix flowgraph manager test (#23009) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-03-27 18:00:05 +08:00
congqixia
4008ee6d60
Remove datanode ddNode forward delta logic (#23005) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-03-27 14:18:00 +08:00
yah01
081572d31c
Refactor QueryNode (#21625) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-03-27 00:42:00 +08:00
Xiaofan
5a914aedf3
Make Datacoord TT to be configurable (#22956) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-03-24 22:24:04 +08:00
XuanYang-cn
da58ba6a49
Fix endless appending sids (#22977) 
See also: #22976

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-03-24 18:00:03 +08:00
congqixia
8f847884c2
Fix misc copylocks and remove exclude rule (#22962) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-03-24 11:17:58 +08:00
XuanYang-cn
93bc805933
Enhance ID allocator in DataNode (#22905) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-03-23 19:43:57 +08:00
yihao.dai
f2ffb5db26
Use buffer size at memory sync policy (#22825) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-03-21 21:37:56 +08:00
SimFG
4a90490a67
Fix the segment not found error (#22772) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-03-17 17:27:56 +08:00