test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-04 12:59:23 +08:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity
18,826 Commits 96 Branches 105 Tags 185 MiB
5164d30287
Commit Graph

499 Commits

Author SHA1 Message Date
wayblink
c5a1b41f95
Update session ttl to 60s (#26346) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-08-18 22:52:20 +08:00
wei liu
79ccf06cf6
refine proxy to querynode heartbeat interval (#26426) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-08-18 14:26:19 +08:00
wei liu
74133a3996
refine retry on grpc (#26360) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-08-16 19:36:18 +08:00
chyezh
4c4b2903f9
[Fixup] nats log configuration, open nats log at info level by deafult (#26168) 
Signed-off-by: chyezh <ye.zhen@zilliz.com>
 2023-08-14 14:21:32 +08:00
yah01
48422dd4c5
Fix spawn too many threads (#26293) 
- Low the thread pool cap
- Limit CGO calls concurrency

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-08-11 18:29:29 +08:00
zhagnlu
411f9ac823
Upgrade minio-go and add region and virtual host config for segcore chunk manager (#26194) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-08-11 10:37:36 +08:00
MrPresent-Han
3421956afa
modify default value for sync max parallel to mitigate oom on dn(#26763) (#26231) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-10 10:23:15 +08:00
congqixia
767955ec6b
Reduce MQ buffer length and flowgraph wait queue length to 16 (#26179) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-09 10:05:14 +08:00
yihao.dai
b6effd7345
Disable deny writing when the growing segment size exceeds the watermark (#26163) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-08-08 12:43:07 +08:00
wei liu
49902f1b37
adjust default value of queryNode.enableDisk (#25404) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-08-04 10:07:06 +08:00
MrPresent-Han
47392b0a0f
support metrics mutex to monitor cost of locks(#26102) (#26103) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-03 15:03:06 +08:00
MrPresent-Han
5634ba777d
add new threadpool with various priority to avoid deadlock(#25781) (#26028) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-08-03 09:31:07 +08:00
zhagnlu
833674c1cb
add glog configurable function and redirect aws log to segcore log (#25664) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-07-27 19:49:02 +08:00
xige-16
6f18587f35
Fix small segment compaction (#21327) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-07-26 14:49:01 +08:00
chyezh
e24a8b3606
[Feature] Add benchmark and retention configuration for nmq (#25768) 
Signed-off-by: chyezh <ye.zhen@zilliz.com>
 2023-07-25 19:33:01 +08:00
yah01
45b89cfc71
Low the predicting memory usage factor to 1 (#25884) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-25 16:37:01 +08:00
MrPresent-Han
f4e72cb170
remove sync segmentLastExpire every time when assigning(#25271) (#25316) (#25557) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-07-24 14:11:07 +08:00
Bingyi Sun
9f31fc9a31
Add broker timeout config item. (#25855) 
Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-07-24 14:07:00 +08:00
cai.zhang
b15e34db21
Add contraint for compaction based indexed segments (#25709) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-07-23 21:31:00 +08:00
aoiasd
3545b1a608
Refine rocksmq (#25031) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-07-18 20:18:57 +08:00
congqixia
8d343bf75a
Make compaction rpc timeout and parallel maxium configurable (#25672) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-18 14:25:20 +08:00
Xiaofan
dbf0130803
Faster garbage collect on compacted data (#25088) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-07-12 10:14:29 +08:00
yah01
8b06941da3
Reduce the load memory usage predict factor to 2 (#25469) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-07-11 13:06:28 +08:00
smellthemoon
d63323d117
Add rate limit and deny write in upsert (#25351) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-07-11 11:20:34 +08:00
wei liu
342cfcad46
decrease default value of group max nq (#25380) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-07-07 15:40:25 +08:00
congqixia
efdd71c640
Make cgo pool size larger than worker pool size (#25318) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-05 16:56:26 +08:00
foxspy
31173727b2
growing segment index memory opt & get vector bugfix (#25272) 
Signed-off-by: xianliang <xianliang.li@zilliz.com>
 2023-07-05 00:04:25 +08:00
chyezh
d7d61f529c
[Feature|Pick] enable scheduler policy and add user-task-polling policy (#24839) 
Signed-off-by: chyezh <ye.zhen@zilliz.com>
 2023-07-03 18:24:25 +08:00
smellthemoon
b30517d303
Enlarge timeout to prevent health check failure (#25173) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-06-29 17:10:23 +08:00
jaime
18df2ba6fd
[Cherry-Pick] Support Database (#24769) 
Support Database(#23742)
Fix db nonexists error for FlushAll (#24222)
Fix check collection limits fails (#24235)
backward compatibility with empty DB name (#24317)
Fix GetFlushAllState with DB (#24347)
Remove db from global meta cache after drop database (#24474)
Fix db name is empty for describe collection response (#24603)
Add RBAC for Database API (#24653)
Fix miss load the same name collection during recover stage (#24941)

RBAC supports Database validation (#23609)
Fix to list grant with db return empty (#23922)
Optimize PrivilegeAll permission check (#23972)
Add the default db value for the rbac request (#24307)

Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-06-25 17:20:43 +08:00
Xiaofan
72c5e2a41a
Fix channel reassigned to other datanodes (#25015) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-21 21:26:42 +08:00
cai.zhang
c9e456c6eb
Remove metric_type check and fix some minor bugs (#24921) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Co-authored-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-06-19 09:54:41 +08:00
yihao.dai
c73219a54d
Limit the number of concurrent sync tasks and allow only one sync task for the same segment (#24881) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-06-16 14:14:39 +08:00
MrPresent-Han
ce5cb3c0c5
[skip e2e] modify default used balancer for 2.3 (#24937) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-06-16 14:00:45 +08:00
wayblink
bfae6b49af
Remove datanode timetick mq, use rpc to report instead (#23156) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-06-14 14:16:38 +08:00
chyezh
f97127ae55
add nats mq wrappers (#24445) 
bug fixup, configurable natsmq, add unittest, pass e2e.



move natsmq to pkg project

Signed-off-by: chyezh <ye.zhen@zilliz.com>
Co-authored-by: yiwangdr <yiwangdr@gmail.com>
 2023-06-07 10:00:37 +08:00
Jiquan Long
29ae1229b6
Support AutoIndex (#24387) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-05-29 20:35:28 +08:00
wei liu
1deac47069
reduce grpc client dial timout (#24427) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-05-29 10:03:28 +08:00
Bingyi Sun
46a8ca5b5b
Change log level to info (#24339) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2023-05-24 10:15:25 +08:00
MrPresent-Han
7744573d3d
support parms for import maxfilesize(#24191) (#24192) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-05-18 18:59:27 +08:00
yihao.dai
7384d83d2c
Support rate limit based on growing segments size (#24121) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-05-17 09:57:22 +08:00
wei liu
4c956fab73
enable config collection level rate limit (#24012) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-05-12 18:13:26 +08:00
MrPresent-Han
b517bc9e6a
refine balance mechanism including:(#23454) (#23763) (#23791) 
1. balance granuity to replica to avoid influence unrelated replicas
2. avoid balance back and forth

Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
 2023-05-04 12:22:40 +08:00
yihao.dai
8c060215e9
Add collection number quota per DB (#23656) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-04-28 11:02:35 +08:00
smellthemoon
45fbe1d1a7
Change proxy max shard num (#23777) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-04-27 22:20:35 +08:00
yah01
eab94489ba
Refine the merge algorithm (#23767) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-04-27 18:26:35 +08:00
Bingyi Sun
f289aed63a
Fix nightly tests timeout (#23751) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2023-04-27 14:56:35 +08:00
smellthemoon
912cf4ef0f
Change some configurations, include change the defaultChannelNum to 16 (#23617) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-04-27 14:26:35 +08:00
yihao.dai
ed8836cd15
Add disk quota at the collection level (#23704) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-04-26 21:52:36 +08:00
SimFG
5cd21893c8
Fix superusers' password verification problem (#23733) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-04-26 21:16:34 +08:00
foxspy
6f4ed517de
add growing segment index (#23615) 
Signed-off-by: xianliang <xianliang.li@zilliz.com>
 2023-04-26 10:14:41 +08:00
yah01
4a2726b3f9
Remove dup config items (#23653) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-04-24 19:02:33 +08:00
aoiasd
695cc769df
Fix watch event timeout when IO Pool busy (#23293) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-04-17 11:02:30 +08:00
cqy123456
7ff6a3a246
Set query node thread pool size in milvus.yaml (#23286) 
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2023-04-17 10:44:29 +08:00
Xiaofan
680ad482b7
Check balance checker chore to 10s (#23304) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-04-09 16:14:32 +08:00
smellthemoon
ce91f61e5a
Pre-create topics (#21907) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-04-03 16:44:23 +08:00
Enwei Jiao
ef57ba3bac
Remove easylogging and replace by GLog (#22979) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-03-27 18:22:02 +08:00
yah01
081572d31c
Refactor QueryNode (#21625) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-03-27 00:42:00 +08:00
yihao.dai
f2ffb5db26
Use buffer size at memory sync policy (#22825) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-03-21 21:37:56 +08:00
MrPresent-Han
77c9e33e70
support dml channel balancer on datacoord (#22324) (#22377) (#22692) 
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
 2023-03-20 10:01:56 +08:00
shaoyue
32581e6452
Support aliyun oss as object storage with ak or IAM (#22376) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2023-03-09 16:33:52 +08:00
Xiaofan
256aaa1944
Change Etcd session timeout to 20s (#22401) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-02-28 11:13:47 +08:00
Xiaofan
fa101e9751
Change RockMQ page size to 256MB (#22432) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-02-28 10:45:47 +08:00
wayblink
416866e42f
Add memory usage too large sync policy (#22244) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-02-27 17:47:51 +08:00
Enwei Jiao
0851e05014
Gernate milvus.yaml by code (#22003) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-02-23 11:37:46 +08:00
aoiasd
148a024e05
Add tickle for datacoord watch event (#21193) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-02-15 16:20:34 +08:00
wayblink
77a27c6dfc
Support print log to file and console at the same time (#21946) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-02-14 16:56:34 +08:00
jaime
a2435cfc4f
[Cherry-Pick]Add a segment metrics for count number of binlog files (#22103) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-02-13 10:24:33 +08:00
Jiquan Long
004a8c25ca
Remove hardcode max shards num (#22004) 
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-02-07 10:17:53 +08:00
jaime
baa035885d
Modify default value of some configurations (#22013) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-02-07 10:05:56 +08:00
cai.zhang
ae305a54bd
Set scheduler duration of index task can be configurable (#21944) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-02-05 17:21:53 +08:00
jaime
748a935a11
Modify segment seal policy with number of statslog files (#21957) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-02-03 17:07:52 +08:00
cai.zhang
a04238c25f
Keep the configuration of indexCoord (#21879) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-31 12:07:50 +08:00
Enwei Jiao
cecd59d940
Disable time-travel by default (#21775) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-01-18 13:57:43 +08:00
Xiaofan
f8e1566b24
Support zstd compression in grpc (#21689) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-01-13 10:55:41 +08:00
Enwei Jiao
fb42466c65
Use opentelemetry (#21509) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-01-12 16:09:39 +08:00
cai.zhang
aa203acfb3
Low IndexCoord weight (#21548) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-06 14:21:37 +08:00
cai.zhang
e5f408dceb
Merge IndexCoord and DataCoord (#21267) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-04 19:37:36 +08:00
SimFG
037ccccfcc
Add the gracefulStopTimeout config for the node (#21449) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-12-29 17:33:31 +08:00
congqixia
8b4d60b073
Add guarantee and serviceable lag too large check (#21441) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-29 15:17:30 +08:00
jaime
d401608899
Add a segment seal policy by number of binlog files (#21263) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-12-20 14:09:25 +08:00
Enwei Jiao
166e9f0da5
Refactor GrpcConfig (#21142) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-12-16 15:59:23 +08:00
Ten Thousand Leaves
ef3fcdf65b
Small candidate compaction should only happen with >1 segments (#21249) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-12-16 10:01:23 +08:00
Ten Thousand Leaves
9d8f422cb8
Reduce maxFieldNum from 256 to 64 (#21104) 
issue: #21077

/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-12-14 10:39:22 +08:00
aoiasd
9079b1d5a4
Rename default accesslog dir name (#21086) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2022-12-09 18:09:19 +08:00
SimFG
4a5c282b62
Add the superuser config (#21090) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-12-09 16:13:20 +08:00
Ten Thousand Leaves
d67db2ed9b
Make compaction target able to exceed segment max size by a bit (#21068) 
issue: #21053
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-12-09 16:03:20 +08:00
yah01
9ebaa10dec
Add more logs for GetShardLeaders (#21046) 
Also increase the heartbeatAvailableInterval from 2.5s to 10s

Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-12-08 19:09:18 +08:00
Xiaofan
e131915207
Support 64M Rpc Limit (#20845) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-12-07 10:07:18 +08:00
SimFG
f8cff79804
Support the graceful stop for the query node (#20851) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-12-06 22:59:19 +08:00
Ten Thousand Leaves
2f5b03fef7
Split small segment proportion and small segment compactable proportion (#20996) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-12-06 14:37:18 +08:00
yah01
aec347e591
Check last heartbeat response time while fetching shard leaders (#20968) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-12-05 15:09:20 +08:00
Ten Thousand Leaves
67390d208b
Reduce seal proportion from 0.25 to 0.23 (#20907) 
When segment are sealed with > 0.25 * segment.maxSize, we won't be able to compact 4 segments like these because the total size will be slightly over 0.25 * 4 * segment.maxSize.

/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-11-30 18:47:15 +08:00
yah01
060649b8aa
Refactor task scheduler and executor (#20828) 
Make the performance able to scale out

Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-11-30 13:57:15 +08:00
Ten Thousand Leaves
0700e56008
Increase MaxWatchDuration and make it configurable (#20884) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-11-29 17:19:14 +08:00
xige-16
1ad702cc40
Reduce the memory used by load disk index (#20475) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-11-21 19:35:11 +08:00
xige-16
81b6703201
Fix spell PQ error (#20725) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-11-21 16:35:11 +08:00
bigsheeper
fef87f879c
Disable tt delay protection (#20657) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-11-17 20:21:14 +08:00
bigsheeper
fc15789da9
Ensure compatibility of channel seek position and move syncPeriod to config (#20504) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-11-12 21:09:04 +08:00
wei liu
701e7b5a6e
support pulsar tenant and auth (#19727) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2022-11-11 10:55:05 +08:00