Commit Graph

324 Commits

Author SHA1 Message Date
wei liu
1097776477
stop heartbeat if reach heartbeat limit (#26728)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-09-04 17:51:48 +08:00
XuanYang-cn
b2e7cbdf4b
Remove TimeTravel in compactor (#26785)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-09-04 17:41:48 +08:00
zhagnlu
7056e9c0f7
Increase minio log level to avoid unnecessary log (#26776)
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-09-04 17:25:48 +08:00
yihao.dai
4340cbfba2
Merge syncCPLagTooBehind policy into syncPeriodically policy (#26713)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-04 14:37:47 +08:00
yah01
3349db4aa7
Refine errors to remove changes breaking design (#26521)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-04 09:57:09 +08:00
yihao.dai
c6024a32f5
Add rate limit ratio monitoring (#26709)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-03 20:47:02 +08:00
congqixia
e8f1b1736e
Remove log.Error(err.error())-style log (#26783)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-01 13:09:01 +08:00
smellthemoon
e56b0018e4
Remove merr mask logic to make it ok to compare resp.status with origin error (#26773)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-09-01 11:09:00 +08:00
XuanYang-cn
ef75784715
Fix LoadSegmentLatency metric p99 (#26761)
See also: #26743

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-09-01 11:07:07 +08:00
Enwei Jiao
d206ebf220
Change default config files (#26780)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-01 09:19:00 +08:00
Cai Yudong
8dc16b599b
Add binary metric types SUBSTRUCTURE/SUPERSTRUCTURE back (#26766)
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-08-31 20:07:00 +08:00
XuanYang-cn
86e9f1818f
Change default compaction timeout to 15mins (#26757)
See also: #26566

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-08-31 18:07:00 +08:00
XuanYang-cn
8d54509e54
Fix CompactionLatency metrics (#26747)
- Refine compactor logs

See also: #26743

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-08-31 17:35:03 +08:00
Gao
7e36f819b4
Make autoindex config clean (#26732)
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-08-31 10:25:01 +08:00
congqixia
d343888f3c
Fix pkg codecov & static-check script (#26720)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-31 09:59:01 +08:00
jaime
c603f1c244
Remove mysql metastore (#26633)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-08-29 14:36:26 +08:00
yah01
c3f5856fbc
Fix data race for config with FileSource (#26518)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-24 09:18:24 +08:00
congqixia
ee7aef9272
Make pulsar request timeout configurable (#26525)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-23 20:46:23 +08:00
congqixia
145387fdcb
Bump proto go-api to v2.3.0 (#26561)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-23 20:18:23 +08:00
wei liu
7af0f7d90c
avoid concurrent sub/unsub on same channel (#26454)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-23 10:10:22 +08:00
jaime
b82fc8d6e7
[skip e2e] Add connection log for etcd and grpc (#26536)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-08-22 21:20:24 +08:00
SimFG
d13ca54414
Change the default gracefulStopTimeout and only warn when fail to refresh policy cache (#26450)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-08-22 09:04:21 +08:00
chyezh
85886f039e
[Fixup] msg context access data race when msgdispatcher (#26463)
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2023-08-21 19:16:22 +08:00
cai.zhang
94846995bf
Refine error message for field not exist (#26330)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-08-21 11:52:20 +08:00
yihao.dai
c3128aaef3
Accelerate server id interceptor validation (#26468)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-08-20 21:24:19 +08:00
wayblink
c5a1b41f95
Update session ttl to 60s (#26346)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-08-18 22:52:20 +08:00
wei liu
79ccf06cf6
refine proxy to querynode heartbeat interval (#26426)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-18 14:26:19 +08:00
cqy123456
7e3323a518
update dimension range of diskann. (#26405)
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2023-08-17 20:26:20 +08:00
jaime
4a388cf450
[skip e2e] Align default value with milvus.yaml file (#25619)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-08-17 20:24:19 +08:00
yihao.dai
63b86b32a6
Add server id validation interceptor (#26395)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-08-17 20:20:20 +08:00
wayblink
ab8214fef9
Add ut to ensure hash result (#26417)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-08-17 18:26:21 +08:00
yihao.dai
3c62acde05
Add coll and segState lebels to DataCoordNumStoredRows monitoring (#26299)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-08-17 15:52:17 +08:00
XuanYang-cn
be30838cf9
Refine some comments (#26401)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-08-17 10:54:18 +08:00
wei liu
4aac7a6642
refine error reason of query/search failed (#26342)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-16 19:38:18 +08:00
wei liu
74133a3996
refine retry on grpc (#26360)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-16 19:36:18 +08:00
yah01
9723787141
Calculate memory usage without page cache (#26389)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-16 17:04:17 +08:00
SimFG
19f210974f
Support to update the hook config in the runtime (#26350)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-08-16 16:03:33 +08:00
SimFG
8b54914287
Add go.mod file for the milvus/pkg package (#26359)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-08-16 14:49:32 +08:00
Enwei Jiao
78bc688d16
Remove QueryMsgStream in MqFactory interface (#26374)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-08-16 10:29:33 +08:00
congqixia
4742049ecf
Fix unstable TestSingleflight unit test (#26363)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-16 10:07:33 +08:00
congqixia
a12e45f693
Use map&mutex for dispatcher client (#26380)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-16 09:09:34 +08:00
congqixia
f8bcf60e55
Adjust some confusing Warning log to INFO (#26356)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-15 17:49:33 +08:00
cqy123456
376642b933
reset diskann min dim limit (#26344)
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2023-08-15 16:45:34 +08:00
chyezh
4c4b2903f9
[Fixup] nats log configuration, open nats log at info level by deafult (#26168)
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2023-08-14 14:21:32 +08:00
Enwei Jiao
a3c176045d
Fix kafka producer init many times (#26314)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-08-13 16:59:30 +08:00
Gao
b6fcbb0998
Support ScaNN index (#26099)
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-08-11 14:21:29 +08:00
zhagnlu
411f9ac823
Upgrade minio-go and add region and virtual host config for segcore chunk manager (#26194)
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-08-11 10:37:36 +08:00
xige-16
1971d98897
Add disk metric info (#25675)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-08-11 10:35:42 +08:00
wei liu
05c87af146
fix unexpected score in look aside balancer (#26213)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-10 19:45:16 +08:00
xige-16
1055c90456
Add default retrieve limit (#24782)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-08-10 14:11:15 +08:00
MrPresent-Han
3421956afa
modify default value for sync max parallel to mitigate oom on dn(#26763) (#26231)
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-08-10 10:23:15 +08:00
wei liu
f33a89387f
enable config shard level cost metrics whether contains worker's cost (#26132)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-09 11:37:15 +08:00
congqixia
767955ec6b
Reduce MQ buffer length and flowgraph wait queue length to 16 (#26179)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-09 10:05:14 +08:00
yihao.dai
b6effd7345
Disable deny writing when the growing segment size exceeds the watermark (#26163)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-08-08 12:43:07 +08:00
xige-16
d1d0169fa3
Delete useless config (#26173)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-08-08 10:23:08 +08:00
PowderLi
a7eecb1be0
support high-level RESTFUL API, listen on the same port as grpc. (#25108)
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-08-08 10:15:07 +08:00
congqixia
241117dd6d
Use last end position in pursuit mode (#26144)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-05 16:55:06 +08:00
SimFG
d2649b63db
Delete the user-role mapping info when deleting the user (#25988)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-08-04 18:37:08 +08:00
wayblink
775f03ed5d
Add default value for mq.type (#26064)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-08-04 18:35:07 +08:00
congqixia
fcde5b8d7f
Add pursuit mode for mqtt msgstream (#26098)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-04 12:33:06 +08:00
wei liu
518b6310a2
refine retry times on replica (#26043)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-03 15:55:09 +08:00
MrPresent-Han
47392b0a0f
support metrics mutex to monitor cost of locks(#26102) (#26103)
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-08-03 15:03:06 +08:00
MrPresent-Han
5634ba777d
add new threadpool with various priority to avoid deadlock(#25781) (#26028)
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-08-03 09:31:07 +08:00
congqixia
0727e017c4
Add Promethues suite test utility (#26046)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-02 10:05:05 +08:00
yah01
9c55a7f422
Add worker num as one of load resource (#26045)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-01 21:47:06 +08:00
yah01
2180ef180c
Record only failed task error (#26033)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-08-01 10:11:05 +08:00
congqixia
8b5e276361
Use different cgo pool for SQ and other operations (#26021)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-01 09:19:05 +08:00
zhagnlu
833674c1cb
add glog configurable function and redirect aws log to segcore log (#25664)
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-07-27 19:49:02 +08:00
congqixia
33cfb2ba2f
Fix MsgPack from mq msgstream ts not set (#25926)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-27 13:57:12 +08:00
xige-16
6f18587f35
Fix small segment compaction (#21327)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-07-26 14:49:01 +08:00
MrPresent-Han
b5e79e7f34
add metrics for assign-segment on proxy(#24570) (#25907)
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-07-26 14:15:01 +08:00
chyezh
e24a8b3606
[Feature] Add benchmark and retention configuration for nmq (#25768)
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2023-07-25 19:33:01 +08:00
wei liu
302897f866
refine look aside balance logic (#25837)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-25 18:51:01 +08:00
congqixia
0bc03ede0d
Add eventlog pkg and support grpc streaming event observation (#25812)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-25 17:23:01 +08:00
wei liu
2ff469771c
fix delegator insert/delete cost metrics (#25871)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-25 10:01:00 +08:00
jaime
8e5d6193f9
Add a timeout config for bulkinsert request (#25789)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-07-24 14:23:00 +08:00
MrPresent-Han
f4e72cb170
remove sync segmentLastExpire every time when assigning(#25271) (#25316) (#25557)
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-07-24 14:11:07 +08:00
Bingyi Sun
9f31fc9a31
Add broker timeout config item. (#25855)
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-07-24 14:07:00 +08:00
congqixia
3c503afe7c
Use typeutil.ConcurrentMap instead of sync.Map (#25846)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-24 10:23:01 +08:00
congqixia
37636b2f4d
Add compaction worker max parallel config item (#25834)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-24 10:17:00 +08:00
cai.zhang
b15e34db21
Add contraint for compaction based indexed segments (#25709)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-07-23 21:31:00 +08:00
cqy123456
0f3350ded4
set knowhere thread pool size in index node (#25804)
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2023-07-21 18:50:59 +08:00
wei liu
32827f538a
add metrics for delegator insert/delete cost (#25733)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-21 15:30:59 +08:00
congqixia
1045c88102
Support replace indexed field in QueryCoord (#25747)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-19 21:22:58 +08:00
congqixia
fc6f0f97a6
Add skip level for rated log (#25765)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-19 20:43:02 +08:00
wei liu
6534396b3d
enable config different interval for different checker (#25514)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-19 16:50:57 +08:00
Cai Yudong
9a4761dcc7
Remove binary metrics TANIMOTO/SUPERSTRUCTURE/SUBSTRUCTURE (#25708)
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-07-19 16:16:58 +08:00
aoiasd
3545b1a608
Refine rocksmq (#25031)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-07-18 20:18:57 +08:00
Enwei Jiao
9fe62cb5f3
Fix kafka producer panic by nil (#25691)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-07-18 18:24:57 +08:00
congqixia
8d343bf75a
Make compaction rpc timeout and parallel maxium configurable (#25672)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-18 14:25:20 +08:00
Cai Yudong
73512c72fd
Remove calc_distance (#25663)
Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2023-07-18 14:23:20 +08:00
congqixia
9a61c0291b
Disable error verbose for zap logger (#25688)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-18 12:53:21 +08:00
aoiasd
77a9553c3f
Add some log when node being reachable or unreachable (#25572)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-07-18 10:35:20 +08:00
yah01
948d1f1f4a
Handle errors by merr for QueryCoord (#24926)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-17 14:59:34 +08:00
wei liu
51c348158b
fix use pChannel in seek position (#25611)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-17 09:18:33 +08:00
yihao.dai
f19621fc8c
Support refund the tokens to limiter (#25598)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-07-14 15:58:32 +08:00
Bingyi Sun
c51922eb0b
Use sync.Map to reduce lock granularity to make create collection faster (#25485)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-07-13 21:46:31 +08:00
wayblink
50454bec4c
Allow empty minio.rootPath (#25534)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-07-13 14:46:29 +08:00
MrPresent-Han
ea3817fbbc
refinement towards sync_cp_lag_too_behind_policy to avoid submit syncTasks too frequently(#25441) (#25442) (#25500)
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-07-12 20:22:29 +08:00
yah01
205a7c430a
Fix QueryNode panic while inserting to released segment (#25495)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-12 19:48:29 +08:00