Commit Graph

739 Commits

Author SHA1 Message Date
Enwei Jiao
66fdc71479
Refactor logs in DataCoord & DataNode (#25574)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-07-14 15:56:31 +08:00
yiwangdr
b9189b9f41
Organize mocks from types.go (#25466)
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2023-07-14 10:12:31 +08:00
wei liu
012fd1152a
fix datacoord consume datanode tt metrics (#25524)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-13 14:26:29 +08:00
congqixia
dbfade77c5
Retry on SyncSegment failure (#25540)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-13 14:12:29 +08:00
Bingyi Sun
b22d6d2a5d
Avoid update all binlogs every a segment is updated (#25526)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-07-13 14:08:29 +08:00
Xiaofan
dbf0130803
Faster garbage collect on compacted data (#25088)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-07-12 10:14:29 +08:00
Bingyi Sun
70c4ddc6c5
Replace LoadWithPrefix with HasPrefix (#25471)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-07-11 19:38:28 +08:00
Bingyi Sun
268368031c
Remove redundant kv mocks (#25420)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-07-07 19:38:26 +08:00
Bingyi Sun
54acb5b725
Remove alter segment interface (#25382)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-07-06 20:46:25 +08:00
Bingyi Sun
fe22720ff3
Avoid rewriting all binlogs every time a segment is updated (#25339)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2023-07-06 18:20:26 +08:00
wei liu
cba0feb119
add coordinator broker, to unify rootcoord api access (#25187)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-03 18:28:25 +08:00
congqixia
597a4d9227
Treat small segment without index as sealed (#25237)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-02 19:50:23 +08:00
Xiaofan
63b6a4a639
use single instance ppol (#25159)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-06-28 14:54:45 +08:00
xige-16
33c2012675
Add more metrics (#25081)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-06-26 17:52:44 +08:00
yiwangdr
c7b851f870
add interface for non-watch metakv (#25092)
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2023-06-26 09:20:44 +08:00
jaime
18df2ba6fd
[Cherry-Pick] Support Database (#24769)
Support Database(#23742)
Fix db nonexists error for FlushAll (#24222)
Fix check collection limits fails (#24235)
backward compatibility with empty DB name (#24317)
Fix GetFlushAllState with DB (#24347)
Remove db from global meta cache after drop database (#24474)
Fix db name is empty for describe collection response (#24603)
Add RBAC for Database API (#24653)
Fix miss load the same name collection during recover stage (#24941)

RBAC supports Database validation (#23609)
Fix to list grant with db return empty (#23922)
Optimize PrivilegeAll permission check (#23972)
Add the default db value for the rbac request (#24307)

Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-06-25 17:20:43 +08:00
xige-16
04082b3de2
Migrate the ability to upload and download binlog to cpp (#22984)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-06-25 14:38:44 +08:00
Xiaofan
72c5e2a41a
Fix channel reassigned to other datanodes (#25015)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-06-21 21:26:42 +08:00
Xiaofan
f57fe6d70b
Add compaction log (#24976)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-06-19 14:18:41 +08:00
cui fliter
35cb0b5ba6
fix function name in comment (#24879)
Signed-off-by: cui fliter <imcusg@gmail.com>
2023-06-14 16:20:38 +08:00
wayblink
bfae6b49af
Remove datanode timetick mq, use rpc to report instead (#23156)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-06-14 14:16:38 +08:00
congqixia
41af0a98fa
Use go-api/v2 for milvus-proto (#24770)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
yah01
ebd0279d3f
Check error by Error() and NoError() for better report message (#24736)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-08 15:36:36 +08:00
yiwangdr
37c02c9927
clean up meta interface (#24729)
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2023-06-08 14:18:35 +08:00
cai.zhang
93ea9c4925
Support return pending index rows when describe index (#24588)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-06-01 18:14:32 +08:00
yihao.dai
19164ea753
Make binlog size metric to collect healthy binlog only (#24557)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-31 16:45:29 +08:00
congqixia
6375236533
Make SegmentInfo.size atomic (#24457)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-29 11:31:28 +08:00
Ikko Eltociear Ashimine
eb9ef23cca
Fix typo in handler.go (#24125)
postitions -> positions

Signed-off-by: Ikko Eltociear Ashimine <eltociear@gmail.com>
2023-05-17 14:25:23 +08:00
congqixia
73a181d226
Fix get vector it timeout and improve some string const usage (#24141)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-16 17:41:22 +08:00
yihao.dai
fe0a1bc2d9
Fix panic caused by wrong logic of getting unindexed segments (#24044)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-15 20:59:28 +08:00
congqixia
f6dec281c4
Add Statistic entity id log for datacoord (#24073)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-12 18:11:21 +08:00
Enwei Jiao
7e645dabf1
Add trace log for datacoord (#24046)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-05-12 09:53:21 +08:00
congqixia
5aa9db0d38
Add collection level auto compaction enabled config (#24013)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-10 17:45:20 +08:00
smellthemoon
146050db82
Fix some wrong ut (#23990)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-05-10 09:31:19 +08:00
yihao.dai
fc010ee86b
Improve the metric of stored binlog size to the level of collection and segment (#23859)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-05-07 20:52:38 +08:00
wayblink
899702f13c
Implement GetIndexStatistics interface (#23603)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-05-06 10:34:39 +08:00
MrPresent-Han
b517bc9e6a
refine balance mechanism including:(#23454) (#23763) (#23791)
1. balance granuity to replica to avoid influence unrelated replicas
2. avoid balance back and forth

Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
2023-05-04 12:22:40 +08:00
Xiaofan
016311ad48
Fix Has Collection Failed on datacoord (#23710)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-04-27 09:50:34 +08:00
yihao.dai
ed8836cd15
Add disk quota at the collection level (#23704)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-04-26 21:52:36 +08:00
XuanYang-cn
72485c9eeb
Remove index check to trigger compaction (#23657)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-04-24 20:26:34 +08:00
congqixia
4a22af6e1a
Unwatch channel in watch buffer (#23548)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-04-20 10:34:31 +08:00
wei liu
cbfe7a45ef
fix pull target (#23491)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-04-18 18:30:32 +08:00
congqixia
d83654c33f
Add Close method for ChannelManager in datacoord (#23493)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-04-18 17:54:31 +08:00
zhenshan.cao
4a32b842e8
Improve the check logic of channel remove (#23473)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2023-04-18 02:58:30 +08:00
congqixia
ba84f52119
Fix watcher loop quit and channel shouldDrop logic (#23402)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-04-14 09:54:28 +08:00
cai.zhang
43a9e175a3
Exit component process when session key is deleted (#21658) (#22164)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-04-12 20:12:28 +08:00
zhenshan.cao
dc5abe086c
Revert "Restrain gc files by collection (#22504)" (#23380)
This reverts commit fa86de530d.

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2023-04-12 19:38:28 +08:00
cai.zhang
9288020da3
Fix bug for batch delete files on gcp of minio (#23052) (#23083) (#23090)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-04-09 16:22:30 +08:00
jaime
c9d0c157ec
Move some modules from internal to public package (#22572)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
yihao.dai
17d3190fcc
Fix DescribeCollection failed with undefined msg type (#23192)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-04-03 19:52:24 +08:00