Commit Graph

265 Commits

Author SHA1 Message Date
XuanYang-cn
a9a332dbcf
Fix empty segment compact and load (#12710)
1. Fix compaction save empty segment bug
2. Fix load empty segment bug
3. Add UT for dmlchannels to 100%

Resolves: #12450

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-12-09 11:03:08 +08:00
Bingyi Sun
a78b2010e9
[skip ci]Improve log style (#13008)
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2021-12-08 20:27:06 +08:00
Bingyi Sun
48c5632277
[skip ci]Improve log style (#13007)
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2021-12-08 20:25:15 +08:00
Bingyi Sun
320cfbb606
[skip ci]Improve log style (#13006)
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2021-12-08 20:19:26 +08:00
congqixia
aea5a61e9a
Add switch for auto compaction (#12974)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-08 19:47:05 +08:00
edward.zeng
260d3e124d
[skip ci] Refine datacoord channel store comment (#12991)
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
2021-12-08 17:57:05 +08:00
edward.zeng
2e84eef229
[skip ci] Refine datacoord channel manager comment (#12989)
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
2021-12-08 17:53:22 +08:00
edward.zeng
8ca248dce6
[skip ci] Refine datacoord allocator comment (#12985)
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
2021-12-08 17:35:06 +08:00
congqixia
de454956fe
Refactor session WatchSessions to allow rewatch when Rev Compacted (#12880)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-08 10:11:04 +08:00
JackLCL
f90410fe8e
[skip ci]Fix error log output format in segment_manager.go (#12856)
Signed-off-by: JackLCL <chenglong.li@zilliz.com>
2021-12-07 14:36:27 +08:00
JackLCL
de00aa038a
[skip ci]Fix error log output format in meta.go (#12854)
Signed-off-by: JackLCL <chenglong.li@zilliz.com>
2021-12-07 14:32:51 +08:00
congqixia
22fddc537c
Fix not all segment state is drop after DropVirtualChannel (#12702)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-03 18:55:34 +08:00
congqixia
58664bb6b9
Fix zero length slice declaration in datacoord/channel_manager.go (#12582)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-02 19:17:38 +08:00
edward.zeng
f8c826c597
Fix typo in datacoord compaction trigger (#12636)
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
2021-12-02 19:09:37 +08:00
congqixia
8f9e62fa18
Apply DropVirtualChannel and FlushManager drop mode (#12563)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-02 16:39:33 +08:00
Bingyi Sun
0778e6f715
Fix alloc segment with dropped segment (#12572)
issue: #12465
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2021-12-01 22:39:37 +08:00
godchen
5e3e3b3b41
Remove grpc returned error (#12537)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-12-01 22:17:46 +08:00
edward.zeng
5db458a44e
[skip ci] Add license for data coord compaction (#12561)
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
2021-12-01 19:33:32 +08:00
edward.zeng
446b5a3a48
[skip ci] Refine datacoord cluster comment (#12562)
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
2021-12-01 19:29:31 +08:00
Bingyi Sun
7ff15490c9
[skip ci]Add comment for GetVChanPosition (#12495)
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2021-11-30 23:15:33 +08:00
Bingyi Sun
27898928b0
[skip ci]Modify WatchChannels comment in services.go (#12417)
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@gmail.com>
2021-11-29 23:09:41 +08:00
congqixia
06ced4d1d1
Add DropVirtualChannel for DataCoord (#12361)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-29 22:35:41 +08:00
congqixia
eb48013042
Set start position for compact result segment (#12280)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-25 17:19:17 +08:00
congqixia
2ee0c7c074
Add compacted segment into check list when drop collection (#12272)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-25 15:13:15 +08:00
bigsheeper
e245ba87a9
Add excluded segments for dropped segemnts (#12247)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-11-25 14:25:15 +08:00
dragondriver
6c4c0ef6b5
Add more deploy metrics (#12248)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-11-24 15:43:15 +08:00
congqixia
7b870e3392
Limit datacoord gc scan prefixes (#12231)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-24 09:55:15 +08:00
congqixia
8e0c8febe9
Add dropped segment ids in vchaninfo (#12230)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-23 18:59:15 +08:00
Bingyi Sun
3be9442c0f
Clear dropped segment meta on etcd (#12191)
issue: #9613
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2021-11-23 11:23:15 +08:00
Bingyi Sun
01be91116e
Add GetFlushState interface (#12170)
issue: #12075
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2021-11-23 10:55:14 +08:00
congqixia
8c107ab15f
Set segment DroppedAt when compacted (#12182)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-22 20:11:14 +08:00
congqixia
4121e31df1
Send SIGINT to runner goroutine after etcd disconnects (#12163)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-22 16:23:17 +08:00
shaoyue
6777d24c65
[skip ci] Add comment for exported items (#12126)
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2021-11-19 18:05:25 +08:00
dragondriver
ee0f753f7a
Fix datarace between GetComponentStates and Register (#11935)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-11-19 13:57:12 +08:00
congqixia
dfc70108c4
Add load collection for AssignSegmentID (#12077)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-18 22:31:34 +08:00
congqixia
b99164f43d
Add datacoord GC configuration (#12061)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-18 22:29:40 +08:00
Bingyi Sun
175a656ff1
Unwatch dropped channel when init channel manager (#11986)
issue: #11558
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2021-11-17 23:25:12 +08:00
XuanYang-cn
7dbb4d10e5
Fix DeltaLogs empty yet appending a deltaInfo bug (#11984)
- Add more logs for startpositions

Fixes: #11970

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-17 17:57:11 +08:00
congqixia
5edbb82610
Add session revoke (#11908)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-16 22:31:14 +08:00
Bingyi Sun
e3e4dc7153
Improve GetVChanPosistion and fix unit tests (#11888)
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2021-11-16 21:41:50 +08:00
Xiangyu Wang
12f50cb22c
[skip ci]Update OWNERS files (#11898)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-11-16 15:41:11 +08:00
congqixia
7e43b568ee
Add dropped ts and apply when gc scan dropped segments (#11838)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-16 14:23:21 +08:00
shaoyue
6322eb31e7
[skip ci] Add comments for exported things in policy.go (#11876)
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2021-11-16 13:21:16 +08:00
Bingyi Sun
b8c5239784
Add compaction retention config (#11793)
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2021-11-16 11:47:10 +08:00
Bingyi Sun
8b60aa8c4e
Skip single compaction if there's no delta log in a segment (#11806)
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2021-11-16 11:23:11 +08:00
Bingyi Sun
902e856a06
[skip ci]Remove redundancy log (#11732)
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2021-11-15 10:25:09 +08:00
congqixia
d38043f235
Filter with partition id in GetVChanPosition (#11746)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-13 08:45:09 +08:00
Xiaofan
71e814f796
Remove common.yaml and component.yaml (#11661)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-11-12 21:25:08 +08:00
bigsheeper
93149c5ad9
Load growing segment in query node (#11664)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-11-12 18:27:10 +08:00
Bingyi Sun
5c8978e5d6
Fix drop segment bug (#11709)
issue: #11708
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2021-11-12 17:31:11 +08:00