Enwei Jiao
c05b9ad539
Add event dispatcher for config ( #20393 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-17 18:59:09 +08:00
Enwei Jiao
19524a5344
Fix nodeID mismatch at standalone mode ( #20648 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-17 17:15:08 +08:00
cai.zhang
3c09a5f2e8
Use GetSegmentUnsafe when get unhealthy segment ( #20617 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-11-15 19:55:07 +08:00
XuanYang-cn
51093dfccd
Add a string func for allocation ( #20611 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-11-15 19:21:12 +08:00
zhenshan.cao
c5dc076b04
Fix start of DataCoord ( #20581 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-11-15 14:19:06 +08:00
Ten Thousand Leaves
480a2b8632
Improve compaction logs ( #20490 )
...
/kind improvement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-11-15 11:13:07 +08:00
bigsheeper
e73bdbe17d
Disk quota only checks healthy segments ( #20574 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-11-14 21:03:09 +08:00
smellthemoon
a7ba416b5a
Remove invalid timestamp ( #20557 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2022-11-14 16:21:07 +08:00
cai.zhang
2b02b869ec
Return real-time indexed rows and total rows ( #20460 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-11-14 14:37:07 +08:00
bigsheeper
fc15789da9
Ensure compatibility of channel seek position and move syncPeriod to config ( #20504 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-11-12 21:09:04 +08:00
congqixia
c21497fbb4
Filter fake segments for recovery info ( #20493 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-11-11 18:03:05 +08:00
bigsheeper
cd19d99ad7
Add channel level checkpoint ( #20350 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-11-10 22:13:04 +08:00
SimFG
2aba154ad3
Make the subName unique for the datanode tt channel ( #20331 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-11-08 20:15:04 +08:00
SimFG
9c6436d72d
Add time log for methods of starting the node ( #20313 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-11-08 20:13:03 +08:00
Enwei Jiao
7e56e7a976
Fix datarace at Setlogger ( #20376 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-08 14:33:03 +08:00
jaime
709a0a94e9
Add some msgstream metrics ( #20296 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-11-07 10:15:02 +08:00
Enwei Jiao
956c5e1b9d
Make Params singleton ( #20088 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-04 14:25:38 +08:00
congqixia
f557af1143
Verify target id for FlushSegments request in datanode ( #20308 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-11-03 22:47:35 +08:00
XuanYang-cn
22fb1d3b93
Remove dup mock_kv.go and mock_txn_kv.go ( #20267 )
...
Replace them with mockery generated mocks in kv/mocks
See also: #19289
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-11-03 14:41:35 +08:00
Ten Thousand Leaves
5708352a80
Improve DataCoord and DataNode logs ( #20265 )
...
/kind improvement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-11-03 12:15:35 +08:00
Bingyi Sun
8d47356aa9
Use compaction generated segment's checkpoint as compacted segment's seekPosition ( #20240 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-11-01 21:01:34 +08:00
congqixia
5f5fdb0789
Compaction bypass stale segments ( #20202 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-10-31 19:13:34 +08:00
Xiaofan
2bfecf5b4e
Refine bloomfilter and memory usage ( #20168 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-31 17:41:34 +08:00
SimFG
c604d2f933
Make param names clearer in the policy.go
( #20136 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-31 11:35:34 +08:00
SimFG
a90e28cf45
Fix sessions
concurrency problem ( #20140 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-27 19:39:31 +08:00
Xiaofan
11efa0bb5f
Fix compacted segment not dropped ( #20128 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-27 17:15:32 +08:00
groot
bee66631e3
Refine bulkload ( #19671 )
...
Signed-off-by: yhmo <yihua.mo@zilliz.com>
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-10-27 16:21:34 +08:00
Xiaofan
def5972e01
fix duplicated handoff request ( #20108 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-27 15:15:32 +08:00
jaime
d8ca56e084
Fix empty segment handoff fails ( #20073 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-26 16:49:31 +08:00
bigsheeper
f0c616050a
Reduce and improve quota center log ( #20058 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-10-25 19:39:30 +08:00
Xiaofan
192151bc11
Refine Data Log ( #19996 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-25 19:31:30 +08:00
Xiaofan
1f170ce1ba
Refine kafka consumer ( #19846 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-25 13:23:30 +08:00
congqixia
2e6ddd7f2a
Add datanode&channel match check for Flush ( #19985 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-10-22 12:09:28 +08:00
smellthemoon
581e1d8c35
Fill MsgBase ALL ( #19912 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2022-10-21 15:57:28 +08:00
yah01
e8ca754dca
Fix compaction produces empty segment ( #19936 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-20 22:49:28 +08:00
yah01
52cd40fbc4
Remove codes related to deprecated field ( #19897 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-19 18:29:26 +08:00
bigsheeper
99df2d9b54
Add binlogSize and ttDelay metrics monitor ( #19872 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-10-19 15:13:26 +08:00
jaime
59bcbf0cf6
Support check health for the service level ( #19779 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-18 13:39:26 +08:00
SimFG
a55f739608
Separate public proto files ( #19782 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
yah01
89c9cb3680
Fix scan may break GC limitation ( #19670 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-15 15:13:24 +08:00
bigsheeper
3a5aaeb7ad
Fix read queue metrics and memory protection ( #19787 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-10-14 18:05:24 +08:00
xige-16
a1db9038fb
Move disk index params to config file ( #19714 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-10-14 17:51:24 +08:00
bigsheeper
bdaddd7b09
Revert DML quota check ( #19790 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-10-14 17:17:23 +08:00
bigsheeper
22b32465e7
Make bulkLoad failed if quota exceeds and remove autoindex config ( #19770 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-10-14 14:27:25 +08:00
Ten Thousand Leaves
be4b0189b1
Fix potential long log ( #19757 )
...
/kind improvement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-10-13 19:21:23 +08:00
jaime
ab88dd77e7
Remove unnecessary log for datacoord gc ( #19698 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-13 14:43:23 +08:00
bigsheeper
e6c76aba0e
Force deny writing if disk quota exceeded ( #19680 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-10-12 10:03:22 +08:00
congqixia
2a2d32c4fc
Fix datacoord panics when collection info not found ( #19706 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-10-11 21:39:24 +08:00
jaime
10c03de628
Refine method name from BroadCastXx to BroadcastXx ( #19700 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-11 21:07:23 +08:00
Eng Zer Jun
438790744d
test: use T.Setenv
to set env vars in tests ( #19649 )
...
This commit replaces `os.Setenv` with `t.Setenv` in tests. The
environment variable is automatically restored to its original value
when the test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.Setenv
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-10-11 18:43:23 +08:00
Enwei Jiao
667982a239
Fix static-check ( #19707 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-10-11 17:59:22 +08:00
XuanYang-cn
28e949cf31
Add GetDataVChanPositions ( #19657 )
...
The original GetVChanPositions are specialized for QueryCoord,
the recovery info needed by DataNode shouldn't take `indexed` state
into account.
This PR splits GetVChanPositions into GetDataVChanPositions and
GetQueryVChanPositions.
See also: #19653
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-10-11 16:51:22 +08:00
jaime
31db01b6ae
Support TTL for collection level ( #19383 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-10 20:31:22 +08:00
wayblink
85e04d845c
Move GetComponentStates to milvus.proto, make it an external interface ( #19594 )
...
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-10-10 15:55:22 +08:00
congqixia
d64f318a80
Add collection schema in watch channel info ( #19663 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-10-10 01:37:21 +08:00
wayblink
e7c2500c71
Implement GetPersistentSegmentInfo through datacoord ( #19205 )
...
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-10-08 11:51:02 +08:00
jaime
1919353f02
Fix parse segment id error from delta log ( #19566 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-09-30 14:18:55 +08:00
Ten Thousand Leaves
b8b2e8eb92
Add logs to better track DataCoord meta update ( #19485 )
...
/kind improvement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-09-30 11:48:55 +08:00
wayblink
54e2c079c4
MEP-30:Support basic coordinators active-standby mechanism #16901 ( #19166 )
...
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-09-29 18:35:02 +08:00
congqixia
838a633584
Add ctx parameter for ChunkManager methods ( #19546 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-29 16:18:56 +08:00
yah01
2dca3688a0
Fix data race in DataCoord unit test ( #19469 )
...
Refine the DataCoord initialization
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-09-27 18:04:53 +08:00
XuanYang-cn
a0ce533727
Add a SyncSegments to sync meta between DN and DC ( #19298 )
...
See also: #19072
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-09-27 16:02:53 +08:00
Ten Thousand Leaves
b30c9d4f60
Support Bulk Load in Milvus 2.2 ( #18982 )
...
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-09-26 18:06:54 +08:00
zhenshan.cao
c15b880f0e
Remove redundant logs ( #19412 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-09-26 15:40:53 +08:00
zhagnlu
59f8e96522
Change compactor policy from loading all binlogs to loading serially ( #13716 ) ( #19280 )
...
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Signed-off-by: zhagnlu <lu.zhang@zilliz.com>
Co-authored-by: zhagnlu <lu.zhang@zilliz.com>
2022-09-25 20:12:52 +08:00
jaime
91df8f2d6a
Use log id instead of log path of the binlog in metastore ( #19123 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-09-25 15:56:51 +08:00
Xiaofan
928a213e31
Fix Chunk Manager Path mis use ( #19141 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-09-23 14:40:51 +08:00
XuanYang-cn
432b5ddf2e
Add SyncSegments into proto ( #19312 )
...
See also: #19072
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-09-23 10:22:52 +08:00
yah01
ee9d634a43
Compaction requires the source segments indexed ( #19368 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-09-22 18:34:50 +08:00
yah01
71d4a32ce6
Fix deadlock ( #19340 )
...
The IsParentDropped() acquires read lock of meta,
which leads to recursively acquire read lock
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-09-22 15:48:50 +08:00
Jiquan Long
5141e05c47
Add policy to seal long time idle segment ( #19222 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-09-20 20:54:50 +08:00
yah01
0306f71b49
Remove incorrect formatting, unused conditions ( #19239 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-09-20 19:16:49 +08:00
SimFG
d7f38a803d
Separate some proto files ( #19218 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
yah01
11b352c3ae
Guarantee flushed segments are all indexed ( #19062 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-09-16 11:32:48 +08:00
cai.zhang
e65ba6331e
Only store segmentID for flushed segments event ( #19075 )
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-09-09 15:52:35 +08:00
wayblink
3927ae9952
Return all flushed segments and time-of-seal in Flush interface ( #18882 )
...
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-09-09 09:58:37 +08:00
jaime
362c663b22
Remove deprecated inner compaction type ( #19024 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-09-07 19:16:34 +08:00
Jiquan Long
a5e2d6b6fb
Refactor RootCoord ( #18930 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: xaxys <tpnnghd@163.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: xaxys <tpnnghd@163.com>
2022-09-05 13:29:11 +08:00
congqixia
29c72cad15
Use segment statslog instead of separate statslog ( #18775 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-02 10:34:59 +08:00
congqixia
43c14055f6
Fix test compact logic and data race ( #18932 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-08-31 12:10:57 +08:00
Enwei Jiao
096149e33f
fix GetCompactionState hang ( #18829 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-08-26 14:22:55 +08:00
jaime
ef9098f84a
Refine save dropped segments into kv metastore ( #18779 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-08-25 16:48:53 +08:00
cai.zhang
c924f73105
Refactor for IndexCoord to support cloud ( #18643 )
...
Co-authored-by: Zach41 <zongmei.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Zach41 <zongmei.zhang@zilliz.com>
2022-08-25 15:48:54 +08:00
wayblink
6fd83464cd
Fix watchDmChannel may be out-of-date after compaction issue and add context ( #18681 )
...
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-08-23 17:44:56 +08:00
Enwei Jiao
d9c7519107
refactor compaction concurency logic ( #18660 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-08-23 15:50:52 +08:00
Jiquan Long
641afae22f
Watch channels with start positions ( #18744 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-08-22 14:42:52 +08:00
codeman
70254c3ec5
Unified catalog interface for segment ( #18289 ) ( #18290 )
...
Signed-off-by: kejiang <ke.jiang@zilliz.com>
Signed-off-by: kejiang <ke.jiang@zilliz.com>
Co-authored-by: kejiang <ke.jiang@zilliz.com>
2022-08-20 10:24:51 +08:00
Ten Thousand Leaves
fc42ee5bb3
Fix # of rows of recovering segment ( #18736 )
...
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-08-19 19:50:50 +08:00
XuanYang-cn
4c86cc63ba
Fix DN flowgraph delete buffer logs ( #18657 )
...
1. Remove logs about not existing segments.
2. Group logs by timestamp.
3. Log changed segments only.
4. Pair the segments reference lock and unlock log by taskID.
Resolves : #18655
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-08-15 16:28:48 +08:00
aoiasd
8a70f2badb
Add ShowConfigurations for all remaining session ( #18594 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2022-08-12 13:20:39 +08:00
jaime
0e95f8a690
Refine indexcoord logs ( #18589 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-08-11 10:00:38 +08:00
jaime
efa5dfaa7b
Fix Parse binlog path failure ( #18583 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-08-10 17:04:37 +08:00
Ten Thousand Leaves
12023e8569
Fix a potential nullptr issue with metric ( #18540 )
...
/kind improvement
related to: #18530
but not the root cause
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-08-09 21:12:42 +08:00
SimFG
ff0200210a
Support Role-Based Access Control ( #18425 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-08-04 11:04:34 +08:00
congqixia
28aadc988d
Use vchannel name in datanode subName ( #18519 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-08-03 17:10:36 +08:00
jaime
c9174d55ba
Refine merge operation during compacting phase ( #18399 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-07-28 14:52:31 +08:00
congqixia
e9d777b336
Fix ParseSegmentIDBinlog panicks with bad input ( #18413 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-07-26 19:32:30 +08:00
xing.zhao
d90313a1a8
Use chunkManager rather than minio.Client in datanode garbage collection ( #18092 )
...
Signed-off-by: xingzhao <xing.zhao@zilliz.com>
Co-authored-by: xingzhao <xing.zhao@zilliz.com>
2022-07-22 22:10:28 +08:00
Ten Thousand Leaves
cd2f64d30c
Fix a nil pointer issue ( #18343 )
...
/kind bug
issue: #18270
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-07-20 17:56:30 +08:00
Ten Thousand Leaves
0ebe407221
Make DataNode not panic if stale segment is not found during flush ( #18322 )
...
/kind improvement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-07-20 15:06:30 +08:00
Jeng.Gwan
638f6c36e9
Support to get real row count of segment ( #18115 )
...
Signed-off-by: xaxys <zheng.guan@zilliz.com>
2022-07-18 09:58:28 +08:00
Xiaofan
6d82ef8c20
fix garbage collector err handling ( #18277 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-07-15 13:54:26 +08:00
congqixia
e5fe46124b
Fix isCompacting flag lost after segment clone ( #18248 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-07-13 10:08:28 +08:00
zhenshan.cao
72910d65e0
Fix inconsistency in DataCoord ( #18170 ) ( #18171 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-07-08 13:48:21 +08:00
wayblink
43ae6791e6
Make VChannelInfo read compatible with 2.0.2 ( #17995 )
...
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-07-07 21:58:22 +08:00
cai.zhang
0ad15a22c8
IndexCoord handle events correctly ( #17878 )
...
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
2022-07-07 14:44:21 +08:00
Ten Thousand Leaves
d4a1e94f32
Fix flush failure caused by empty stale segments ( #18097 )
...
/bug
issue: #18096
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-07-06 18:10:21 +08:00
zhenshan.cao
9bd35dd84e
Refine logs to reduce print ( #17759 ) ( #18028 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-07-04 11:10:20 +08:00
zhenshan.cao
0baeb609dc
Fix bugs in monitoring metrics ( #17894 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-06-30 17:26:19 +08:00
yah01
ef6859a4d6
Refactor handoff handler ( #17806 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Co-authored-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-06-29 23:26:19 +08:00
XuanYang-cn
ff3c4ab2a8
Fix compaction panic ( #17840 )
...
See also: #17823
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-06-29 16:54:17 +08:00
Ten Thousand Leaves
f0b036a35a
Move bulk load segment lock happen early ( #17612 )
...
issue: #17600
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-06-27 13:56:17 +08:00
bigsheeper
230404499d
Check duplication before adding channel info in DataCoord ( #17742 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-06-24 16:30:15 +08:00
Bingyi Sun
05273309e8
Drop empty sealed segment ( #17708 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-06-24 15:16:14 +08:00
XuanYang-cn
21b52bfcf2
Enhance channel manager ut ( #17682 )
...
Resolves : #17676
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-06-23 18:08:14 +08:00
SimFG
50d5d04552
Refine some codes about the datanode ( #17720 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-06-23 09:58:14 +08:00
Xiaofan
6fdf88f452
Fix compaction selection policy ( #17690 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-06-22 19:00:14 +08:00
Enwei Jiao
746aeea35b
fix typo ( #17668 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-06-21 18:54:13 +08:00
cai.zhang
ed94ecf847
Segments can't be compacted when they have reference lock ( #17649 )
...
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
2022-06-20 21:56:12 +08:00
Xiaofan
516d91470f
Fix Delta log loss after compaction ( #17634 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-06-19 20:12:11 +08:00
Letian Jiang
b15c24a554
Acquire reference lock of compacted from segments during index building ( #17619 )
...
- Accquire reference lock during index building
- Keep dropped segment record
Co-authored-by: Letian Jiang <letian.jiang@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2022-06-18 01:24:11 +08:00
wayblink
074ec3060a
Support return dropped segments info in GetSegmentInfo rpc ( #17617 )
...
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-06-17 18:24:12 +08:00
cai.zhang
53a49a9255
GetIndexState only get the state generated call CreateIndex ( #17549 )
...
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
2022-06-16 20:12:11 +08:00
wayblink
eb5b0b7fc8
Move SegmentInfo out of VchannelInfo, leave Id only to decrease kv size. Get complete SegmentInfo through RPC ( #17441 )
...
Resolves : #17233 #16047
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-06-16 12:00:10 +08:00
Xiaofan
54aeb077e9
Refine compaction selection poliy ( #17486 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-06-15 23:14:10 +08:00
cai.zhang
ea5041aec2
Acquiring the segment reference lock on task level ( #17544 )
...
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
2022-06-15 21:38:10 +08:00
Ten Thousand Leaves
51a9e54b21
Fix a bug that causes infinite loop ( #17550 )
...
This will cause AddNode() to keep its lock forever so DataCoord would just hang
issue: #17366
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-06-15 20:50:10 +08:00
Bingyi Sun
d4fdfab818
Fix VDC-510 ( #17554 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-06-15 19:58:10 +08:00
Ten Thousand Leaves
65ab847815
Add segment reference lock for bulk load ( #17457 )
...
So we can support DataQueryable and DataIndexed states
issue: #16607
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-06-15 12:20:10 +08:00
XuanYang-cn
e6225d923b
Cancel timers while adding new timer ( #17511 )
...
See also: #17335
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-06-13 14:46:08 +08:00
XuanYang-cn
2b5340581c
Skip remove if reassigns to the original node ( #17450 )
...
Fix ut race
See also: #15966 , #17432
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-06-10 10:54:07 +08:00
Xiaofan
6426e3d037
Split Etcd save operation into small txns ( #17386 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-06-08 15:08:07 +08:00
XuanYang-cn
1dbfebf827
Add debug log in removing assignment ( #17427 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-06-08 10:44:06 +08:00
groot
f31be941bb
Fix bulkload row count bug for multi datanodes ( #17384 )
...
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-06-07 16:56:07 +08:00
xige-16
b5c11a216d
Alter varChar type params's name to max_length ( #17409 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-06-07 15:58:06 +08:00
Xiaofan
af994b5e1c
Fix etcd watch error ( #17364 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-06-06 16:26:06 +08:00
XuanYang-cn
c70af73005
Fix datacoord set wrong state for node registering ( #17376 )
...
Fix datacoord datarace
See also: #17335
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-06-06 15:10:05 +08:00
shaoyue
76eaa3fc50
Support access s3 through AWS IAM role ( #17292 )
...
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2022-06-02 19:42:03 +08:00
Ten Thousand Leaves
f5e63177d2
Fix an issue with bulk load with multiple DataNodes ( #17324 )
...
issue: #17300
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-06-02 18:54:04 +08:00
XuanYang-cn
42f643e727
Make DataNode release rather than delete when reassign ( #17293 )
...
1. Reassgin now will assign to the original Node if no other nodes
avaliable
2. Make AddNode balance async: ToRealse + Reassign
See also: #16114 , #17270
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-06-02 13:56:04 +08:00
cai.zhang
bcf3b7426a
Add distributed lock for segment refgerence ( #16782 )
...
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
2022-05-31 16:36:03 +08:00
groot
affcc2acad
Fix bulkload entity number bug ( #17278 )
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2022-05-31 15:40:04 +08:00
congqixia
55f1e03ca7
Allow DataNode clean vchannel with meta error without panicking ( #17247 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-05-27 16:20:00 +08:00
Ten Thousand Leaves
5e1e7a6896
Fix an issue that causes flush hanging ( #17169 )
...
issue: #16628
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-05-25 14:34:00 +08:00
bigsheeper
9eeec4a2d5
Add collection load cache and InvalidateCollMetaCache by collID ( #16882 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-05-19 10:13:56 +08:00
SimFG
cf08b5aa11
Fix 'TestChannelManager_StateTransfer' ut when etcd is slow ( #17082 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-05-18 17:19:57 +08:00
XuanYang-cn
127dd34b37
Make proxy use roundrobin to choose replica ( #17063 )
...
Fixes : #17055
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-05-17 22:35:57 +08:00
Xiaofan
62658dcda6
Fix pulsar unscubsribe fail because of consumer not found ( #16839 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-05-09 12:07:52 +08:00
Ten Thousand Leaves
6b2d3c3ec1
Make each dataNode only accept one bulk load task simultaneously ( #16820 )
...
issue: #16674
/kind improvement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-05-07 14:05:52 +08:00
Enwei Jiao
67ce206cf0
make SegmentMaxLifetime configurable ( #16783 )
...
Signed-off-by: Enwei Jiao <jiaoew2011@gmail.com>
2022-05-06 23:41:51 +08:00