Commit Graph

306 Commits

Author SHA1 Message Date
congqixia
ae2a301662
Add injection logic for FlushManager (#10580)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-25 20:17:34 +08:00
XuanYang-cn
21d98c2825
Add binlog io for compaction (#10536)
See also: #9530

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-10-25 20:13:51 +08:00
godchen
d2e0943f21
Fix delete node flush error (#10549)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-25 18:49:20 +08:00
congqixia
2713988ff2
Merge auto manual flush with same segment id (#10550)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-25 18:03:42 +08:00
congqixia
bd90a3831c
Allow flushed segments trigger flushe process (#10539)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-25 18:01:50 +08:00
XuanYang-cn
cd06f50645
Remove schema in delete codec (#10517)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-10-24 09:59:10 +08:00
godchen
31393d8801
[skip ci]Fix datanode comment typo error (#10502)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-23 16:57:35 +08:00
godchen
ffc0c07610
Change delete data primary key to int64 (#10438)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-22 15:37:12 +08:00
godchen
d6bf0c928f
Get stats delta path from datacoord (#10362)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-22 14:31:13 +08:00
congqixia
7ce7cb7a5e
Add a timed flush trigger mechanism (#10197)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-20 15:02:36 +08:00
godchen
6279fb817a
Fix stats log load with prefix error (#10216)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-19 20:50:35 +08:00
godchen
8daeb0d519
Change bloom filter use pk (#10193)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-19 20:18:47 +08:00
sunby
4031b8746f
Fix panic when flushing segment (#10182)
issue: #10180
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-10-19 19:06:37 +08:00
congqixia
ed4e516ce7
Save stats and delta logs from DataNode (#10159)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-19 14:34:35 +08:00
congqixia
5737e0075f
Apply flush manager logic in datanode (#10142)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-19 11:04:34 +08:00
congqixia
3711fcd82f
Add more info for Delta log compaction (#10083)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-18 18:24:45 +08:00
congqixia
0438ed2bfe
Use single signal chan to notify flush (#10075)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-18 12:34:34 +08:00
congqixia
90f88c6559
Add flush manager structure (#9986)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-18 10:38:35 +08:00
Xiaofan
d845153de4
Add Buffer for consumer channel (#9578)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-10-15 20:31:16 +08:00
XuanYang-cn
d0e6b52ce9
[skip ci]Update license for datanode (#9963)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-10-15 18:09:00 +08:00
XuanYang-cn
dcd56a575d
[skip ci]Update license for flowgraph of dn (#9962)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-10-15 18:07:09 +08:00
XuanYang-cn
9c706c6dc3
[skip ci]Update license on datanode (#9960)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-10-15 18:03:25 +08:00
XuanYang-cn
880056e664
[skip ci]Update license for datesync (#9959)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-10-15 18:01:38 +08:00
godchen
bf71965f58
Check watchlist error (#9953)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-15 17:02:37 +08:00
godchen
9e9ad7c7e6
Check pk filter merge error (#9952)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-15 16:58:42 +08:00
Xiaofan
a0b98dacda
Skip reconsume tt msgstream when datacoord restart (#9684)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-10-15 11:46:33 +08:00
godchen
2e0f7dfe42
Check data sync service error (#9856)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-14 17:40:42 +08:00
congqixia
ec6e8e4127
Hide liveChn into sessionutil and fix liveness init order (#9860)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-14 16:40:35 +08:00
sunby
044b06421a
Use etcd to watch channel on datanode (#7115)
issue: #6578
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-10-14 15:44:34 +08:00
yukun
92815f4542
Change timestamp to timestamps in internalproto.DeleteRequest (#9802)
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-10-14 14:48:34 +08:00
godchen
77a9b521c6
Check data sync service test error (#9854)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-14 14:12:34 +08:00
dragondriver
72d3d470ba
Use the server id as identifier in system topology graph (#9825)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-10-14 11:12:38 +08:00
godchen
a2dc0d8808
Init segment replica bloom filter (#9770)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-14 10:24:33 +08:00
congqixia
c50233639f
Add allocIDBatch for datanode allocator (#9782)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-13 22:12:32 +08:00
XuanYang-cn
c2c927bd55
Use reader in the right way (#9790)
Avoid to mantaining the position myself.

Make Read bytes more easy to read and maintain.

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-10-13 21:32:33 +08:00
congqixia
c90a97f95a
Fix Watch Channel canceled due to revision compacted (#9788)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-13 17:02:33 +08:00
XuanYang-cn
aac6323cff
Use config as parameter (#9707)
Make functions in flowgraph lesser parameters

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-10-13 11:16:32 +08:00
godchen
59ab0e441c
Add bloom filter for stats (#9630)
* Add bloom filter for stats

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* trigger GitHub actions

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-13 10:22:33 +08:00
congqixia
d8f490082c
Normalize dmInputNode seek log (#9714)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-12 23:44:35 +08:00
godchen
653ba91cc7
[skip ci ]Add join id path comment (#9718)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-12 19:23:39 +08:00
godchen
f496bc70c0
[skip ci]Fix delete node comment lint (#9717)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-12 18:32:33 +08:00
godchen
13c1a1c746
Add joinIDPath for reuse (#9650)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-12 17:20:33 +08:00
congqixia
80ac456cf8
Normalize dmInputNode log (#9652)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-11 21:54:33 +08:00
godchen
3e26c19bad
[skip ci]Fix segment replica comment (#9654)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-11 21:12:44 +08:00
XuanYang-cn
f0bedc7046
Add ut for data_node.go (#9489)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-10-11 18:50:30 +08:00
congqixia
e05877f828
Change datanode cache implementation to sync.Map (#9607)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-11 17:32:30 +08:00
Cai Yudong
84e568ee32
Support Delete in datanode (#8505)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-10-11 16:31:44 +08:00
xige-16
e99ecc8cab
Add ProduceMark interface to return messageID (#9556)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-10-11 11:38:57 +08:00
godchen
d0bc0cd697
Skip data sync service test (#9571)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-09 17:41:00 +08:00
godchen
5e7648fc7c
Check data node test error (#9532)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-09 15:45:15 +08:00