godchen
bb7a0766fe
Add dependency factory ( #16204 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-04-07 22:05:32 +08:00
godchen
78557ca6d7
Replace minio kv with minio chunk manager ( #15936 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-03-17 18:03:23 +08:00
zhenshan.cao
f3eeecf146
Reduce the number of metrics ( #16030 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-03-15 21:51:21 +08:00
jaime
7a3a721380
Reconstruct mqstream module ( #15784 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-03-03 21:57:56 +08:00
Letian Jiang
2be217bcf4
Specify appropriate timestamp in initPKBloomFilter ( #15823 ) ( #15826 )
...
Signed-off-by: Letian Jiang <letian.jiang@zilliz.com>
2022-03-03 17:33:56 +08:00
Xiaofan
bd7d294c6d
Refine log level in data ( #15821 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-03-02 15:35:55 +08:00
bigsheeper
13177a90af
Add prometheus metrics for DataNode ( #15650 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
2022-02-28 19:11:55 +08:00
zhenshan.cao
a5f8bdeb42
Fix bug: data race in MsgFactory ( #14839 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-01-06 14:57:20 +08:00
godchen
e1eb721532
[skip e2e]Sort imports ( #14819 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-01-05 14:23:52 +08:00
zhenshan.cao
7abebe576c
Add more logs on write path ( #14600 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-01-04 15:47:18 +08:00
Cai Yudong
891c202b73
Use GlobalParamTable for all components ( #13885 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-23 18:39:11 +08:00
congqixia
b20a238c38
Fix newDataSyncService panicking with normal segment has no dml pos ( #12771 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-06 16:21:35 +08:00
congqixia
8f9e62fa18
Apply DropVirtualChannel and FlushManager drop mode ( #12563 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-02 16:39:33 +08:00
congqixia
e813867652
Implement Drop mode for flush manager ( #12469 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-01 10:11:39 +08:00
XuanYang-cn
8a8ebed23d
Fix DN GC bug ( #12258 )
...
Fixes : #12249
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-25 09:43:15 +08:00
congqixia
5e550a8793
Add close for flushManager ( #11874 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-17 10:39:15 +08:00
congqixia
579416e3d3
Apply retry for flush manager execution ( #11808 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-15 17:19:10 +08:00
XuanYang-cn
f3852c1db0
Add more specific logs ( #11724 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-15 15:25:09 +08:00
Bingyi Sun
5c8978e5d6
Fix drop segment bug ( #11709 )
...
issue: #11708
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2021-11-12 17:31:11 +08:00
XuanYang-cn
c4bb73dc74
Complete drop collection procedure in DN ( #11552 )
...
drop, cancel compaction, flush all buffers and release fg
See also: #11426
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-11 20:56:49 +08:00
congqixia
3ac7d7a9de
Add detailed comments for dataSyncService in DataNode ( #11285 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-05 22:59:04 +08:00
godchen
9186d5527e
Forward delete msg ( #11210 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-11-05 11:59:02 +08:00
XuanYang-cn
8863c01ff7
Make DN fail faster when watching an invalid chan ( #11102 )
...
See also: #11098
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-04 15:56:28 +08:00
congqixia
169af67eb1
Fix segments not removed from flushingCache ( #10738 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-27 16:06:24 +08:00
godchen
d2e0943f21
Fix delete node flush error ( #10549 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-25 18:49:20 +08:00
godchen
d6bf0c928f
Get stats delta path from datacoord ( #10362 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-22 14:31:13 +08:00
congqixia
ed4e516ce7
Save stats and delta logs from DataNode ( #10159 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-19 14:34:35 +08:00
congqixia
5737e0075f
Apply flush manager logic in datanode ( #10142 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-19 11:04:34 +08:00
congqixia
0438ed2bfe
Use single signal chan to notify flush ( #10075 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-18 12:34:34 +08:00
XuanYang-cn
880056e664
[skip ci]Update license for datesync ( #9959 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-10-15 18:01:38 +08:00
godchen
2e0f7dfe42
Check data sync service error ( #9856 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-14 17:40:42 +08:00
XuanYang-cn
aac6323cff
Use config as parameter ( #9707 )
...
Make functions in flowgraph lesser parameters
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-10-13 11:16:32 +08:00
Cai Yudong
84e568ee32
Support Delete in datanode ( #8505 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-10-11 16:31:44 +08:00
godchen
376be1b532
Add data sync service error check ( #9498 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-09 15:38:57 +08:00
godchen
937dbf4279
Add flushed segments ( #9476 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-08 19:09:12 +08:00
XuanYang-cn
3be6672753
Add flush channel for delete_node ( #8762 )
...
Resolves : #8761
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-09-28 18:22:16 +08:00
godchen
77a773dff0
Change segment replica struct ( #8608 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-09-27 14:38:00 +08:00
congqixia
2aa9deda1c
Improve logic when watch dm channel fails ( #8386 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-09-23 18:31:55 +08:00
XuanYang-cn
d816a81ee3
Make FlushSegments async ( #8159 )
...
Resolves : #8122
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-09-23 16:03:54 +08:00
Cai Yudong
d10c6abc9e
Apply for msgstream from pool when creating collection ( #7738 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-09-18 09:13:50 +08:00
XuanYang-cn
3db76e8e59
[skip-ci]Add comment in data sync service ( #8002 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-09-15 21:25:49 +08:00
Cai Yudong
4e23ed9a18
Revert apply physical channels from pool ( d5f7e358
and 3b960969
) ( #7687 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-09-10 13:00:00 +08:00
XuanYang-cn
64aad49959
Add ut in datasync service to 88% ( #7615 )
...
See also: #6357
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-09-09 15:36:01 +08:00
XuanYang-cn
a5767e05c6
Add ut for flowgraph_delete_node to 100% ( #7550 )
...
See also: #6357
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-09-08 10:41:59 +08:00
Cai Yudong
3b9609692b
Apply for msgstream from pool when creating collection ( #7377 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-08-31 18:35:58 +08:00
Cai Yudong
f4b53df727
Revert "Apply for msgstream from pool when creating collection ( #7323 )" ( #7376 )
...
This reverts commit e6de86a433
.
2021-08-31 11:07:58 +08:00
Cai Yudong
e6de86a433
Apply for msgstream from pool when creating collection ( #7323 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-08-31 09:47:58 +08:00
congqixia
65f168eac1
Fix datanode corner cases ( #7336 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-08-30 10:03:58 +08:00
godchen
b4c55f6b3c
Add delete node ( #7261 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-08-25 11:41:52 +08:00
sunby
4f7c3e7d3a
Save binlog path in SegmentInfo ( #6590 )
...
issue: #6589
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-08-19 13:00:12 +08:00
XuanYang-cn
f0a68704e2
Add unittests for getChannelName in DataNode ( #6346 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-07-08 15:52:40 +08:00
sunby
7f2d7c1da3
Rename dataservice to datacoordinator ( #5912 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-21 18:22:13 +08:00
XuanYang-cn
441300140d
Tidy replica ( #5920 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-21 16:00:22 +08:00
sunby
3cc0ab1345
Remove panic in NewDataSyncService ( #5888 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-19 15:18:06 +08:00
sunby
33f5225968
Use physical channel to init msg stream ( #5815 )
...
* Use physical channel to init msg stream
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* fix search collection error check (#1 )
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: bigsheeper <954206947@qq.com>
2021-06-18 16:02:05 +08:00
XuanYang-cn
5bcb94294c
Set start positions in datanode ( #5757 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 16:36:36 +08:00
XuanYang-cn
ab2fd34a2f
fix double messages bug in datanode recovery ( #5733 )
...
* fix double messages bug in datanode recovery
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* remove debug log
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 16:14:29 +08:00
XuanYang-cn
46699c4c2b
fix datanode dropcollection error ( #5690 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 16:06:13 +08:00
sunby
189ac881f3
Fix bugs ( #5676 )
...
* Remove redundant session startup
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Register datanode after start success
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* fix meta snap shot
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix datanode message stream channel
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Fix bugs when drop empty collection
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Fix bug of getting pchan statistics from task scheduler
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Fix i/dist/dataservice test code
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Fix epoch lifetime not applied
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* fix datanode flowgraph dd node
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Fix handle datanode timetick bug
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Remove repack function of dml stream
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* fix proxynode
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Apply extended seal policy
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* add check for time tick
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix check
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix the repack function of dml stream
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Fix the bug when send statistics of pchan
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Fix the repack function when craete dml stream
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* fix bugs
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix describe collection
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix bug when send timestamp statistics
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* fix data node
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Add length check before flush request
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* add log for data node
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix SaveBinlog bugs
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Add more log in datanode
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Put SegmentState.Flushing as the last one in enum to fit the client
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Fix params in GetInsertBinlogPaths
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Rename policy
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Remove unused ddl functions and fields
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Remove pchan when drop collection
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Add balanced assignment policy
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* fix master ut
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Add lock in session manager
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* add log for debug
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix some logic bug and typo
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Fix recover bugs
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Get collection scheme of a specific timestamp
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Change CheckPoint to SegmentInfo in VchannelInfo
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Recover Unflushed segment numOfRows
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Fix dataservice unit tests
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:06:11 +08:00
XuanYang-cn
e7fd3aa926
Flowgraph recovery ( #5640 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 16:04:48 +08:00
XuanYang-cn
ac19711d74
Destroy DataNode when drop collections ( #5638 )
...
* Destroy DataNode when drop collections
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* golanci-lint
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
neza2017
74c73fa75f
fix datanode ( #5635 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
neza2017
3c70675313
data node, save binlog to minIO, and let data service save these meta ( #5618 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
neza2017
0a82c6381f
add auto flush for data node ( #5609 )
...
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:04:48 +08:00
congqixia
d0d845805b
Change WatchDmChannelsRequest proto ( #5577 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:04:48 +08:00
XuanYang-cn
e96d39bf6a
refactor datanode flowgraph ( #5538 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
congqixia
68bf983f21
Add Collection ID in ddNode and filter logic ( #5496 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
XuanYang-cn
7a8b34ee9f
Remove not using code in DataNode ( #5476 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
XuanYang-cn
55eac3ee7f
Complete Flush procedure in DataNode ( #5454 )
...
* Complete Flush proceduer in DataNode
Resolves : #5220
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* fix unittest
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
sunby
c53afee616
DataNode scales flowgraph
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
XuanYang-cn
1c779f3efd
Refactor ddl binlog flush ( #5303 )
...
DDL won't auto-flush now
See also: #5289 , #5220
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-05-19 11:42:07 +00:00
XuanYang-cn
fc630bc1c8
Correct auto-flush behaviour ( #5282 )
...
Before this PR, DataNode considered auto-flush a valid flush
complete. It's wrong. So I open this PR to correct this behaviour
in DataNode.
Now binlog paths from auto-flush will be buffered in replica,
waiting until the manul flush to save into etcd all together.
See also: #5220 , #5268
A follow up job of #5271
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-05-18 11:45:00 +00:00
Cai Yudong
5f0006d0f9
Improve code coverage for distributed/datanode ( #5133 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-05-08 06:07:04 +00:00
Xiangyu Wang
82ccd4cec0
Rename module ( #4988 )
...
* Rename module
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-04-22 14:45:57 +08:00
XuanYang-cn
6deba1faa5
Add license in datanode package
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-04-19 15:16:33 +08:00
XuanYang-cn
91f5c48037
Fix datanode goroutine leak
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-03-30 09:47:27 +08:00
XuanYang-cn
51f669f1cb
Flush segments concurrently & change etcd meta
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-03-23 18:50:13 +08:00
zhenshan.cao
c2734fa55f
Fix bug and enchance system
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-03-22 16:36:10 +08:00
quicksilver
74154a11a4
Fix deploy error during regression stage
...
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
2021-03-05 16:52:45 +08:00
cai.zhang
f5977a1302
Change retry times and add retry logic
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2021-02-26 15:17:47 +08:00
XuanYang-cn
10374f15d7
Update log in datanode
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-02-26 10:13:36 +08:00
Cai Yudong
84110d2684
Add tests/benchmark and tests/python_test using new python SDK
...
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2021-02-25 17:35:36 +08:00
groot
dc3736281a
Integrate message stream
...
Signed-off-by: groot <yihua.mo@zilliz.com>
2021-02-08 14:30:54 +08:00
XuanYang-cn
a64c4efde9
Refactor collectionReplica in datanode
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-02-04 20:31:23 +08:00
Xiangyu Wang
fd9e42d3ed
Update msgstream/unmarshal interface
...
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-02-03 17:30:10 +08:00
bigsheeper
abffdbff62
Remove partition tag
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-02-03 16:44:38 +08:00
XuanYang-cn
73a8e7de84
Fix datanode insertchannel refresh bug
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-02-03 15:18:05 +08:00
XuanYang-cn
9fe672ab86
Enchance datanode interface
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-01-24 21:20:11 +08:00
XuanYang-cn
377ee8687d
Modify flush logic
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-01-22 09:36:40 +08:00
XuanYang-cn
6a13386393
Add DataNode package
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-01-19 11:37:16 +08:00