sunby
9246c663fc
Add start position in segment ( #5753 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 16:39:01 +08:00
XuanYang-cn
5bcb94294c
Set start positions in datanode ( #5757 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 16:36:36 +08:00
sunby
189ac881f3
Fix bugs ( #5676 )
...
* Remove redundant session startup
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Register datanode after start success
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* fix meta snap shot
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix datanode message stream channel
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Fix bugs when drop empty collection
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Fix bug of getting pchan statistics from task scheduler
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Fix i/dist/dataservice test code
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Fix epoch lifetime not applied
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* fix datanode flowgraph dd node
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Fix handle datanode timetick bug
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Remove repack function of dml stream
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* fix proxynode
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Apply extended seal policy
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* add check for time tick
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix check
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix the repack function of dml stream
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Fix the bug when send statistics of pchan
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Fix the repack function when craete dml stream
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* fix bugs
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix describe collection
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix bug when send timestamp statistics
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* fix data node
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Add length check before flush request
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* add log for data node
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix SaveBinlog bugs
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Add more log in datanode
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Put SegmentState.Flushing as the last one in enum to fit the client
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Fix params in GetInsertBinlogPaths
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Rename policy
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Remove unused ddl functions and fields
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Remove pchan when drop collection
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Add balanced assignment policy
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* fix master ut
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Add lock in session manager
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* add log for debug
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix some logic bug and typo
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Fix recover bugs
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Get collection scheme of a specific timestamp
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Change CheckPoint to SegmentInfo in VchannelInfo
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Recover Unflushed segment numOfRows
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Fix dataservice unit tests
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:06:11 +08:00
sunby
f054fc9be2
Add GetRecoveryInfo in dataservice ( #5629 )
...
When loading a partition, QueryNode fetches the binlogs need to be
loaded and the channels needed to be watched by GetRecoveryInfo
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 16:04:48 +08:00
sunby
3a5c8c4d3a
Add seek position to WatchDmChannels response ( #5601 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 16:04:48 +08:00
sunby
e7521afed5
Change SaveBinlogPath ( #5576 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 16:04:48 +08:00
congqixia
d0d845805b
Change WatchDmChannelsRequest proto ( #5577 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:04:48 +08:00
congqixia
41794ec3a6
Change SaveBinLogPath proto & func ( #5551 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
sunby
38b1f7dabe
Change the logic of getting vchannel position ( #5502 )
...
Datanode send `SaveBinlogPath` request after every segment flush finish
with the binlog paths and dml/ddl position. But the flush of segments is
not sorted. So we sort the segments according to segment id and find the
largest segment id with not nil dml position which is the position of
the msgstream to recover.
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
congqixia
607a8ffa1f
Update WatchDmChannels signature ( #5447 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
yangxuan
ef563fda8f
Updata flushe procedure
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
sunby
c53afee616
DataNode scales flowgraph
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
sunby
81ac20143b
Add persistency of datanode cluster ( #5387 )
...
We save channels registered in datanode in etcd and restore the cluster info after
restarting.
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
sunby
3662b3f0e5
Refactor dataservice ( #5400 )
...
Pick up grpc calls to grpc_handler.go and refactor code format
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:01 +08:00
sunby
50b3d39370
Add GetRecoveryInfo interface and proto ( #5598 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-07 14:16:36 +08:00
congqixia
4aa6a8fe18
Dataservice use lastest grpc design and saves ddl/dml pos pair ( #5348 )
...
Dataservice use lastest grpc design and saves ddl/dml pos pair
* Update SaveBinlogPaths grpc definition
* Saves segment ddl & dml flush pos pair(start, end)
See also #5220
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-05-21 11:38:58 +00:00
sunby
02f2348500
Save segment allocation status to meta ( #5184 )
...
To ensure segments being flushed correctly, we need to load
the segment allocation status from meta after starting the service.
2021-05-21 10:30:41 +00:00
sunby
c1efa53779
Remove redundant fields in SegmentInfo and related structure ( #5280 )
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-05-21 06:51:25 +00:00
congqixia
38f5b0826f
Add SaveBinlogPaths grpc in dataservice ( #5277 )
...
* Add SaveBinlogPaths grpc in dataservice
* fix merged compile error
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-05-20 11:34:45 +08:00
Xiangyu Wang
82ccd4cec0
Rename module ( #4988 )
...
* Rename module
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-04-22 14:45:57 +08:00
sunby
751c9d4e81
Refactor meta.go
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-04-09 09:55:04 +08:00
XuanYang-cn
4c98f4efbf
Update binlog meta proto
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-03-20 09:50:42 +08:00
godchen
f3649f0419
Refactor interface and proto
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-03-12 14:22:09 +08:00
sunby
aeb3038545
Remove GetCount in data service
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-03-08 10:49:44 +08:00
XuanYang-cn
1aafe86ffe
Change MsgPosition Logic
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-02-04 11:19:48 +08:00
zhenshan.cao
1b616fb874
Add persistent_segmentinfo
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-02-03 18:55:00 +08:00
sunby
f3aad3a71c
Change SegmentInfo
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-02-02 18:53:10 +08:00
sunby
ca9c2f5c12
Add GetCount and implement GetCollectionStatistics
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-02-02 14:25:58 +08:00
zhenshan.cao
3afa7dd50d
Fix GetSegmentStates
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-02-02 10:58:39 +08:00
sunby
dbfe68512d
Add dbID to data service proto
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-01-27 10:33:17 +08:00
neza2017
1fe099775d
Refactor data service
...
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-01-25 15:17:17 +08:00
neza2017
e89e89b894
Add main for masterservice
...
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-01-25 12:03:44 +08:00
sunby
54fcd486bb
Refactor data service.
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-01-25 13:45:56 +08:00
XuanYang-cn
9fe672ab86
Enchance datanode interface
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-01-24 21:20:11 +08:00
XuanYang-cn
50e2369000
Change name and tag to ID
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-01-22 19:36:09 +08:00
neza2017
90a0b8b2f5
Update proto
...
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-01-22 11:44:02 +08:00
sunby
ef79b8d8c7
Add client and grpc service
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-01-22 11:07:07 +08:00
XuanYang-cn
377ee8687d
Modify flush logic
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-01-22 09:36:40 +08:00
zhenshan.cao
2ec573d2f9
Recover delete docs
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-01-20 09:36:50 +08:00
cai.zhang
fb493cc235
Update interface name of GetIndexStates
...
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2021-01-19 18:32:57 +08:00
neza2017
918e279869
Set msg stream
...
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-01-19 16:14:13 +08:00
sunby
76a7684d96
Add data service
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-01-19 12:10:49 +08:00
xige-16
d22c2cd203
Add proto for query service loading collection
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-01-18 19:32:08 +08:00
XuanYang-cn
37688559f1
Fix multi rowdata writenode crash bug
...
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-01-16 15:06:19 +08:00
sunby
67e6ea8051
Add data service
...
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-01-15 17:09:41 +08:00
yukun
6e36f10a26
Fix rocksdb illegal instruction in some machine
...
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-01-14 14:24:14 +08:00