test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-05 05:18:52 +08:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity
15,536 Commits 96 Branches 105 Tags 186 MiB
5117017355
Commit Graph

208 Commits

Author SHA1 Message Date
bigsheeper
13177a90af
Add prometheus metrics for DataNode (#15650) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
 2022-02-28 19:11:55 +08:00
Xiaofan
be42627ab2
Move Logutil to Util dir (#15447) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-02-09 18:55:46 +08:00
Cai Yudong
8255b713ac
Move ProxySubName/QueryNodeSubName/DataNodeSubName to msgChannelConfig (#15501) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-09 14:41:45 +08:00
Cai Yudong
1f337891e1
Rename base_param to service_param, rename global_param to component_param (#15463) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-08 20:57:47 +08:00
XuanYang-cn
5eef9886b3
Add more specific log when exiting BackGroundGC goroutine (#15273) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-02-07 16:55:44 +08:00
Cai Yudong
f1ea9613ae
Add EtcdConfig and move PulsarConfig/RocksdbConfig/MinioConfig into BaseParamTable (#15434) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-07 10:09:45 +08:00
Cai Yudong
4bccc85534
Use MsgChannelConfig in GlobalParams for all components (#15163) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-02-02 00:35:43 +08:00
XuanYang-cn
008e08a996
Add flowgraph manager and event manager (#15102) 
1. Add flowgraph manager to manager start and release of dataSyncService
2. Add event manager to manager etcd watchInfo event
  - Make put event able to retry
  - Make delete event able to terminate the retry loop

See also: #14604, #14300

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-01-13 14:21:34 +08:00
XuanYang-cn
6e847ff194
Make dn handleChannelInfo log more consist (#14936) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-01-11 09:37:35 +08:00
Cai Yudong
b6a48817b0
Add MinioConfig in GlobalParams for all components (#15099) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-01-10 17:29:34 +08:00
Cai Yudong
7efb02a4c5
Use PulsarConfig in GlobalParams for all components (#15046) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-01-09 23:45:33 +08:00
Cai Yudong
f8f61fb116
Use BaseParams etcd configuration for all components (#15012) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-01-07 16:15:26 +08:00
zhenshan.cao
a5f8bdeb42
Fix bug: data race in MsgFactory (#14839) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-01-06 14:57:20 +08:00
zhenshan.cao
76a9687310
Adjust usage of lock in datanode (#14631) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Co-authored-by: yangxuan <xuan.yang@zilliz.com>
 2022-01-04 19:35:32 +08:00
zhenshan.cao
7abebe576c
Add more logs on write path (#14600) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-01-04 15:47:18 +08:00
Xiaofan
69087ff8dd
Support embedded etcd (#14333) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-12-29 14:35:21 +08:00
congqixia
a8c7e0c827
[skip e2e] Add comment for Compaction of DataNode (#14350) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-28 09:38:07 +08:00
zhenshan.cao
9a2c91d326
Fix bug: refresh config after session init (#14194) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-12-24 22:22:17 +08:00
Cai Yudong
891c202b73
Use GlobalParamTable for all components (#13885) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-12-23 18:39:11 +08:00
cai.zhang
9f23fc7f2a
Register the service when the component state is healthy (#13248) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2021-12-15 11:47:10 +08:00
Bingyi Sun
0c39e0370c
Move segment statistics update to datanode tt loop (#13233) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-12-15 10:53:16 +08:00
congqixia
1195f441b2
Update DataNode state before stop (#12946) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-08 14:19:04 +08:00
congqixia
8f9e62fa18
Apply DropVirtualChannel and FlushManager drop mode (#12563) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-02 16:39:33 +08:00
godchen
5e3e3b3b41
Remove grpc returned error (#12537) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-12-01 22:17:46 +08:00
dragondriver
b6d0150dc5
Fix health check when disconnect to etcd (#12292) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-11-26 11:39:16 +08:00
XuanYang-cn
8a8ebed23d
Fix DN GC bug (#12258) 
Fixes: #12249

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-25 09:43:15 +08:00
congqixia
4121e31df1
Send SIGINT to runner goroutine after etcd disconnects (#12163) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-22 16:23:17 +08:00
dragondriver
ee0f753f7a
Fix datarace between GetComponentStates and Register (#11935) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-11-19 13:57:12 +08:00
XuanYang-cn
9bd3664672
Fix compaction bug (#11816) 
This PR fixes:
- timeout in compaction
- BinlogIterator using rawID as PKField bug

See also: #11757

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-17 10:07:13 +08:00
congqixia
5edbb82610
Add session revoke (#11908) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-16 22:31:14 +08:00
XuanYang-cn
c4bb73dc74
Complete drop collection procedure in DN (#11552) 
drop, cancel compaction, flush all buffers and release fg

See also: #11426

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-11 20:56:49 +08:00
XuanYang-cn
3cd0584a04
Add compaction in datanode (#11353) 
See also: #9530

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-08 19:49:07 +08:00
sunby
c26cd79532
Add compaction (#9906) 
issue: #9904
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-11-05 22:25:00 +08:00
XuanYang-cn
225b2c66a0
Fix datanode ut using register bug (#10885) 
Resolves: #10881
See also: #8058

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-01 11:01:49 +08:00
XuanYang-cn
1e3fc5076f
Change etcdkv clientv3 into MetaKv interface (#10903) 
This pr:
- changed etcdkv clientv3 into MetaKv interface
- replaced fmt.Sprintf with path.Join for kv key

See also: #8058

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-01 10:19:55 +08:00
Xiaofan
2df2d488e1
Fix Unit test fatal fail (#10933) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-10-30 10:24:38 +08:00
Xiaofan
6fddcb3925
Fix Session checker for proxy (#10737) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-10-27 21:58:33 +08:00
congqixia
bd90a3831c
Allow flushed segments trigger flushe process (#10539) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-25 18:01:50 +08:00
godchen
31393d8801
[skip ci]Fix datanode comment typo error (#10502) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-23 16:57:35 +08:00
congqixia
7ce7cb7a5e
Add a timed flush trigger mechanism (#10197) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-20 15:02:36 +08:00
congqixia
0438ed2bfe
Use single signal chan to notify flush (#10075) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-18 12:34:34 +08:00
XuanYang-cn
9c706c6dc3
[skip ci]Update license on datanode (#9960) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-10-15 18:03:25 +08:00
godchen
bf71965f58
Check watchlist error (#9953) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-15 17:02:37 +08:00
congqixia
ec6e8e4127
Hide liveChn into sessionutil and fix liveness init order (#9860) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-14 16:40:35 +08:00
sunby
044b06421a
Use etcd to watch channel on datanode (#7115) 
issue: #6578
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-14 15:44:34 +08:00
godchen
a2dc0d8808
Init segment replica bloom filter (#9770) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-14 10:24:33 +08:00
congqixia
c90a97f95a
Fix Watch Channel canceled due to revision compacted (#9788) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-13 17:02:33 +08:00
XuanYang-cn
f0bedc7046
Add ut for data_node.go (#9489) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-10-11 18:50:30 +08:00
godchen
55ec988544
[skip ci]Fix data node comment lint (#9534) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-09 11:28:57 +08:00
groot
735b02cf2b
Add unittest for distributed/datanode (#9503) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-10-09 10:10:59 +08:00
XuanYang-cn
30395cb1b1
Add more specific comment in datanode (#9488) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-10-09 00:09:47 +08:00
godchen
445f5426d1
Check data node error (#9303) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-07 22:16:56 +08:00
congqixia
becf6c8f7d
Add DataNode implementation Assertion (#9144) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-03 19:46:05 +08:00
Xiaofan
2c2b800ff4
Change Log Name with NodeId (#8721) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-10-01 08:52:50 +08:00
zhenshan.cao
d1d5f9e409
Replace proto.MarshalTextString with proto.Marshal (#8542) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-09-29 20:26:00 +08:00
congqixia
db944cd09c
Refactor flowgraph and related invocation (#8770) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-29 20:19:59 +08:00
XuanYang-cn
3be6672753
Add flush channel for delete_node (#8762) 
Resolves: #8761

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-28 18:22:16 +08:00
congqixia
d374ea71ba
Unify liveness check in datacoord and datanode (#8480) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-28 16:10:04 +08:00
Xiaofan
585989935a
Revert the logger change (#8677) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-09-27 17:37:57 +08:00
Xiaofan
d38adda8c2
Rename log file based on node id (#8517) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-09-26 21:18:11 +08:00
XuanYang-cn
2539dd4593
[skip-ci]Fix golint in data_node.go (#8462) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-24 20:44:05 +08:00
congqixia
2aa9deda1c
Improve logic when watch dm channel fails (#8386) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-23 18:31:55 +08:00
XuanYang-cn
d816a81ee3
Make FlushSegments async (#8159) 
Resolves: #8122

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-23 16:03:54 +08:00
congqixia
6469d83072
Add datanode etcd alive check (#8363) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-23 14:29:54 +08:00
dragondriver
daf001f683
Expose created time and updated time to datanode metrics (#8173) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-17 21:32:47 +08:00
XuanYang-cn
3db76e8e59
[skip-ci]Add comment in data sync service (#8002) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-15 21:25:49 +08:00
XuanYang-cn
ce33855652
[skip-ci]Fix datanode comment typo (#7689) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-10 14:46:00 +08:00
sunby
06cee85a39
Add data node unit tests (#7622) 
Issue: #6357
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-09 16:40:00 +08:00
dragondriver
89748ac639
Expose metrics of data cluster (#7177) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-01 10:13:15 +08:00
Xiaofan
58b2089692
Import Embedded Etcd Kv Part 1(#7112) (#7174) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-08-24 09:45:51 +08:00
sunby
4f7c3e7d3a
Save binlog path in SegmentInfo (#6590) 
issue: #6589
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-08-19 13:00:12 +08:00
XuanYang-cn
86ca96a02e
Fix master branch unittest error (#7083) 
Resolves: #7080

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-08-13 14:00:14 +08:00
congqixia
e1b3273d1c
Add Datanode watch etcd channel (#6965) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-08-13 10:50:09 +08:00
XuanYang-cn
9d81a90402
Fix mem leak of flush cache in DataNode (#6862) 
Resolves: #6858

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-08-11 14:24:09 +08:00
XuanYang-cn
47df4a2e57
Fix DataNode BackGroudGC problem (#6551) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-08-03 09:59:24 +08:00
XuanYang-cn
769cbb292a
Update doc in datanode (#6194) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-29 17:34:13 +08:00
Cai Yudong
7dbf33b788
Update component names in config files (#5989) 
* rename master/indexService in .yaml

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename proxyNode to proxy in config files

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* move master.yaml to root_coord.yaml, move proxy_node.yaml to proxy.yaml

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* move proto/query_service.proto to proto/query_coord.proto

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* move proxy_node_manager.go to proxy_manager.go

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add query_coord.pb.go

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update ci

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-22 19:08:03 +08:00
Cai Yudong
d5f4ee6f44
Rename proto master/index_service/proxy_service (#5973) 
* rename proto proxy_service to proxy

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename proto index_service to index_coord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename proto master to root_coord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-22 16:14:09 +08:00
sunby
7f2d7c1da3
Rename dataservice to datacoordinator (#5912) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-21 18:22:13 +08:00
XuanYang-cn
688a587f4f
Add metrics in datanode (#5926) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-21 18:08:15 +08:00
Cai Yudong
92e429d812
Rename IndexService to IndexCoord (#5932) 
* rename package indexservice to indexcoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename indexservice to indexcoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix queryservice static-check

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* move distributed/indexservice to distributed/indexcoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* move internal/indexservice to internal/indexcoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename indexservice to indexcoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename MasterComponent to RootCoordComponent

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename master to rootcoord for queryservice

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename master to rootcoord for dataservice

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename master to rootcoord for datanode

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename master to rootcoord for proxynode

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename master to rootcoord for querynode

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename master to rootcoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename IndexService to IndexCoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename IndexService to IndexCoord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix rebase issue

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-21 17:28:03 +08:00
XuanYang-cn
441300140d
Tidy replica (#5920) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-21 16:00:22 +08:00
sunby
3cc0ab1345
Remove panic in NewDataSyncService (#5888) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-19 15:18:06 +08:00
sunby
33f5225968
Use physical channel to init msg stream (#5815) 
* Use physical channel to init msg stream

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix search collection error check (#1)

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: bigsheeper <954206947@qq.com>
 2021-06-18 16:02:05 +08:00
XuanYang-cn
18eb27aa7a
Tidy replica in datanode (#5805) 
- Remove not using functions
- Remove `initNode` logic in ParamTable
- Delete collection.go and collection_test.go

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-16 19:03:57 +08:00
Cai Yudong
0f4bd50ce3 Support ETCD endpoints array (#5755) 2021-06-15 16:36:36 +08:00
XuanYang-cn
dea0a4e522 Idempotent of FlushSegment (#5746) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 16:36:36 +08:00
XuanYang-cn
ab2fd34a2f fix double messages bug in datanode recovery (#5733) 
* fix double messages bug in datanode recovery

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* remove debug log

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 16:14:29 +08:00
XuanYang-cn
46699c4c2b fix datanode dropcollection error (#5690) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 16:06:13 +08:00
sunby
189ac881f3 Fix bugs (#5676) 
* Remove redundant session startup

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Register datanode after start success

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix meta snap shot

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix datanode message stream channel

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix bugs when drop empty collection

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Fix bug of getting pchan statistics from task scheduler

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix i/dist/dataservice test code

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix epoch lifetime not applied

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* fix datanode flowgraph dd node

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix handle datanode timetick bug

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Remove repack function of dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix proxynode

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Apply extended seal policy

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add check for time tick

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix check

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix the repack function of dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix the bug when send statistics of pchan

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix the repack function when craete dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix bugs

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix describe collection

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix bug when send timestamp statistics

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix data node

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Add length check before flush request

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add log for data node

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix SaveBinlog bugs

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Add more log in datanode

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Put SegmentState.Flushing as the last one in enum to fit the client

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Fix params in GetInsertBinlogPaths

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Rename policy

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Remove unused ddl functions and fields

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Remove pchan when drop collection

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Add balanced assignment policy

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix master ut

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Add lock in session manager

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add log for debug

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix some logic bug and typo

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix recover bugs

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Get collection scheme of a specific timestamp

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Change CheckPoint to SegmentInfo in VchannelInfo

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Recover Unflushed segment numOfRows

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix dataservice unit tests

Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:06:11 +08:00
XuanYang-cn
e7fd3aa926 Flowgraph recovery (#5640) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 16:04:48 +08:00
XuanYang-cn
ac19711d74 Destroy DataNode when drop collections (#5638) 
* Destroy DataNode when drop collections

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* golanci-lint

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 16:04:48 +08:00
neza2017
0a82c6381f add auto flush for data node (#5609) 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 16:04:48 +08:00
congqixia
d0d845805b Change WatchDmChannelsRequest proto (#5577) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:04:48 +08:00
godchen
275881dbdb Remove address param (#5556) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-06-15 16:04:46 +08:00
XuanYang-cn
e96d39bf6a refactor datanode flowgraph (#5538) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 15:59:04 +08:00
XuanYang-cn
6c02ae4eb1 Remove Old msgposition logic (#5487) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 15:59:04 +08:00
congqixia
8657251f41 DataNode stores segment msgpack positions (#5472) 
* DataNode stores segment msgpack positions

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix typo caused deadlock

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Use MsgPack EndPositions for flush pos

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 15:59:04 +08:00
XuanYang-cn
7a8b34ee9f Remove not using code in DataNode (#5476) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 15:59:04 +08:00
XuanYang-cn
55eac3ee7f Complete Flush procedure in DataNode (#5454) 
* Complete Flush proceduer in DataNode

Resolves: #5220

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* fix unittest

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 15:59:04 +08:00