test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-05 05:18:52 +08:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity
16,495 Commits 96 Branches 105 Tags 186 MiB
2c25a2d649
Commit Graph

135 Commits

Author SHA1 Message Date
Ten Thousand Leaves
45be3deb3f
Implement RC, DC, DN calling path for import. (#16321) 
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-01 11:33:28 +08:00
Ten Thousand Leaves
0d54697d42
Implement ReportImport related logic. (#16312) 
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-03-31 13:51:28 +08:00
Ten Thousand Leaves
a23ca745bb
Add Etcd support for Import Tasks. (#16164) 
issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-03-25 11:03:25 +08:00
groot
1c4b949a1d
modify import rpc interface (#16129) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-03-22 15:11:24 +08:00
groot
89356ebcb9
Modify import rpc interfaces (#16006) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-03-11 17:13:59 +08:00
groot
5fdef607d5
Add rpc interfaces for import (#15930) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-03-09 18:33:59 +08:00
congqixia
54862a31b0
Extend ChannelWatchState in protobuf (#15847) 
Related #15846
/kind enhancement

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-03-03 18:43:56 +08:00
congqixia
5751759c85
Implement mix compaction logic (#15542) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-02-18 14:51:49 +08:00
congqixia
1dcd06cc78
Add log file info for all kind of segment binlogs (#13606) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-19 20:00:42 +08:00
Bingyi Sun
0c39e0370c
Move segment statistics update to datanode tt loop (#13233) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-12-15 10:53:16 +08:00
congqixia
06ced4d1d1
Add DropVirtualChannel for DataCoord (#12361) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-29 22:35:41 +08:00
bigsheeper
e245ba87a9
Add excluded segments for dropped segemnts (#12247) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-11-25 14:25:15 +08:00
congqixia
8e0c8febe9
Add dropped segment ids in vchaninfo (#12230) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-23 18:59:15 +08:00
Bingyi Sun
01be91116e
Add GetFlushState interface (#12170) 
issue: #12075
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-23 10:55:14 +08:00
congqixia
7e43b568ee
Add dropped ts and apply when gc scan dropped segments (#11838) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-16 14:23:21 +08:00
godchen
5357e301f4
Fix retrieve search error (#11501) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-11 00:54:45 +08:00
congqixia
e5f599b616
Add dropped flag in SaveBinlogPathRequest (#11519) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-09 22:27:37 +08:00
Bingyi Sun
c3f0c5a3ff
Add compaction interfaces in proxy (#11431) 
issue: #9904
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-09 14:47:02 +08:00
sunby
c26cd79532
Add compaction (#9906) 
issue: #9904
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-11-05 22:25:00 +08:00
godchen
d6bf0c928f
Get stats delta path from datacoord (#10362) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-22 14:31:13 +08:00
congqixia
7ce7cb7a5e
Add a timed flush trigger mechanism (#10197) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-20 15:02:36 +08:00
congqixia
fc22537774
Fix static-check fail due to proto (#10258) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-20 13:12:35 +08:00
sunby
54459ac3b3
Add compaction proto (#9874) 
issue: #9873
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-10-20 11:58:35 +08:00
congqixia
a7f7bff651
Modify SaveBinlogPaths proto to support delete logs (#10080) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-19 00:26:35 +08:00
XuanYang-cn
40d3562ec0
Fix comment on SegmentInfo message (#10066) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-10-18 21:27:09 +08:00
XuanYang-cn
98ed253e2c
Update SegmentInfo proto to support deltalogs (#9925) 
Resolves: #9922
See also: #9530

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-10-15 18:24:39 +08:00
godchen
937dbf4279
Add flushed segments (#9476) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-08 19:09:12 +08:00
congqixia
ffbfaf7f76
Apply collection start position when open segment (#8697) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-27 20:34:03 +08:00
bigsheeper
b7103357fd
Load failed if collection beyonds the memory limit (#7434) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-07 11:35:18 +08:00
dragondriver
89748ac639
Expose metrics of data cluster (#7177) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-01 10:13:15 +08:00
sunby
4d56d53c0b
Add a tool to move binlog path to SegmentInfo (#7243) 
issue: #7214
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-08-24 15:51:51 +08:00
sunby
4f7c3e7d3a
Save binlog path in SegmentInfo (#6590) 
issue: #6589
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-08-19 13:00:12 +08:00
congqixia
e1b3273d1c
Add Datanode watch etcd channel (#6965) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-08-13 10:50:09 +08:00
sunby
07b0989628
Add GetFlushedSegments in data coordinator (#6253) 
Send segment flush message by grpc and add a new interface
`GetFlushedSegments`

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-07-02 11:16:20 +08:00
sunby
46f86a0354
Return sealed segments when calling flush (#6011) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-23 16:56:11 +08:00
sunby
c863a193de
Rename filename from dataservice to datacoordinator (#5946) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-22 10:42:07 +08:00
sunby
7f2d7c1da3
Rename dataservice to datacoordinator (#5912) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-21 18:22:13 +08:00
sunby
0300b682f8
Remove deprecated interfaces in data coordinator (#5929) 
* Remove derecated interfaces in data coordinator

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Remove RegisterNode in data coordinator

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-21 17:40:05 +08:00
sunby
9246c663fc Add start position in segment (#5753) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 16:39:01 +08:00
XuanYang-cn
5bcb94294c Set start positions in datanode (#5757) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 16:36:36 +08:00
sunby
189ac881f3 Fix bugs (#5676) 
* Remove redundant session startup

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Register datanode after start success

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix meta snap shot

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix datanode message stream channel

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix bugs when drop empty collection

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Fix bug of getting pchan statistics from task scheduler

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix i/dist/dataservice test code

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix epoch lifetime not applied

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* fix datanode flowgraph dd node

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix handle datanode timetick bug

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Remove repack function of dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix proxynode

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Apply extended seal policy

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add check for time tick

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix check

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix the repack function of dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix the bug when send statistics of pchan

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix the repack function when craete dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix bugs

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix describe collection

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix bug when send timestamp statistics

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix data node

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Add length check before flush request

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add log for data node

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix SaveBinlog bugs

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Add more log in datanode

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Put SegmentState.Flushing as the last one in enum to fit the client

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Fix params in GetInsertBinlogPaths

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Rename policy

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Remove unused ddl functions and fields

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Remove pchan when drop collection

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Add balanced assignment policy

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix master ut

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Add lock in session manager

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add log for debug

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix some logic bug and typo

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix recover bugs

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Get collection scheme of a specific timestamp

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Change CheckPoint to SegmentInfo in VchannelInfo

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Recover Unflushed segment numOfRows

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix dataservice unit tests

Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:06:11 +08:00
sunby
f054fc9be2 Add GetRecoveryInfo in dataservice (#5629) 
When loading a partition, QueryNode fetches the binlogs need to be
loaded and the channels needed to be watched by GetRecoveryInfo

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 16:04:48 +08:00
sunby
3a5c8c4d3a Add seek position to WatchDmChannels response (#5601) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 16:04:48 +08:00
sunby
e7521afed5 Change SaveBinlogPath (#5576) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 16:04:48 +08:00
congqixia
d0d845805b Change WatchDmChannelsRequest proto (#5577) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:04:48 +08:00
congqixia
41794ec3a6 Change SaveBinLogPath proto & func (#5551) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby
38b1f7dabe Change the logic of getting vchannel position (#5502) 
Datanode send `SaveBinlogPath` request after every segment flush finish
with the binlog paths and dml/ddl position. But the flush of segments is
not sorted. So we sort the segments according to segment id and find the
largest segment id with not nil dml position which is the position of
the msgstream to recover.

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:04 +08:00
congqixia
607a8ffa1f Update WatchDmChannels signature (#5447) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 15:59:04 +08:00
yangxuan
ef563fda8f Updata flushe procedure 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby
c53afee616 DataNode scales flowgraph 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby
81ac20143b Add persistency of datanode cluster (#5387) 
We save channels registered in datanode in etcd and restore the cluster info after
restarting.

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby
3662b3f0e5 Refactor dataservice (#5400) 
Pick up grpc calls to grpc_handler.go and refactor code format

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 15:59:01 +08:00
sunby
50b3d39370
Add GetRecoveryInfo interface and proto (#5598) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-07 14:16:36 +08:00
congqixia
4aa6a8fe18
Dataservice use lastest grpc design and saves ddl/dml pos pair (#5348) 
Dataservice use lastest grpc design and saves ddl/dml pos pair

* Update SaveBinlogPaths grpc definition
* Saves segment ddl & dml flush pos pair(start, end)

See also #5220 

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-05-21 11:38:58 +00:00
sunby
02f2348500
Save segment allocation status to meta (#5184) 
To ensure segments being flushed correctly, we need to load 
the segment allocation status from meta after starting the service.
 2021-05-21 10:30:41 +00:00
sunby
c1efa53779
Remove redundant fields in SegmentInfo and related structure (#5280) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-05-21 06:51:25 +00:00
congqixia
38f5b0826f
Add SaveBinlogPaths grpc in dataservice (#5277) 
* Add SaveBinlogPaths grpc in dataservice

* fix merged compile error

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-05-20 11:34:45 +08:00
Xiangyu Wang
82ccd4cec0
Rename module (#4988) 
* Rename module

Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-04-22 14:45:57 +08:00
sunby
751c9d4e81 Refactor meta.go 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-04-09 09:55:04 +08:00
XuanYang-cn
4c98f4efbf Update binlog meta proto 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-03-20 09:50:42 +08:00
godchen
f3649f0419 Refactor interface and proto 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-03-12 14:22:09 +08:00
ThreadDao
ca916a5c47 Add ci nightly 
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
 2021-03-11 14:14:29 +08:00
sunby
aeb3038545 Remove GetCount in data service 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-03-08 10:49:44 +08:00
XuanYang-cn
1aafe86ffe Change MsgPosition Logic 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-02-04 11:19:48 +08:00
zhenshan.cao
1b616fb874 Add persistent_segmentinfo 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-02-03 18:55:00 +08:00
sunby
f3aad3a71c Change SegmentInfo 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-02 18:53:10 +08:00
sunby
ca9c2f5c12 Add GetCount and implement GetCollectionStatistics 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-02-02 14:25:58 +08:00
zhenshan.cao
3afa7dd50d Fix GetSegmentStates 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-02-02 10:58:39 +08:00
sunby
dbfe68512d Add dbID to data service proto 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-27 10:33:17 +08:00
neza2017
1fe099775d Refactor data service 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-25 15:17:17 +08:00
neza2017
e89e89b894 Add main for masterservice 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-25 12:03:44 +08:00
sunby
54fcd486bb Refactor data service. 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-25 13:45:56 +08:00
XuanYang-cn
9fe672ab86 Enchance datanode interface 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-24 21:20:11 +08:00
XuanYang-cn
50e2369000 Change name and tag to ID 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-22 19:36:09 +08:00
neza2017
90a0b8b2f5 Update proto 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-22 11:44:02 +08:00
sunby
ef79b8d8c7 Add client and grpc service 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-22 11:07:07 +08:00
XuanYang-cn
377ee8687d Modify flush logic 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-22 09:36:40 +08:00
zhenshan.cao
2ec573d2f9 Recover delete docs 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-01-20 09:36:50 +08:00
cai.zhang
fb493cc235 Update interface name of GetIndexStates 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-01-19 18:32:57 +08:00
neza2017
918e279869 Set msg stream 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-01-19 16:14:13 +08:00
sunby
76a7684d96 Add data service 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-19 12:10:49 +08:00
xige-16
d22c2cd203 Add proto for query service loading collection 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2021-01-18 19:32:08 +08:00
XuanYang-cn
37688559f1 Fix multi rowdata writenode crash bug 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-16 15:06:19 +08:00
sunby
67e6ea8051 Add data service 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-01-15 17:09:41 +08:00
yukun
6e36f10a26 Fix rocksdb illegal instruction in some machine 
Signed-off-by: yukun <kun.yu@zilliz.com>
 2021-01-14 14:24:14 +08:00