test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-04 12:59:23 +08:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity
17,674 Commits 96 Branches 105 Tags 185 MiB
12a7747a95
Commit Graph

134 Commits

Author SHA1 Message Date
bigsheeper
13177a90af
Add prometheus metrics for DataNode (#15650) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
 2022-02-28 19:11:55 +08:00
zhenshan.cao
a5f8bdeb42
Fix bug: data race in MsgFactory (#14839) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-01-06 14:57:20 +08:00
godchen
e1eb721532
[skip e2e]Sort imports (#14819) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-01-05 14:23:52 +08:00
zhenshan.cao
7abebe576c
Add more logs on write path (#14600) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-01-04 15:47:18 +08:00
Cai Yudong
891c202b73
Use GlobalParamTable for all components (#13885) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-12-23 18:39:11 +08:00
congqixia
b20a238c38
Fix newDataSyncService panicking with normal segment has no dml pos (#12771) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-06 16:21:35 +08:00
congqixia
8f9e62fa18
Apply DropVirtualChannel and FlushManager drop mode (#12563) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-02 16:39:33 +08:00
congqixia
e813867652
Implement Drop mode for flush manager (#12469) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-12-01 10:11:39 +08:00
XuanYang-cn
8a8ebed23d
Fix DN GC bug (#12258) 
Fixes: #12249

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-25 09:43:15 +08:00
congqixia
5e550a8793
Add close for flushManager (#11874) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-17 10:39:15 +08:00
congqixia
579416e3d3
Apply retry for flush manager execution (#11808) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-15 17:19:10 +08:00
XuanYang-cn
f3852c1db0
Add more specific logs (#11724) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-15 15:25:09 +08:00
Bingyi Sun
5c8978e5d6
Fix drop segment bug (#11709) 
issue: #11708
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2021-11-12 17:31:11 +08:00
XuanYang-cn
c4bb73dc74
Complete drop collection procedure in DN (#11552) 
drop, cancel compaction, flush all buffers and release fg

See also: #11426

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-11 20:56:49 +08:00
congqixia
3ac7d7a9de
Add detailed comments for dataSyncService in DataNode (#11285) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-11-05 22:59:04 +08:00
godchen
9186d5527e
Forward delete msg (#11210) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-11-05 11:59:02 +08:00
XuanYang-cn
8863c01ff7
Make DN fail faster when watching an invalid chan (#11102) 
See also: #11098

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-11-04 15:56:28 +08:00
congqixia
169af67eb1
Fix segments not removed from flushingCache (#10738) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-27 16:06:24 +08:00
godchen
d2e0943f21
Fix delete node flush error (#10549) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-25 18:49:20 +08:00
godchen
d6bf0c928f
Get stats delta path from datacoord (#10362) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-22 14:31:13 +08:00
congqixia
ed4e516ce7
Save stats and delta logs from DataNode (#10159) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-19 14:34:35 +08:00
congqixia
5737e0075f
Apply flush manager logic in datanode (#10142) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-19 11:04:34 +08:00
congqixia
0438ed2bfe
Use single signal chan to notify flush (#10075) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-18 12:34:34 +08:00
XuanYang-cn
880056e664
[skip ci]Update license for datesync (#9959) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-10-15 18:01:38 +08:00
godchen
2e0f7dfe42
Check data sync service error (#9856) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-14 17:40:42 +08:00
XuanYang-cn
aac6323cff
Use config as parameter (#9707) 
Make functions in flowgraph lesser parameters

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-10-13 11:16:32 +08:00
Cai Yudong
84e568ee32
Support Delete in datanode (#8505) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-10-11 16:31:44 +08:00
godchen
376be1b532
Add data sync service error check (#9498) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-09 15:38:57 +08:00
godchen
937dbf4279
Add flushed segments (#9476) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-08 19:09:12 +08:00
XuanYang-cn
3be6672753
Add flush channel for delete_node (#8762) 
Resolves: #8761

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-28 18:22:16 +08:00
godchen
77a773dff0
Change segment replica struct (#8608) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-09-27 14:38:00 +08:00
congqixia
2aa9deda1c
Improve logic when watch dm channel fails (#8386) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-23 18:31:55 +08:00
XuanYang-cn
d816a81ee3
Make FlushSegments async (#8159) 
Resolves: #8122

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-23 16:03:54 +08:00
Cai Yudong
d10c6abc9e
Apply for msgstream from pool when creating collection (#7738) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-18 09:13:50 +08:00
XuanYang-cn
3db76e8e59
[skip-ci]Add comment in data sync service (#8002) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-15 21:25:49 +08:00
Cai Yudong
4e23ed9a18
Revert apply physical channels from pool (d5f7e358 and 3b960969) (#7687) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-10 13:00:00 +08:00
XuanYang-cn
64aad49959
Add ut in datasync service to 88% (#7615) 
See also: #6357

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-09 15:36:01 +08:00
XuanYang-cn
a5767e05c6
Add ut for flowgraph_delete_node to 100% (#7550) 
See also: #6357
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-08 10:41:59 +08:00
Cai Yudong
3b9609692b
Apply for msgstream from pool when creating collection (#7377) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-08-31 18:35:58 +08:00
Cai Yudong
f4b53df727
Revert "Apply for msgstream from pool when creating collection (#7323)" (#7376) 
This reverts commit e6de86a433.
 2021-08-31 11:07:58 +08:00
Cai Yudong
e6de86a433
Apply for msgstream from pool when creating collection (#7323) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-08-31 09:47:58 +08:00
congqixia
65f168eac1
Fix datanode corner cases (#7336) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-08-30 10:03:58 +08:00
godchen
b4c55f6b3c
Add delete node (#7261) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-08-25 11:41:52 +08:00
sunby
4f7c3e7d3a
Save binlog path in SegmentInfo (#6590) 
issue: #6589
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-08-19 13:00:12 +08:00
XuanYang-cn
f0a68704e2
Add unittests for getChannelName in DataNode (#6346) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-07-08 15:52:40 +08:00
sunby
7f2d7c1da3
Rename dataservice to datacoordinator (#5912) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-21 18:22:13 +08:00
XuanYang-cn
441300140d
Tidy replica (#5920) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-21 16:00:22 +08:00
sunby
3cc0ab1345
Remove panic in NewDataSyncService (#5888) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-19 15:18:06 +08:00
sunby
33f5225968
Use physical channel to init msg stream (#5815) 
* Use physical channel to init msg stream

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix search collection error check (#1)

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: bigsheeper <954206947@qq.com>
 2021-06-18 16:02:05 +08:00
XuanYang-cn
5bcb94294c Set start positions in datanode (#5757) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 16:36:36 +08:00
XuanYang-cn
ab2fd34a2f fix double messages bug in datanode recovery (#5733) 
* fix double messages bug in datanode recovery

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* remove debug log

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 16:14:29 +08:00
XuanYang-cn
46699c4c2b fix datanode dropcollection error (#5690) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 16:06:13 +08:00
sunby
189ac881f3 Fix bugs (#5676) 
* Remove redundant session startup

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Register datanode after start success

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix meta snap shot

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix datanode message stream channel

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix bugs when drop empty collection

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Fix bug of getting pchan statistics from task scheduler

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix i/dist/dataservice test code

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix epoch lifetime not applied

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* fix datanode flowgraph dd node

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix handle datanode timetick bug

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Remove repack function of dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix proxynode

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Apply extended seal policy

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add check for time tick

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix check

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix the repack function of dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix the bug when send statistics of pchan

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix the repack function when craete dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix bugs

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix describe collection

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix bug when send timestamp statistics

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix data node

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Add length check before flush request

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add log for data node

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix SaveBinlog bugs

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Add more log in datanode

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Put SegmentState.Flushing as the last one in enum to fit the client

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Fix params in GetInsertBinlogPaths

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Rename policy

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Remove unused ddl functions and fields

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Remove pchan when drop collection

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Add balanced assignment policy

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix master ut

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Add lock in session manager

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add log for debug

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix some logic bug and typo

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix recover bugs

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Get collection scheme of a specific timestamp

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Change CheckPoint to SegmentInfo in VchannelInfo

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Recover Unflushed segment numOfRows

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix dataservice unit tests

Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:06:11 +08:00
XuanYang-cn
e7fd3aa926 Flowgraph recovery (#5640) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 16:04:48 +08:00
XuanYang-cn
ac19711d74 Destroy DataNode when drop collections (#5638) 
* Destroy DataNode when drop collections

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* golanci-lint

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 16:04:48 +08:00
neza2017
74c73fa75f fix datanode (#5635) 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 16:04:48 +08:00
neza2017
3c70675313 data node, save binlog to minIO, and let data service save these meta (#5618) 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 16:04:48 +08:00
neza2017
0a82c6381f add auto flush for data node (#5609) 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 16:04:48 +08:00
congqixia
d0d845805b Change WatchDmChannelsRequest proto (#5577) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:04:48 +08:00
XuanYang-cn
e96d39bf6a refactor datanode flowgraph (#5538) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 15:59:04 +08:00
congqixia
68bf983f21 Add Collection ID in ddNode and filter logic (#5496) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 15:59:04 +08:00
XuanYang-cn
7a8b34ee9f Remove not using code in DataNode (#5476) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 15:59:04 +08:00
XuanYang-cn
55eac3ee7f Complete Flush procedure in DataNode (#5454) 
* Complete Flush proceduer in DataNode

Resolves: #5220

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* fix unittest

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 15:59:04 +08:00
sunby
c53afee616 DataNode scales flowgraph 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-15 15:59:04 +08:00
XuanYang-cn
1c779f3efd
Refactor ddl binlog flush (#5303) 
DDL won't auto-flush now

See also: #5289 , #5220 

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-05-19 11:42:07 +00:00
XuanYang-cn
fc630bc1c8
Correct auto-flush behaviour (#5282) 
Before this PR, DataNode considered auto-flush a valid flush
complete. It's wrong. So I open this PR to correct this behaviour
in DataNode.

Now binlog paths from auto-flush will be buffered in replica,
waiting until the manul flush to save into etcd all together.

See also: #5220, #5268
A follow up job of #5271 

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-05-18 11:45:00 +00:00
Cai Yudong
5f0006d0f9
Improve code coverage for distributed/datanode (#5133) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-08 06:07:04 +00:00
Xiangyu Wang
82ccd4cec0
Rename module (#4988) 
* Rename module

Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-04-22 14:45:57 +08:00
XuanYang-cn
6deba1faa5 Add license in datanode package 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-04-19 15:16:33 +08:00
XuanYang-cn
91f5c48037 Fix datanode goroutine leak 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-03-30 09:47:27 +08:00
XuanYang-cn
51f669f1cb Flush segments concurrently & change etcd meta 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-03-23 18:50:13 +08:00
zhenshan.cao
c2734fa55f Fix bug and enchance system 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-03-22 16:36:10 +08:00
quicksilver
74154a11a4 Fix deploy error during regression stage 
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
 2021-03-05 16:52:45 +08:00
cai.zhang
f5977a1302 Change retry times and add retry logic 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-02-26 15:17:47 +08:00
XuanYang-cn
10374f15d7 Update log in datanode 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-02-26 10:13:36 +08:00
Cai Yudong
84110d2684 Add tests/benchmark and tests/python_test using new python SDK 
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2021-02-25 17:35:36 +08:00
groot
dc3736281a Integrate message stream 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2021-02-08 14:30:54 +08:00
XuanYang-cn
a64c4efde9 Refactor collectionReplica in datanode 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-02-04 20:31:23 +08:00
Xiangyu Wang
fd9e42d3ed Update msgstream/unmarshal interface 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-02-03 17:30:10 +08:00
bigsheeper
abffdbff62 Remove partition tag 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-02-03 16:44:38 +08:00
XuanYang-cn
73a8e7de84 Fix datanode insertchannel refresh bug 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-02-03 15:18:05 +08:00
XuanYang-cn
9fe672ab86 Enchance datanode interface 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-24 21:20:11 +08:00
XuanYang-cn
377ee8687d Modify flush logic 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-22 09:36:40 +08:00
XuanYang-cn
6a13386393 Add DataNode package 
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
 2021-01-19 11:37:16 +08:00