Commit Graph

7118 Commits

Author SHA1 Message Date
neza2017
5966076880 fix data service (#5535)
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 15:59:04 +08:00
dragondriver
31b400a9f7 Refactor the workflow of receiving search result from query node (#5527)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-15 15:59:04 +08:00
godchen
54ab03e28f Fix queryservice connect error (#5526)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-06-15 15:59:04 +08:00
dragondriver
fca3eb7126 Add implementation of getMinTsStatistics function (#5524)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-15 15:59:04 +08:00
congqixia
62eaa0390a Add dataservice metrics with grpc prometheus middleware (#5523)
* Add dataservice metrics with grpc prometheus middleware

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* register server metrics

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
sunby
38b1f7dabe Change the logic of getting vchannel position (#5502)
Datanode send `SaveBinlogPath` request after every segment flush finish
with the binlog paths and dml/ddl position. But the flush of segments is
not sorted. So we sort the segments according to segment id and find the
largest segment id with not nil dml position which is the position of
the msgstream to recover.

Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
congqixia
31c4a4f5ed Simplify dd buffer & add lock (#5513)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
dragondriver
6ea11f67cb Dock channelsTimeTicker, channelsMgr with master service (#5509)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-15 15:59:04 +08:00
Cai Yudong
2cc330b117 Remove proxy service from master (#5510)
* remove proxy service

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* remove time sync

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* remove proxy service

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* remove proxy service

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* remove proxy service

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* remove proxy service from distributed/masterservice

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix static-check

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 15:59:04 +08:00
congqixia
68bf983f21 Add Collection ID in ddNode and filter logic (#5496)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
dragondriver
1f3c9d5dc7 Add statistics function to task scheduler (#5500)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-15 15:59:04 +08:00
XuanYang-cn
6c02ae4eb1 Remove Old msgposition logic (#5487)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
dragondriver
c213dc468a Add dqlChannelsMgr (#5488)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-15 15:59:04 +08:00
sunby
9a54f1f72d Watch channel if needed when allocating segmnets (#5482)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
dragondriver
0b41031ce4 Add channelsTimeTicker to proxy (#5462)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-15 15:59:04 +08:00
congqixia
a2dd16472d Implement random reassign and add test case (#5474)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
congqixia
8657251f41 DataNode stores segment msgpack positions (#5472)
* DataNode stores segment msgpack positions

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix typo caused deadlock

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Use MsgPack EndPositions for flush pos

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
sunby
43534ef63f Add unit tests in dataservice (#5478)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
XuanYang-cn
7a8b34ee9f Remove not using code in DataNode (#5476)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
Xiangyu Wang
e7368c9447 Add OWNERS file
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-06-15 15:59:04 +08:00
sunby
a14c35274f Fix bugs in dataservice (#5457)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
XuanYang-cn
55eac3ee7f Complete Flush procedure in DataNode (#5454)
* Complete Flush proceduer in DataNode

Resolves: #5220

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* fix unittest

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
dragondriver
59a3d83f97 Replace insertChannelsMap with channelsMgr (#5453)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-06-15 15:59:04 +08:00
congqixia
607a8ffa1f Update WatchDmChannels signature (#5447)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
sunby
cfdd98b70f Fix recovery branch after rebasing master (#5444)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
sunby
07c6a4a669 Add dataservice register and discovery (#5435)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
congqixia
e3956ad13f Change SaveBinLogPaths field2Path to repeated (#8)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 15:59:04 +08:00
yangxuan
ef563fda8f Updata flushe procedure
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
sunby
c53afee616 DataNode scales flowgraph
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 15:59:04 +08:00
sunby
b9cf4c1095 Fix recovery (#5417)
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
sunby
81ac20143b Add persistency of datanode cluster (#5387)
We save channels registered in datanode in etcd and restore the cluster info after
restarting.

Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:04 +08:00
sunby
3662b3f0e5 Refactor dataservice (#5400)
Pick up grpc calls to grpc_handler.go and refactor code format

Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:01 +08:00
congqixia
ef7339ceda DataService stores flush msgstream pos at VChannel granularity (#5397)
SaveBinLogPaths saves stream positions at VChannel granularity

resolves #5396

Signed-off-by: Congqi Xia congqi.xia@zilliz.com
2021-06-15 15:54:57 +08:00
del-zhenwu
cb58a85a5f
bug fix and enable case (#5765)
Signed-off-by: zw <zw@milvus.io>

Co-authored-by: zw <zw@milvus.io>
2021-06-15 15:13:59 +08:00
binbin
a2787cad9a
Add test case for search with expressions. (#5768)
Signed-off-by: Binbin Lv <binbin.lv@zilliz.com>
2021-06-15 15:11:57 +08:00
FluorineDog
bec9f2c182
Split segcore and plan proto for future feature (#5767)
* Split segcore and plan proto for future feature

Signed-off-by: fluorinedog <fluorinedog@gmail.com>

* lint

Signed-off-by: fluorinedog <fluorinedog@gmail.com>
2021-06-15 14:43:57 +08:00
bigsheeper
cdbc6d2c94
Refactor query node and query service (#5751)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-15 12:41:40 +08:00
FluorineDog
4165b761c9
fix not in bug (#5741)
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
2021-06-11 17:08:25 +08:00
yanliang567
902307327b
[skip ci] Update partition tests for fixed issues. (#5728)
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2021-06-11 10:02:50 +08:00
FluorineDog
006dae35c3
fix retrieve bug (#5727)
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
2021-06-11 00:06:55 +08:00
紫晴
faf5752096
Add test cases of connection interface (#5732)
* [skip ci] Add test cases of connection interface

Signed-off-by: wangting0128 <ting.wang@zilliz.com>

* [skip ci] Update test code

Signed-off-by: wangting0128 <ting.wang@zilliz.com>

* [skip ci] Update test codes

Signed-off-by: wangting0128 <ting.wang@zilliz.com>
2021-06-10 19:19:49 +08:00
quicksilver
9f4bb832b7
Add publish milvus images pipeline (#5722)
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
2021-06-10 17:33:49 +08:00
quicksilver
5b2f8d5552
[skip ci] Fix send email bug on jenkins pipeline failure (#5721)
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
2021-06-10 16:44:49 +08:00
quicksilver
f00b3bcbaa
[cronjob] Update periodically scheduled tasks to "50 22 * * *" (#5715)
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
2021-06-10 15:33:49 +08:00
yukun
e97936ba1c
Fix query hang bug (#5709)
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-06-10 14:33:49 +08:00
binbin
346366386a
[skip ci] Update test case. (#5678)
* [skip ci] Update test case.

Signed-off-by: Binbin Lv <binbin.lv@zilliz.com>

* [skip ci] Refine test case.

Signed-off-by: Binbin Lv <binbin.lv@zilliz.com>

* [skip ci] Resolve conflicts

Signed-off-by: Binbin Lv <binbin.lv@zilliz.com>
2021-06-10 11:46:49 +08:00
ThreadDao
f6dd244354
update insert cases for error check (#5700)
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2021-06-10 09:49:49 +08:00
FluorineDog
9bdf5acf61
Add query_boolean_expr docs (#5692)
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
2021-06-10 07:43:49 +08:00
yanliang567
a0ed3dc8c6
refaceor ttest_infor to log_config (#5685)
* [skip ci] refactor test_info to log_config

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>

* [skip ci] update chaos checker class

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>

* add pytest-parallel in requirements

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>

* [skip ci] update the error msg

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>

* [skip ci] limit the length of one log row

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2021-06-09 16:19:48 +08:00
ThreadDao
cad67a63c8
update collection property check (#5688)
Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2021-06-09 15:11:48 +08:00