Commit Graph

244 Commits

Author SHA1 Message Date
Cai Yudong
4fba738bec
Update internal proto DeleteRequest (#8466)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-09-24 15:21:55 +08:00
Xiangyu Wang
eb413f1396
Add comments for exposed structures and functions (#8435)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-09-23 21:57:55 +08:00
Xiangyu Wang
884d0d672e
Improve unit test in mq_msgstream.go (#8427)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-09-23 20:07:55 +08:00
godchen
3c8f031968
[skip ci]Add msgstream trace comment (#8349)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-09-23 17:11:54 +08:00
Xiangyu Wang
57618ac624
Remove unused codes in msgstream (#8368)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-09-23 16:29:55 +08:00
Xiangyu Wang
f203f750aa
Improve import statements in msgstream (#8237)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-09-18 19:06:09 +08:00
Xiangyu Wang
32c6033f4a
Add unit tests for msg.go (#8227)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-09-18 19:04:04 +08:00
Xiangyu Wang
c314b2b4a8
Remove unused label rules (#7929)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-09-18 18:19:51 +08:00
Xiangyu Wang
8b21a4b596
Add unit tests for msg.go (#8150)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-09-17 23:55:50 +08:00
Xiangyu Wang
b7b28abd77
Remove mem_msgstream that is no longer needed (#8155)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-09-17 18:43:54 +08:00
Xiangyu Wang
422b7dcf3e
Add unit tests for mq_factory (#8140)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-09-17 16:23:51 +08:00
Xiaofan
d6f0c5069f
[skip ci] Add comments for memory message queue (#8018)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-09-16 23:41:49 +08:00
Xiangyu Wang
0aa65f559a
Rename unit tests to meet new naming conventions (#8017)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-09-15 23:11:49 +08:00
congqixia
98732b3d43
Remove duplicated code in msgs (#7801)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-09-14 17:07:48 +08:00
Xiangyu Wang
60c11c85ad
[skip ci]Rename unit tests (#7844)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-09-13 21:24:58 +08:00
Xiangyu Wang
494600c37c
Add unit tests for msg.go (#7691)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-09-10 14:24:02 +08:00
Xiangyu Wang
6b86a18a7e
Comments unused msg from msg.go (#7686)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-09-10 11:12:05 +08:00
Xiangyu Wang
42aafdc407
Remove unused msg from msg.go (#7612)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-09-09 19:56:00 +08:00
Xiangyu Wang
624ea2f676
Add unit tests for msg.go (#7659)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-09-09 19:03:40 +08:00
dragondriver
9b21b3b5fd
Initialize test framework of proxy (#7563)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-09-09 10:06:29 +08:00
Xiangyu Wang
2224099f22
Add unit tests for msg.go (#7577)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-09-08 16:54:00 +08:00
congqixia
76f5264e79
Fix MqMsgStream Seek logic (#7536)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-09-07 16:35:59 +08:00
Xiangyu Wang
79a0a5482f
Add unit test for mq_factory (#7477)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-09-06 14:59:21 +08:00
congqixia
d3027c0d28
Fix possible deadlock (#7428)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-09-02 14:56:10 +08:00
yukun
847586eb95
Add rocksmq retention (#6617)
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-08-16 18:46:10 +08:00
XuanYang-cn
6c5be83ea1
Refactor NewEtcdKV API (#7075)
This PR mainly did the following 3 things:

- Refactor NewEtcdKV API
- Add unittests for `kv/etcd/`, raise the coverage
up to 94%
- Correct some APIs from version to revision

The old NewEtcdKV() API took `clientv3.Client` as
one of the input, making everyone who using this API
had to create a new `clientv3.Client`. So there're a lot
of repeated codes everywhere.

Meanwhile, someplace didn't even check whether the
`clientv3.New()` returned error or not, and the old API
also didn't check whether the clientv3.Client was Nil.
This caused issue#6955.

Ectd's version and revision have nothing alike, they
represent different concepts, but our APIs kind of
mixed them all up. So I correct those APIs' names.

Resolves: #6955

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-08-13 11:04:09 +08:00
Xiangyu Wang
5e78fd6a97
[skip ci]Update approvers (#7063)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-08-12 19:36:09 +08:00
Xiaofan
aba234fa3f
Fix Msgstream exponential retry (#6807)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-08-03 10:39:24 +08:00
sunby
07b0989628
Add GetFlushedSegments in data coordinator (#6253)
Send segment flush message by grpc and add a new interface
`GetFlushedSegments`

Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-07-02 11:16:20 +08:00
neza2017
a518e408b1
add log when seek in msgstream (#6244)
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-07-01 13:46:16 +08:00
zhenshan.cao
ff4c6c7ff4
Fix travel timestamp and guarantee timestamp (#6234)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-06-30 21:02:13 +08:00
xige-16
ff93d1611f
merge retrive and search code in query node (#6227)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-06-30 17:50:15 +08:00
neza2017
b22ab71222
fix rmq (#6147)
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-26 22:26:20 +08:00
yukun
39614aa8eb
Add rocksmq_path in config and paramtable (#6099)
* Add rocksmq_path in config and paramtable

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add rocksdbPath in NewRmsFactory

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Change rdb default path to /vat/lib/milvus/rdb_data

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-06-25 19:44:11 +08:00
groot
a807dca132
Fix #6082 Unstable issue for standalone mode in CI (#6084)
* Fix #6082 Unstable issue for standalone mode in CI

Signed-off-by: yhmo <yihua.mo@zilliz.com>

* Fix #6082 Unstable issue for standalone mode in CI

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2021-06-24 19:28:07 +08:00
yukun
0f409da136
Fix standalone seek hang after restart (#6073)
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-06-24 16:44:07 +08:00
neza2017
bf068d84ad
fix index coord (#6042)
* fix index coord

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* remove memstream from RmsFactory

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-24 14:36:08 +08:00
yukun
b72e4c6372
Merge search_collection and retrieve_colletion into query_collection (#6037)
* Merge search_collection and retrieve_colletion into query_collection

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Fix static-check

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-06-23 20:26:10 +08:00
xige-16
c310c70c00
Assign values to msgPack's startPosition in the form of deepCopy (#5900)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-06-21 14:16:03 +08:00
XuanYang-cn
ab7f642740
Add alias in paramtable (#5878)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-19 12:38:06 +08:00
xige-16
e4c51aae36
Add recovery logic for querynode and queryservice (#5843)
* merge milvus/recovery2

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* add recovery logic in queryservice

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* debug smoke case

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* add etcd to querynode

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* fix release partition error

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* fix load balance error

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* debug querynode down and recovery

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* add log

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* fix showCollection

Signed-off-by: xige-16 <xi.ge@zilliz.com>

* skip smoke test search without insert

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-06-19 11:45:09 +08:00
godchen
c2ecce61c0
Make pulsar client singleton (#5782)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-06-15 19:03:56 +08:00
Cai Yudong
0f4bd50ce3 Support ETCD endpoints array (#5755) 2021-06-15 16:36:36 +08:00
neza2017
ac9dde7352 fix seek on pulsar msgstream (#5726)
* fix seek on pulsar msgstream

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* test seek on pulsar msgstream

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* msg-seek test test

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* add pure pulsar client seek test

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:14:29 +08:00
neza2017
351d87055a fix ttmsgstream (#5689)
* fix msgstream

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix asconsumer

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:06:13 +08:00
sunby
189ac881f3 Fix bugs (#5676)
* Remove redundant session startup

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Register datanode after start success

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix meta snap shot

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix datanode message stream channel

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix bugs when drop empty collection

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Fix bug of getting pchan statistics from task scheduler

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix i/dist/dataservice test code

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix epoch lifetime not applied

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* fix datanode flowgraph dd node

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix handle datanode timetick bug

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Remove repack function of dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix proxynode

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Apply extended seal policy

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add check for time tick

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix check

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix the repack function of dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix the bug when send statistics of pchan

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix the repack function when craete dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix bugs

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix describe collection

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix bug when send timestamp statistics

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix data node

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Add length check before flush request

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add log for data node

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix SaveBinlog bugs

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Add more log in datanode

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Put SegmentState.Flushing as the last one in enum to fit the client

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Fix params in GetInsertBinlogPaths

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Rename policy

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Remove unused ddl functions and fields

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Remove pchan when drop collection

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Add balanced assignment policy

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix master ut

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Add lock in session manager

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add log for debug

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix some logic bug and typo

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix recover bugs

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Get collection scheme of a specific timestamp

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Change CheckPoint to SegmentInfo in VchannelInfo

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Recover Unflushed segment numOfRows

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix dataservice unit tests

Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:06:11 +08:00
sunby
3662b3f0e5 Refactor dataservice (#5400)
Pick up grpc calls to grpc_handler.go and refactor code format

Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-06-15 15:59:01 +08:00
bigsheeper
cdbc6d2c94
Refactor query node and query service (#5751)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-15 12:41:40 +08:00
Cai Yudong
2ba93deb5e
Support master get segment info channel name from config file (#5606)
* optimize msgstream unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* get data service segment info channel name from config file directly

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-04 16:28:34 +08:00
Xiangyu Wang
f791ccb246 Update OWNERS file
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-06-03 20:01:24 +08:00