Commit Graph

37 Commits

Author SHA1 Message Date
Xiaofan
69087ff8dd
Support embedded etcd (#14333)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-12-29 14:35:21 +08:00
godchen
f31ed089b5
Add timeout for reader next (#12308)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-11-26 22:45:24 +08:00
godchen
e3f49858c5
Add subscription prefix (#12251)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-11-24 17:47:15 +08:00
Xiaofan
0aaa87a79a
Fix MsgStream inconsistent seek (#12042)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-11-24 10:25:15 +08:00
godchen
51b353b52c
Add msgstream reader (#12130)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-11-19 15:57:12 +08:00
Xiaofan
a4715996ef
Fix Pulsar seek to latest may block forever (#11128)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-11-05 14:55:44 +08:00
XuanYang-cn
5b79c5ee92
[skip ci]Update license for ms mq (#11260)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-04 19:12:03 +08:00
XuanYang-cn
91be4b10cc
Make mq_msgstream_test log more infos (#11168)
See also: #11146

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-04 15:58:17 +08:00
congqixia
be0daafdb0
Fix rocksmq SeekToLatest (#11029)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-01 19:35:51 +08:00
yukun
92815f4542
Change timestamp to timestamps in internalproto.DeleteRequest (#9802)
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-10-14 14:48:34 +08:00
xige-16
e99ecc8cab
Add ProduceMark interface to return messageID (#9556)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-10-11 11:38:57 +08:00
Xiangyu Wang
d6f96ec9fc
Add unit tests for mq_msgstream.go (#8812)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-09-29 09:54:04 +08:00
Xiangyu Wang
f6632acc4c
Add unit tests for mq_msgstream.go (#8792)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-09-28 22:02:15 +08:00
Cai Yudong
6ab00add27
Change expr_plan to primary_keys in internal proto DeleteRequest (#8655)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-09-27 14:17:58 +08:00
congqixia
e7117f67ee
Add BroadcastMark for Msgstream returning MessageIDs (#8654)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-09-27 14:10:09 +08:00
Cai Yudong
31590bde2c
Rename ChannelID to ShardName to fix build fail (#8646)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-09-27 11:27:30 +08:00
Cai Yudong
ece0a06798
Update repack delete msg (#8631)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-09-27 10:35:58 +08:00
zhenshan.cao
b71bffe26f
Replace ChannelID with ShardName in DML Request (#8539)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-09-27 10:01:59 +08:00
Xiangyu Wang
cd980dd13b
Add unit test for mq_msgstream.go (#8548)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-09-26 13:09:56 +08:00
Cai Yudong
4fba738bec
Update internal proto DeleteRequest (#8466)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-09-24 15:21:55 +08:00
Xiangyu Wang
884d0d672e
Improve unit test in mq_msgstream.go (#8427)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-09-23 20:07:55 +08:00
yukun
847586eb95
Add rocksmq retention (#6617)
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-08-16 18:46:10 +08:00
XuanYang-cn
6c5be83ea1
Refactor NewEtcdKV API (#7075)
This PR mainly did the following 3 things:

- Refactor NewEtcdKV API
- Add unittests for `kv/etcd/`, raise the coverage
up to 94%
- Correct some APIs from version to revision

The old NewEtcdKV() API took `clientv3.Client` as
one of the input, making everyone who using this API
had to create a new `clientv3.Client`. So there're a lot
of repeated codes everywhere.

Meanwhile, someplace didn't even check whether the
`clientv3.New()` returned error or not, and the old API
also didn't check whether the clientv3.Client was Nil.
This caused issue#6955.

Ectd's version and revision have nothing alike, they
represent different concepts, but our APIs kind of
mixed them all up. So I correct those APIs' names.

Resolves: #6955

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-08-13 11:04:09 +08:00
godchen
c2ecce61c0
Make pulsar client singleton (#5782)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-06-15 19:03:56 +08:00
Cai Yudong
0f4bd50ce3 Support ETCD endpoints array (#5755) 2021-06-15 16:36:36 +08:00
neza2017
ac9dde7352 fix seek on pulsar msgstream (#5726)
* fix seek on pulsar msgstream

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* test seek on pulsar msgstream

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* msg-seek test test

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* add pure pulsar client seek test

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:14:29 +08:00
neza2017
351d87055a fix ttmsgstream (#5689)
* fix msgstream

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix asconsumer

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-06-15 16:06:13 +08:00
Cai Yudong
2ba93deb5e
Support master get segment info channel name from config file (#5606)
* optimize msgstream unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* get data service segment info channel name from config file directly

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-06-04 16:28:34 +08:00
godchen
918458a1be
Add msgstream seek (#5483)
* Add msgstream seek

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* change test name

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* remove file

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* resolve conversation

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-05-31 17:46:30 +08:00
Cai Yudong
c468481a36
Optimize MqTtMsgStream and unittest (#5498)
* code optimize

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add getTsMsgFromConsumerMsg

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename some variables for better readability

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* code optimize

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* code optimize

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add chanWaitGroup

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add chanTtMsgTime

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename lastTimeStamp to currTimeStamp

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add comments

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* code optimize

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* support unsynced ttMsg

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* code optimize

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update testcase

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update testcase

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update testcase

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add TestStream_PulsarTtMsgStream_3

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* remove debug log

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* change ttMsg logic back

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix ci block issue

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update testcase

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix codacy

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-05-31 11:31:30 +08:00
Cai Yudong
b414800d49
Update Seek interface (#5492)
* update Seek

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update Seek for mqTtMsgStream

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* use Retry in Seek

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix static-check

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-05-29 23:21:34 +08:00
xige-16
fce792b8bf
Add historical and streaming module in querynode (#5469)
* add historical and streaming

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* fix GetSegmentInfo

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* pass regression test

Signed-off-by: xige-16 <xi.ge@zilliz.com>

Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
2021-05-28 10:26:30 +08:00
xige-16
3630eec92c
Fix receving wrong msg after seek (#5441)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-05-27 02:38:37 +00:00
dragondriver
e5d4963ba0
Distinguish the dsl and expr in proxy and query node (#5118)
Pass a parameter `IsExpr` from proxy to query node, so query 
node can create search plan according to the variable `IsExpr`.
It makes `segcore` unnecessary to distinguish `dsl` or `expr`
using dynamic exception. When `IsExpr` is set to true, query
node will call `CreatePlanByExpr` according to the passed
information about expression. Otherwise query node will keep
still to use `CreatePlan` according to the `dsl` information. At
the same time, this pr adds some unittests to `CreatePlanByExpr`,
these unittests translate already exist case with `dsl` to case with
`expr`.

Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-05-07 07:20:47 +00:00
Xiangyu Wang
82ccd4cec0
Rename module (#4988)
* Rename module

Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-04-22 14:45:57 +08:00
xige-16
51b2b45e03 Add copyright for msgStream
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2021-04-19 11:30:19 +08:00
Xiangyu Wang
87a1e0b662 Reorganize msgstream
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-04-02 13:48:25 +08:00