Commit Graph

66 Commits

Author SHA1 Message Date
wei liu
701e7b5a6e
support pulsar tenant and auth (#19727)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2022-11-11 10:55:05 +08:00
jaime
e5419b3bc7
Remove kafka consumer ack (#20424)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-11-09 14:27:03 +08:00
SimFG
deb9963d0e
Delete unused messages for the mq (#20295)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-11-03 21:41:35 +08:00
congqixia
e1e3ed486e
Fix rmq consumer cancel may block forever (#20269)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-11-02 19:01:35 +08:00
jaime
48f5c60070
Refine msgstream close (#20194)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-31 15:13:33 +08:00
aoiasd
e470cd3dfa
Move Rocksmq page info log to Rokcsmq monitor (#20022)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2022-10-27 15:13:31 +08:00
Xiaofan
1f170ce1ba
Refine kafka consumer (#19846)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-25 13:23:30 +08:00
smellthemoon
581e1d8c35
Fill MsgBase ALL (#19912)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2022-10-21 15:57:28 +08:00
aoiasd
7fe8e50689
print some group info in rocksmq monitor (#19863)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2022-10-20 16:05:27 +08:00
SimFG
a55f739608
Separate public proto files (#19782)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
xige-16
a1db9038fb
Move disk index params to config file (#19714)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-10-14 17:51:24 +08:00
aoiasd
0a77561e78
Fix some message in rocksmq could not get acked timestamp (#19592)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2022-10-11 23:17:23 +08:00
Xiaofan
6826964e60
Rename some interface and files in id allocator (#19605)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-09 10:06:58 +08:00
bigsheeper
d6406387f8
Dedup only for dml messages in msgstream (#19591)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-09-30 17:52:55 +08:00
jaime
8ac30397dd
Disable kafka msgstream ut (#19260)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-09-19 16:24:51 +08:00
SimFG
d7f38a803d
Separate some proto files (#19218)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
Xiaofan
eabdc2b114
Disable pulsar batch and change background flush goroutine to larger interval (#18888)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-09-09 11:26:35 +08:00
Jiquan Long
a5e2d6b6fb
Refactor RootCoord (#18930)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: xaxys <tpnnghd@163.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: xaxys <tpnnghd@163.com>
2022-09-05 13:29:11 +08:00
Xiaofan
10d1140885
Fix RocksMQ flaky test (#18954)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-09-01 19:46:59 +08:00
Xiaofan
8e22d03cf3
Support trigger compaction everyday to eliminiate deleted data in rocksmq (#18752)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-08-29 10:56:55 +08:00
wgcn
d443b5420a
add user special kafka config (#18742)
Signed-off-by: wgcn <1026688210@qq.com>

Signed-off-by: wgcn <1026688210@qq.com>
Co-authored-by: wanggang11335 <wanggang11335@autohome.com.cn>
2022-08-25 11:02:53 +08:00
jaime
909e46b6c2
Fix LessOrEqualThan method of rocksmq (#18789)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-08-23 16:56:52 +08:00
wei liu
5c4f854271
fix load segment hangs (#18770)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2022-08-23 16:52:52 +08:00
jaime
c345df41a7
Fix kafka ut (#18527)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-08-04 16:02:35 +08:00
jaime
4a32c1bb8e
Enable kafka ut with a mocked kafka server (#18366)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-08-03 19:06:35 +08:00
jaime
6e7831470c
Revert settings of Pulsar address and refine config code (#18494)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-08-02 21:26:33 +08:00
jaime
3ec147acfe
Upgrade confluent kafka go version to 1.9.1 (#18293)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-07-22 13:28:28 +08:00
congqixia
d90308e9a0
Handle topic not exist in rqm & mqtt seek (#18314)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-07-20 15:42:29 +08:00
bigsheeper
1da970af18
Remove useless statsService in QueryNode (#17395)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-07-11 15:12:25 +08:00
Xiaofan
653dce4948
Fix pulsar error check (#18207)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-07-11 10:56:25 +08:00
congqixia
e61f61a9bd
Dedup insert msgs by MsgID in mqtt MsgPack (#18180)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-07-08 20:42:22 +08:00
Jiquan Long
6df47c7faf
Change pulsar compression type to zstd-faster (#18121)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-07-07 11:16:22 +08:00
Jiquan Long
44024c1b1d
Replace pulsar compression type zstd with lz4 (#18085)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-07-05 22:06:21 +08:00
Xiaofan
b31f937301
Fix unsubscribe with subscription not exist (#18042)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-07-05 10:12:20 +08:00
Xiaofan
a803f9e4b0
Fix Wait Group Race in Msgstream (#17931)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-06-30 10:38:18 +08:00
Bingyi Sun
24b3ad22d7
Fix msgstream close block. (#17880)
issue: #17513
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-06-29 16:24:20 +08:00
jaime
762e3b78a6
support configurable sasl mechanisms for kafka (#17542)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-06-29 10:32:17 +08:00
Xiaofan
e36fce1fd4
Add log for mqtt msgstream seek (#17640)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-06-20 15:18:13 +08:00
Xiaofan
d70f483806
Support rocksmq dynamic lru cache size (#17626)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-06-20 10:56:12 +08:00
Xiaofan
1f6fbf91b2
Fix pulsar unsubsribe issue (#17562)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-06-16 17:28:11 +08:00
jaime
dbef4c6ec2
Align kafka consume from latest with puslar and rocksmq (#17433)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-06-08 19:00:07 +08:00
xige-16
8c69790383
Fix lost delete msg caused by loadSegment after watchDeltaChannel (#17308)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-06-02 16:56:04 +08:00
jaime
5698bf4236
Add sasl configurations for kafka (#17323)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-06-02 12:12:03 +08:00
bigsheeper
e0cbacba59
Check ignorable error to prevent unnecessary panic (#17317)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-06-02 10:12:03 +08:00
zhenshan.cao
08a4dff41d
Add logic for search merging and a simple task scheduler for read tasks (#17022)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: bigsheeper <yihao.dai@zilliz.com>
2022-05-23 16:41:58 +08:00
Xiaofan
65269a4744
Remove rocksmq reader (#17149)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-05-23 13:55:59 +08:00
jaime
a20e0dfc74
Refactor kafka msgstream implementation (#17069)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-05-23 12:39:58 +08:00
jaime
328ab7ecf6
Fix kafka client hang after killing broker pod of parition leader (#16983)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-05-14 23:05:54 +08:00
Xiaofan
000c5ff3de
Fix msgstream unsubscription (#16883)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-05-10 19:43:52 +08:00
Xiaofan
62658dcda6
Fix pulsar unscubsribe fail because of consumer not found (#16839)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-05-09 12:07:52 +08:00