Commit Graph

94 Commits

Author SHA1 Message Date
SimFG
9c1772f659
enhance: avoid to create many timer object in the target (#36570)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-29 19:23:16 +08:00
SimFG
ddadefcb62
enhance: get msg type from the msg header to reduce the Unmarshal usage (#36409)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-24 10:19:20 +08:00
SimFG
c50fe71163
fix: long buffering causes mq to be unable to receive messages. (#36420)
- issue: #36397

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-23 16:33:18 +08:00
congqixia
3b01b7dc9a
enhance: Produce messages of multiple topics in parallel (#36344)
Related to #36343

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-23 11:23:11 +08:00
SimFG
07bb071aaf
enhance: add the unmarshal method for the rbac msg (#35692)
/kind improvement
- issue: #35391

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-30 07:23:00 +08:00
SimFG
9dc1311f38
enhance: add the param to control whether to include the current msg (#35656)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-23 15:56:57 +08:00
congqixia
2736a8b88c
enhance: Update Makefile to generate mockery (#35517)
Some mockery cmd is out-of-date and fail to work. This PR update these
commands to match current pkg.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-16 18:44:53 +08:00
SimFG
b2cc4b0776
feat: add the rbac msg and send them to the replicate channel (#35392)
- issue: #35391

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-15 12:06:52 +08:00
yihao.dai
7721a28584
fix: Fix unstable TestDispatchToVchannels ut (#35163)
issue: https://github.com/milvus-io/milvus/issues/35150

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-01 14:56:13 +08:00
congqixia
de8a266d8a
enhance: Enable linux code checker (#35084)
See also #34483

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-30 15:53:51 +08:00
SimFG
0e41f104c5
enhance: use the collection id to group msg pack in the msg dispatcher (#34871)
- issue: #34870

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-29 11:45:49 +08:00
wei liu
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 (#34394)
issue: #34252

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-29 11:31:51 +08:00
yihao.dai
4a3d98d88c
enhance: Fast close msgdispatcher target (#34803)
/kind improvement

issue: https://github.com/milvus-io/milvus/issues/34075

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-29 00:25:49 +08:00
chyezh
39c7e06bc5
enhance: add message and msgstream msgpack adaptor (#34874)
issue: #33285

- make message builder and message conversion type safe
- add adaptor type and function to adapt old msgstream msgpack and
message interface

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-22 20:59:42 +08:00
chyezh
1bc3c0b925
enhance: implement balancer at streaming coord (#34435)
issue: #33285

- add balancer implementation
- add channel count fair balance policy
- add channel assignment discover grpc service

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-11 09:58:48 +08:00
yihao.dai
aef7664fc6
fix: Prevent dispatcher merging if curTs is 0 (#34562)
When the main dispatcher has not yet consumed data, curTs is 0. During
this time, merging dispatchers should not be allowed; otherwise, the
data of the solo dispatcher will be skipped.

issue: https://github.com/milvus-io/milvus/issues/34255

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-11 00:02:50 +08:00
yihao.dai
391d2aa5b5
fix: Fix unstable ut of msgdispatcher (#34485)
issue: https://github.com/milvus-io/milvus/issues/34484

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-09 10:08:11 +08:00
yihao.dai
43fd8d19c2
enhance: Check segment existence when FlushSegments and add some key logs (#34438)
Check if the segment exists during FlushSegments and add some key logs
in write path.

issue: https://github.com/milvus-io/milvus/issues/34255

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-06 08:50:11 +08:00
SimFG
0817802db8
enhance: use the key lock and concurrent map in the msg dispatcher client (#34278)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-05 15:11:31 +08:00
SimFG
c6e2900cfc
enhance: add the tick log for the tt msgstream seek method (#34397)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-04 10:18:09 +08:00
yihao.dai
ff51c7e628
fix: Fix panic due to failed to seek (#34229)
Converting the same msgposition's vchannel to a pchannel multiple times
would result in an invalid pchannel, leading to seek failure and panic.
This PR:
1. Make a copy of msgposition in msgdispatcher.
2. Check if channel is already a pchannel, no further channel conversion
is performed.

issue: https://github.com/milvus-io/milvus/issues/34221

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-01 16:08:12 +08:00
jaime
0426390f06
enhance: improve check health (#33800)
issue: #34264

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-01 10:16:06 +08:00
chyezh
d2bc4a53be
enhance: implement rmq and pulsar as wal (#34046)
issue: #33285

- use reader but not consumer for pulsar
- advanced test framework
- move some streaming related package into pkg

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-27 15:11:05 +08:00
jaime
9630974fbb
enhance: move rocksmq from internal to pkg module (#33881)
issue: #33956

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-25 21:18:15 +08:00
wayblink
a1232fafda
feat: Major compaction (#33620)
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-10 21:34:08 +08:00
Aldrin
8a4ef1b836
fix: Masked kafka credentials getting printed in logs (#33730)
issue: https://github.com/milvus-io/milvus/issues/33727

Signed-off-by: Aldrin <imagesai32@gmail.com>
2024-06-10 00:05:54 +08:00
XuanYang-cn
1629833060
enhance: Add consts of MsgDispatcher to configs (#33679)
See also: #33676

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-06-07 14:21:59 +08:00
SimFG
cb99e3db34
enhance: add the includeCurrentMsg param for the Seek method (#33326)
/kind improvement
- issue: #33325

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-27 10:31:41 +08:00
congqixia
12e8c6c583
enhance: Try LatestMessageID when checkpoint unmarshal fails (#33158)
See also #33122

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-21 16:59:39 +08:00
congqixia
c2ac692008
enhance: Add param item to ignore bad message id in checkpoint (#33123)
See also #33122

This pr add param item `mq.ignoreBadPosition` to control behavior when
mq failed to parse message id from checkpoint

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-20 11:29:37 +08:00
XuanYang-cn
39337e09b8
fix: Using zero serverID for metrics (#31518)
Fixes: #31516

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-04-01 16:55:19 +08:00
Xiaofan
b6fefee0cf
fix: etcd not connectable when auth enabled (#31633)
Fix etcd config source didn't respect auth enabled
Also removed pulsar recoverable error when pulsar return ConsumerBusy.
It could happen that pulsar didn't find the original consumer is dead
and recover takes some time.
fix #31631

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-04-01 15:23:19 +08:00
groot
ba6d33cd57
fix: Support TLS for kafka connection (#30468)
#27977

Add extra configurations in milvus.yaml to pass certificates for kafka.

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2024-02-28 18:43:07 +08:00
XuanYang-cn
e0f371c99b
fix: [skip e2e]use unique topic name for test pulsarCtl (#30364)
See also: #30363

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-02-01 11:05:04 +08:00
smellthemoon
e52ce370b6
enhance:don't store logPath in meta to reduce memory (#28873)
don't store logPath in meta to reduce memory, when service get
segmentinfo, generate logpath from logid.
#28885

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-18 22:06:31 +08:00
XuanYang-cn
f9508a4851
fix: Remove logging data when logging skip msg (#29707)
See also: #29696

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-15 11:40:51 +08:00
yihao.dai
8febbc79bb
fix: Fix unstable msg dispatcher ut (#29925)
This fix will not augment the execution time of unit tests, but solely
enhances tolerance for waiting for failure.

issue: https://github.com/milvus-io/milvus/issues/29921

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-14 10:40:52 +08:00
wei liu
1f759837c4
fix: remove Unnecessary lock in config manager (#29836)
issue: #29709 #291712
to avoid concurrent recursive RLock and Lock cause deadlock, This PR
remove the unnecessary lock in config manager

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-11 13:48:49 +08:00
smellthemoon
1c1f2a1371
enhance:change some logs (#29579)
related #29588

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-05 16:12:48 +08:00
aoiasd
9e6da45497
fix: Use uber atomic instead sync/atomic which only supported after go v1.20 (#29377)
relate: https://github.com/milvus-io/milvus/issues/29376

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-26 16:06:49 +08:00
yah01
a0e1a1eb31
feat: support enable/disable mmap for index (#29005)
support enable/disable mmap for index, the user could alter the index's
mode by `AlterIndex` method
related: https://github.com/milvus-io/milvus/issues/21866

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-21 18:07:24 +08:00
XuanYang-cn
48f506b077
fix: Seperate msgstream initctx with lifecycle ctx (#29308)
The close of msgstream should be controlled by Close() not ctx from
outside when init clients

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-21 11:56:45 +08:00
aoiasd
46a937ac43
fix: msgstream memory leak caused by config event don't deregister (#29266)
relate: https://github.com/milvus-io/milvus/issues/28620

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-20 20:32:44 +08:00
SimFG
de13865769
enhance: Add load/release partitions to replicate msg stream (#28399)
/kind improvement
issue: #25655

Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-11-23 15:38:24 +08:00
Enwei Jiao
3c3a57cb92
fix: Fix kafka config type error (#28642)
issue https://github.com/milvus-io/milvus/issues/28588

---------

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-11-23 14:36:22 +08:00
congqixia
371875f650
Fix MQ Factory timeout unit test (#28395)
This PR change timeout case to a longer duration in case of slow runner
Also change timeout case behavior of using timeout deadline instead of waiting
See also #28394

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-13 21:44:18 +08:00
jaime
6749957e71
Refine RPC call in unwatch drop channel (#27864)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-10-24 17:46:15 +08:00
SimFG
086220dd15
Add the mock factory of msg stream (#27834)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-10-20 20:14:08 +08:00
SimFG
9b0ecbdca7
Support to replicate the mq message (#27240)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-10-20 14:26:09 +08:00
zhenshan.cao
2b13078b14
Refine insert when send message (#27491)
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: Cai Zhang <cai.zhang@zilliz.com>
2023-10-18 21:12:08 +08:00