wei liu
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 ( #34394 )
...
issue: #34252
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-29 11:31:51 +08:00
XuanYang-cn
314f4d995b
enhance: Tidy dc channel manager ( #34515 )
...
See also: #34518
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-09 18:26:12 +08:00
jaime
21fc5f5d46
enhance: Remove datanode reporting TT based on MQ implementation ( #34421 )
...
issue: #34420
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-05 15:48:09 +08:00
jaime
d1f57aa4ba
enhance: remove deprecated code within channel manager ( #34340 )
...
issue: https://github.com/milvus-io/milvus/issues/33994
only remove deprecated code, no additional changes.
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-03 19:46:09 +08:00
jaime
d6afb31b94
enhance: make subfunctions of datanode component modular ( #33992 )
...
issue: #33994
also remove deprecated channel manager based on the etcd implementation
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-01 14:46:07 +08:00
jaime
9630974fbb
enhance: move rocksmq from internal to pkg module ( #33881 )
...
issue: #33956
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-25 21:18:15 +08:00
XuanYang-cn
5e39aa9272
enhance: Make channel meta able to writer 200k plus segments ( #33279 )
...
See also: #33125
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-05-28 12:33:42 +08:00
yiwangdr
b1eacb2ae8
feat: datacoord/node watch based on rpc ( #32036 )
...
issue: https://github.com/milvus-io/milvus/issues/25309
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-05-07 15:49:30 +08:00
congqixia
83da08c388
enhance: Use map instead of slice to maintain channel info ( #32273 )
...
See also #32165
`ChannelManager.Match` is a frequent operation for datacoord. When the
collection number is large, iteration over all channels will cost lots
of CPU time and time consuming.
This PR change the data structure storing datanode-channel info to map
avoiding this iteration when checking channel existence.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-16 15:57:19 +08:00
wei liu
0d849a6c0a
fix: fix collectionInfo leak in datacoord ( #32175 )
...
issue: #32029
lack of logic to clean collection info in datacoord's meta, This PR
clean collection info after drop channel, to avoid collection info leak
in datacoord
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-15 16:33:19 +08:00
XuanYang-cn
ae180d1628
enhance: Change ChannelManager to interface ( #29300 )
...
Rewrite cluster test
issue: #28854
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-25 19:24:46 +08:00
wei liu
fdbca10e23
fix: Fix channel manager bg checker exit when disable auto balance ( #28459 )
...
issue: #28454
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-20 18:20:22 +08:00
XuanYang-cn
a153950b10
Change channel to Interface ( #27839 )
...
This PR changes `*channel` into RWChannel interface
See also: #25309
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-13 11:16:18 +08:00
wei liu
5b45a138b1
disable auto balance when old node exists ( #28191 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-11-07 14:02:20 +08:00
jaime
6749957e71
Refine RPC call in unwatch drop channel ( #27864 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-10-24 17:46:15 +08:00
jaime
d2dbbbc11b
Reduce write lock scope in channel manager ( #27823 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-10-21 07:58:16 +08:00
congqixia
49516d44b4
Add ctx parameter and log tracer for watch and selectNodes ( #27809 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-20 04:22:11 +08:00
SimFG
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
congqixia
3c503afe7c
Use typeutil.ConcurrentMap instead of sync.Map ( #25846 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-24 10:23:01 +08:00
Xiaofan
63b6a4a639
use single instance ppol ( #25159 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-06-28 14:54:45 +08:00
yiwangdr
c7b851f870
add interface for non-watch metakv ( #25092 )
...
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2023-06-26 09:20:44 +08:00
yah01
ebd0279d3f
Check error by Error() and NoError() for better report message ( #24736 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-08 15:36:36 +08:00
congqixia
ba84f52119
Fix watcher loop quit and channel shouldDrop logic ( #23402 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-04-14 09:54:28 +08:00
jaime
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
MrPresent-Han
77c9e33e70
support dml channel balancer on datacoord ( #22324 ) ( #22377 ) ( #22692 )
...
Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
2023-03-20 10:01:56 +08:00
aoiasd
148a024e05
Add tickle for datacoord watch event ( #21193 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-02-15 16:20:34 +08:00
XuanYang-cn
a8db19fbdb
Fix ut multi-remove the timer ( #22021 )
...
See also: #22019
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-02-07 14:15:54 +08:00
XuanYang-cn
71cab36e37
Fix channelmanager ut ( #21140 )
...
See also: #17199
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-12-12 19:31:22 +08:00
Enwei Jiao
89b810a4db
Refactor all params into ParamItem ( #20987 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-07 18:01:19 +08:00
XuanYang-cn
243d8cff82
Provide different channelNames for channelmanager test ( #20969 )
...
Fixes : #20790
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-12-05 15:39:22 +08:00
Ten Thousand Leaves
0700e56008
Increase MaxWatchDuration and make it configurable ( #20884 )
...
/kind improvement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-11-29 17:19:14 +08:00
Eng Zer Jun
438790744d
test: use T.Setenv
to set env vars in tests ( #19649 )
...
This commit replaces `os.Setenv` with `t.Setenv` in tests. The
environment variable is automatically restored to its original value
when the test and all its subtests complete.
Reference: https://pkg.go.dev/testing#T.Setenv
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-10-11 18:43:23 +08:00
Jiquan Long
641afae22f
Watch channels with start positions ( #18744 )
...
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-08-22 14:42:52 +08:00
XuanYang-cn
21b52bfcf2
Enhance channel manager ut ( #17682 )
...
Resolves : #17676
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-06-23 18:08:14 +08:00
Enwei Jiao
746aeea35b
fix typo ( #17668 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-06-21 18:54:13 +08:00
XuanYang-cn
2b5340581c
Skip remove if reassigns to the original node ( #17450 )
...
Fix ut race
See also: #15966 , #17432
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-06-10 10:54:07 +08:00
XuanYang-cn
c70af73005
Fix datacoord set wrong state for node registering ( #17376 )
...
Fix datacoord datarace
See also: #17335
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-06-06 15:10:05 +08:00
XuanYang-cn
42f643e727
Make DataNode release rather than delete when reassign ( #17293 )
...
1. Reassgin now will assign to the original Node if no other nodes
avaliable
2. Make AddNode balance async: ToRealse + Reassign
See also: #16114 , #17270
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-06-02 13:56:04 +08:00
SimFG
cf08b5aa11
Fix 'TestChannelManager_StateTransfer' ut when etcd is slow ( #17082 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-05-18 17:19:57 +08:00
Xiaofan
be5d1f1243
Fix Datacoord ut when etcd is slow ( #16689 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-04-27 21:49:47 +08:00
godchen
bb7a0766fe
Add dependency factory ( #16204 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-04-07 22:05:32 +08:00
XuanYang-cn
5494f3c318
Remove offLineNodes from watch states check ( #16347 )
...
This PR also
- adds more information in log
- makes Session able to logged by zap.Any/zap.String
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-04-01 22:33:29 +08:00
XuanYang-cn
a77dd1034c
Adopt the extended state in DataCoord ( #16200 )
...
ChannelWatchInfo with ToWatch, ToRelease will trigger a timer.
ChannelManager now reacts to different ChannelWatch states.
- WatchSuccess > log this info
- WatchFailure/WatchTimeout > ToRelease
- ReleaseSuccess > Delete, reassign if not from DropCollection
- ReleaseFailure/ReleaseTimeout > Cleanup subscription and Delete,
reassgin if not from DropCollection.
Some Notes:
1. Reassignment will add this channel to buffer if there's only one node.
See also: #15846
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-03-28 22:33:27 +08:00
Xiangyu Wang
562d53fb15
Improve import statement ( #13724 )
...
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-12-20 14:30:55 +08:00
Xiangyu Wang
006a095370
Improve import statements ( #13437 )
...
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-12-15 19:57:13 +08:00
Bingyi Sun
175a656ff1
Unwatch dropped channel when init channel manager ( #11986 )
...
issue: #11558
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2021-11-17 23:25:12 +08:00
Bingyi Sun
54b40da4c5
Add segment drop logic ( #11621 )
...
issue: #11558
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2021-11-12 00:22:42 +08:00
XuanYang-cn
acb4dd2069
[skip ci]Update license for dc channel manager ( #10591 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-10-25 20:19:25 +08:00
sunby
044b06421a
Use etcd to watch channel on datanode ( #7115 )
...
issue: #6578
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-10-14 15:44:34 +08:00