Commit Graph

63 Commits

Author SHA1 Message Date
Zhen Ye
cae9e1c732
fix: drop collection failed if enable streaming service (#37444)
issue: #36858

- Start channel manager on datacoord, but with empty assign policy in
streaming service.
- Make collection at dropping state can be recovered by flusher to make
sure that
 milvus consume the dropCollection message.
- Add backoff for flusher lifetime.
- remove the proxy watcher from timetick at rootcoord in streaming
service.

Also see the better fixup: #37176

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-07 10:26:26 +08:00
Zhen Ye
99dff06391
enhance: using streaming service in insert/upsert/flush/delete/querynode (#35406)
issue: #33285

- using streaming service in insert/upsert/flush/delete/querynode
- fixup flusher bugs and refactor the flush operation
- enable streaming service for dml and ddl
- pass the e2e when enabling streaming service
- pass the integration tst when enabling streaming service

---------

Signed-off-by: chyezh <chyezh@outlook.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-29 10:03:08 +08:00
congqixia
de8a266d8a
enhance: Enable linux code checker (#35084)
See also #34483

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-30 15:53:51 +08:00
smellthemoon
1c1f2a1371
enhance:change some logs (#29579)
related #29588

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-05 16:12:48 +08:00
smellthemoon
79c0edb1d8
enhance:Remove msgbase unnecessary assignments (#28511)
remove some unnecessary assignments, for the reason that
commonpbutil.NewMsgBase has default value.

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-11-24 15:02:39 +08:00
chyezh
85886f039e
[Fixup] msg context access data race when msgdispatcher (#26463)
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2023-08-21 19:16:22 +08:00
congqixia
3c503afe7c
Use typeutil.ConcurrentMap instead of sync.Map (#25846)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-24 10:23:01 +08:00
Enwei Jiao
66fdc71479
Refactor logs in DataCoord & DataNode (#25574)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-07-14 15:56:31 +08:00
congqixia
41af0a98fa
Use go-api/v2 for milvus-proto (#24770)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
Xiaofan
87d790f052
Fix upgrade casue panic (#23833)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-05-02 14:06:37 +08:00
Jiquan Long
156eba846f
Fix synced timetick not updated properly (#23697)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-04-29 16:02:38 +08:00
smellthemoon
912cf4ef0f
Change some configurations, include change the defaultChannelNum to 16 (#23617)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-04-27 14:26:35 +08:00
Jiquan Long
d85f673a95
Fix time tick lag too much (#23265)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-04-09 16:40:29 +08:00
jaime
c9d0c157ec
Move some modules from internal to public package (#22572)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
Jiquan Long
8530dfadb4
Fix metric RootCoordInsertChannelTimeTick (#23236)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-04-06 15:30:29 +08:00
Jiquan Long
556c1cfc8a
Fix confused timetick log (#22733)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-03-14 09:55:54 +08:00
jaime
d126f06946
Decouple mq module from internal proto definition (#22536)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-03-04 23:21:50 +08:00
Enwei Jiao
89b810a4db
Refactor all params into ParamItem (#20987)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-07 18:01:19 +08:00
Xiaofan
af66a0b621
Fix Time Tick session2ChanTs map use the same nodeID (#20537)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-11-14 14:41:11 +08:00
Jiquan Long
cc9dc0f0fa
Fix log of RootCoord (#20412)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-11-09 15:07:03 +08:00
jaime
709a0a94e9
Add some msgstream metrics (#20296)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-11-07 10:15:02 +08:00
smellthemoon
581e1d8c35
Fill MsgBase ALL (#19912)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2022-10-21 15:57:28 +08:00
SimFG
a55f739608
Separate public proto files (#19782)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
Jiquan Long
42d371fd41
Make ddl follow tt mechanism (#19279)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-09-21 15:46:51 +08:00
SimFG
d7f38a803d
Separate some proto files (#19218)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
Jiquan Long
a5e2d6b6fb
Refactor RootCoord (#18930)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: xaxys <tpnnghd@163.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: xaxys <tpnnghd@163.com>
2022-09-05 13:29:11 +08:00
congqixia
b62cb82ebe
Simplify Dml-DeltaChannel mapping logic (#18702)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-08-18 14:24:50 +08:00
cai.zhang
01fc411566
Handle etcd compacted error (#17886)
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
2022-06-30 18:54:19 +08:00
zhenshan.cao
0a953948af
Simplify monitoring metrics (#16687)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-04-27 23:03:47 +08:00
Xiaofan
fce8f6cfab
Remove Timetick channel (#16449)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-04-17 23:47:36 +08:00
Ten Thousand Leaves
0d54697d42
Implement ReportImport related logic. (#16312)
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-03-31 13:51:28 +08:00
zhenshan.cao
f3eeecf146
Reduce the number of metrics (#16030)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-03-15 21:51:21 +08:00
Cai Yudong
3e862ecdfb
Merge msgChannelConfig and knowhereConfig into commonConfig (#15843)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-03-04 11:17:56 +08:00
jaime
7a3a721380
Reconstruct mqstream module (#15784)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-03-03 21:57:56 +08:00
cai.zhang
9f4c59e621
Add prometheus metrics for RootCoord (#15684)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2022-03-02 21:11:57 +08:00
Cai Yudong
4bccc85534
Use MsgChannelConfig in GlobalParams for all components (#15163)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-02 00:35:43 +08:00
Xiaofan
885ecaf222
Refine time sync logic log (#15251)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-01-27 11:25:41 +08:00
Cai Yudong
356efee492
Rename timetickSync APIs and variables for better readability (#15264)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-18 14:47:36 +08:00
Cai Yudong
1b1d4e502b
Remove session from timetickSync (#15255)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-01-18 00:07:37 +08:00
Jiquan Long
d5e6adf381
Add log for timetickSync (#14517)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-12-30 10:32:03 +08:00
Cai Yudong
30382d8bf9
Let rootcoord share proxy.TimeTickInterval (#14326)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-27 21:38:45 +08:00
Cai Yudong
891c202b73
Use GlobalParamTable for all components (#13885)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-23 18:39:11 +08:00
XuanYang-cn
83df76d7af
[skip e2e]Update license for rc timetick (#13888)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-12-21 18:50:59 +08:00
congqixia
9af7e179f2
Fix zero length slice declaration in timeticksync.go (#12372)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-29 20:21:41 +08:00
Cai Yudong
cbbeb2a383
Add debug log when proxy idle for a long time (#12319)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-11-27 13:57:16 +08:00
Cai Yudong
858e95f377
Move dmlChannel and deltaChannel into timeticksync (#12254)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-11-25 10:07:15 +08:00
Cai Yudong
d4c297b1a8
Enhance dml channel operations (#12143)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-11-21 21:33:13 +08:00
Cai Yudong
b0f268e087
Add more debug log for proxy registry in rootcoord (#11899)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-11-16 19:21:49 +08:00
Cai Yudong
29e9adc7be
Fix golint warnings for rootcoord (#11872)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-11-16 14:27:11 +08:00
Cai Yudong
d10c6abc9e
Apply for msgstream from pool when creating collection (#7738)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-09-18 09:13:50 +08:00