XuanYang-cn
7a6aa8552a
fix: add back existing datanode metrics ( #29360 )
...
See also: #29204
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-22 14:20:43 +08:00
XuanYang-cn
5c5f9aa05e
Enhance newDataSyncService ( #27277 )
...
- Add flowgraph.Assemble assembles nodes in flowgraph.go
- remove fgCtx in newDataSyncService
- Add newServiceWithEtcdTickler func, reduce param numbers to 3
- Remove unnecessary params
- config.maxQueueLength, config.maxParallelish
See also: #27207
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-09-27 11:07:25 +08:00
XuanYang-cn
916528f51a
Add initCtx in getDataSyncService ( #27199 )
...
Passing initCtx to all IO funcs in newDataSyncService,
so when ctx.Canceled, newDataSyncService would return.
See also: #25309
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-09-20 16:03:23 +08:00
XuanYang-cn
7f1ae35e72
Add timeout in dispatcher, AsConsumer and Seek ( #26686 )
...
See also: #25309
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-09-08 09:51:17 +08:00
XuanYang-cn
08fd28b30b
Only do gracefully stop when DN Stop ( #26399 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-08-21 11:16:20 +08:00
Enwei Jiao
66fdc71479
Refactor logs in DataCoord & DataNode ( #25574 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-07-14 15:56:31 +08:00
congqixia
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
jaime
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
aoiasd
1a485044a9
fix NumConsumers metric not work ( #23167 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-04-05 08:18:28 +08:00
smellthemoon
ce91f61e5a
Pre-create topics ( #21907 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-04-03 16:44:23 +08:00
jaime
d126f06946
Decouple mq module from internal proto definition ( #22536 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-03-04 23:21:50 +08:00
Enwei Jiao
697dedac7e
Use cockroachdb/errors to replace other error pkg ( #22390 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-26 11:31:49 +08:00
bigsheeper
d2667064bb
Add msgDispatcher to support sharing msgs for different vChannel ( #21917 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-02-13 16:38:33 +08:00
Enwei Jiao
89b810a4db
Refactor all params into ParamItem ( #20987 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-07 18:01:19 +08:00
jaime
709a0a94e9
Add some msgstream metrics ( #20296 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-11-07 10:15:02 +08:00
Enwei Jiao
956c5e1b9d
Make Params singleton ( #20088 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-04 14:25:38 +08:00
bigsheeper
d528aa3d90
Print seek postion time and seek elapse ( #20271 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-11-03 15:03:35 +08:00
Xiaofan
1f170ce1ba
Refine kafka consumer ( #19846 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-25 13:23:30 +08:00
congqixia
28aadc988d
Use vchannel name in datanode subName ( #18519 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-08-03 17:10:36 +08:00
XuanYang-cn
53734bef58
Add seek msgID in DataNode ( #17881 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-07-08 10:18:28 +08:00
Xiaofan
b657b10eb5
Remove useless log ( #16624 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-04-27 10:01:47 +08:00
Xiaofan
89b4a34892
Fix Data race in NodeID ( #16603 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-04-24 22:03:44 +08:00
zhenshan.cao
f3eeecf146
Reduce the number of metrics ( #16030 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-03-15 21:51:21 +08:00
Cai Yudong
3e862ecdfb
Merge msgChannelConfig and knowhereConfig into commonConfig ( #15843 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-03-04 11:17:56 +08:00
Xiaofan
bd7d294c6d
Refine log level in data ( #15821 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-03-02 15:35:55 +08:00
bigsheeper
13177a90af
Add prometheus metrics for DataNode ( #15650 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
2022-02-28 19:11:55 +08:00
Cai Yudong
8255b713ac
Move ProxySubName/QueryNodeSubName/DataNodeSubName to msgChannelConfig ( #15501 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-09 14:41:45 +08:00
Cai Yudong
4bccc85534
Use MsgChannelConfig in GlobalParams for all components ( #15163 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-02 00:35:43 +08:00
XuanYang-cn
87394e29b6
Fix compaction ut datarace ( #15267 )
...
Fixes : #15241
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-01-18 17:49:39 +08:00
Jiquan Long
ebb9b24b47
Improve name of flowgraph node ( #14538 )
...
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-12-30 10:33:46 +08:00
Cai Yudong
891c202b73
Use GlobalParamTable for all components ( #13885 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-12-23 18:39:11 +08:00
congqixia
7e91bcf115
Normalize datanode dminput node log ( #12128 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-19 14:21:13 +08:00
Xiaofan
d845153de4
Add Buffer for consumer channel ( #9578 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-10-15 20:31:16 +08:00
XuanYang-cn
dcd56a575d
[skip ci]Update license for flowgraph of dn ( #9962 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-10-15 18:07:09 +08:00
XuanYang-cn
aac6323cff
Use config as parameter ( #9707 )
...
Make functions in flowgraph lesser parameters
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-10-13 11:16:32 +08:00
congqixia
d8f490082c
Normalize dmInputNode seek log ( #9714 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-12 23:44:35 +08:00
congqixia
80ac456cf8
Normalize dmInputNode log ( #9652 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-10-11 21:54:33 +08:00
godchen
ac35aa7e34
Add flowgraph dmstream input node error check ( #9497 )
...
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-10-08 19:25:00 +08:00
XuanYang-cn
5bb832e6cc
[skip ci]Add comment on flowgraph dmstream inputnode ( #9359 )
...
Signed-off-by: Yang Xuan <xuan.yang@zilliz.com>
2021-10-06 23:02:08 +08:00
congqixia
db944cd09c
Refactor flowgraph and related invocation ( #8770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-09-29 20:19:59 +08:00
congqixia
2aa9deda1c
Improve logic when watch dm channel fails ( #8386 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-09-23 18:31:55 +08:00
Cai Yudong
d10c6abc9e
Apply for msgstream from pool when creating collection ( #7738 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-09-18 09:13:50 +08:00
Cai Yudong
4e23ed9a18
Revert apply physical channels from pool ( d5f7e358
and 3b960969
) ( #7687 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-09-10 13:00:00 +08:00
Cai Yudong
3b9609692b
Apply for msgstream from pool when creating collection ( #7377 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-08-31 18:35:58 +08:00
Cai Yudong
f4b53df727
Revert "Apply for msgstream from pool when creating collection ( #7323 )" ( #7376 )
...
This reverts commit e6de86a433
.
2021-08-31 11:07:58 +08:00
Cai Yudong
e6de86a433
Apply for msgstream from pool when creating collection ( #7323 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-08-31 09:47:58 +08:00
XuanYang-cn
b640627a38
Fix datanode restart bug ( #5976 ) ( #6004 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-23 12:26:10 +08:00
sunby
189ac881f3
Fix bugs ( #5676 )
...
* Remove redundant session startup
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Register datanode after start success
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* fix meta snap shot
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix datanode message stream channel
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Fix bugs when drop empty collection
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Fix bug of getting pchan statistics from task scheduler
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Fix i/dist/dataservice test code
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Fix epoch lifetime not applied
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* fix datanode flowgraph dd node
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Fix handle datanode timetick bug
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Remove repack function of dml stream
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* fix proxynode
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Apply extended seal policy
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* add check for time tick
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix check
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix the repack function of dml stream
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Fix the bug when send statistics of pchan
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Fix the repack function when craete dml stream
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* fix bugs
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* fix describe collection
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix bug when send timestamp statistics
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* fix data node
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Add length check before flush request
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* add log for data node
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix SaveBinlog bugs
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Add more log in datanode
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Put SegmentState.Flushing as the last one in enum to fit the client
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Fix params in GetInsertBinlogPaths
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Rename policy
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Remove unused ddl functions and fields
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Remove pchan when drop collection
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
* Add balanced assignment policy
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* fix master ut
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Add lock in session manager
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* add log for debug
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
* Fix some logic bug and typo
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
* Fix recover bugs
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Get collection scheme of a specific timestamp
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Change CheckPoint to SegmentInfo in VchannelInfo
Signed-off-by: sunby <bingyi.sun@zilliz.com>
* Recover Unflushed segment numOfRows
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
* Fix dataservice unit tests
Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:06:11 +08:00
XuanYang-cn
e7fd3aa926
Flowgraph recovery ( #5640 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-06-15 16:04:48 +08:00
congqixia
d0d845805b
Change WatchDmChannelsRequest proto ( #5577 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:04:48 +08:00