yihao.dai
86a36b105a
enhance: Tidy compaction executor ( #33778 )
...
Move compaction executor to compaction pacakge.
issue: https://github.com/milvus-io/milvus/issues/32451
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-14 14:34:01 +08:00
Xiaofan
36cbce4def
enhance: optimize datanode cpu usage under large collection number ( #33267 )
...
fix #33266
try to improve cpu usage by refactoring the ttchecker logic and caching
string
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-05-25 04:43:41 +08:00
XuanYang-cn
29b621f759
fix: Make compactor able to clear empty segments ( #32821 )
...
See also: #32553
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-05-13 18:21:32 +08:00
Xiaofan
1e47d7afc4
improve: change some frequent log to debug ( #32779 )
...
remove the frequent log "filter insert messages"
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-05-06 20:51:29 +08:00
congqixia
6ef677f79e
fix: Remove metrics after flowgraph closed ( #32515 )
...
See also #32403
`fg_buffer_size` was decreased after metrics removed in flowgraph
ddnode, which make metrics value negative.
This PR move remove metrics logic into `dataSyncService.Close`
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-23 17:33:23 +08:00
XuanYang-cn
aad3ed3835
fix: [cherry-pick]Skip changing meta if nodeID not match with channel ( #31672 )
...
See also: #31648
pr: #31665 , #31694
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-04-10 15:09:18 +08:00
XuanYang-cn
86f48861c1
fix: Add more throughput in related metrics ( #30038 )
...
This PR also fixes bugs in l0 compactor where
l0 results would never be removed from datanode
See also: #30099
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-19 11:34:54 +08:00
XuanYang-cn
5164377e68
fix: Skip updating checkpoint after dropcollection ( #29220 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-15 16:04:45 +08:00
XuanYang-cn
321c5c32e3
fix: Separate schedule and check results loop ( #28692 )
...
This PR:
- Separates compaction scheduler and check results loop So that slow in
check-loop doesn't influence execution.
- Cleans compaction tasks when drop a vchannel so dropped-channel's
compaction tasks won't be checked over and over again.
- Skips meta change when meta's already changed, avoid panic
- Remove not inuse injectDone(bool) parameter
See also: #28628 , #28209
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-29 10:50:29 +08:00
smellthemoon
0aa90de141
Reduce the goroutine in flowgraph to 2 ( #28233 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-11-13 10:50:17 +08:00
SimFG
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
congqixia
cc584551fb
Remove unused hash value setting in DDNode ( #26575 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-28 11:30:26 +08:00
congqixia
f8bcf60e55
Adjust some confusing Warning log to INFO ( #26356 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-15 17:49:33 +08:00
Enwei Jiao
66fdc71479
Refactor logs in DataCoord & DataNode ( #25574 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-07-14 15:56:31 +08:00
congqixia
41af0a98fa
Use go-api/v2 for milvus-proto ( #24770 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
congqixia
ba84f52119
Fix watcher loop quit and channel shouldDrop logic ( #23402 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-04-14 09:54:28 +08:00
jaime
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
congqixia
4008ee6d60
Remove datanode ddNode forward delta logic ( #23005 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-03-27 14:18:00 +08:00
jaime
d126f06946
Decouple mq module from internal proto definition ( #22536 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-03-04 23:21:50 +08:00
bigsheeper
d2667064bb
Add msgDispatcher to support sharing msgs for different vChannel ( #21917 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-02-13 16:38:33 +08:00
Enwei Jiao
fb42466c65
Use opentelemetry ( #21509 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-01-12 16:09:39 +08:00
Xiaofan
4b4944ecee
Support sync all segments while close ( #21421 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-01-06 14:49:36 +08:00
Enwei Jiao
89b810a4db
Refactor all params into ParamItem ( #20987 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-07 18:01:19 +08:00
jaime
039e9ce4bb
Refine msgstream interface ( #20832 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-12-05 20:55:17 +08:00
jaime
709a0a94e9
Add some msgstream metrics ( #20296 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-11-07 10:15:02 +08:00
Enwei Jiao
956c5e1b9d
Make Params singleton ( #20088 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-04 14:25:38 +08:00
Xiaofan
192151bc11
Refine Data Log ( #19996 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-25 19:31:30 +08:00
smellthemoon
581e1d8c35
Fill MsgBase ALL ( #19912 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2022-10-21 15:57:28 +08:00
SimFG
a55f739608
Separate public proto files ( #19782 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
Xiaofan
db3f4be49d
Simplify flow graph node to pipeline mode ( #19667 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-10 22:15:22 +08:00
SimFG
d7f38a803d
Separate some proto files ( #19218 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
bigsheeper
5cdd6ca251
Add quotaCenter, rateLimiter and rateCollector ( #18636 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-09-16 09:56:47 +08:00
congqixia
91663a6603
Fix bloom filter size not match ( #19140 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-09-13 14:12:31 +08:00
wei liu
9338ad45c5
fix insert meets vChannels reuse pChannel ( #18695 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2022-08-23 19:12:55 +08:00
congqixia
f0846fb79b
Handles DropPartitionMsg in datanode flowgraph ( #18292 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-07-15 17:12:27 +08:00
XuanYang-cn
d9e8231728
Filter sealed segments ( #18065 )
...
- Remove FilterThreshold in DataNode
- Alter filter logic in flowgraph DDNode
See also: #17943
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-07-07 18:26:20 +08:00
bigsheeper
cdbd75d4dd
Log delta channel timetick in DataNode ( #17947 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-06-30 17:44:18 +08:00
wayblink
eb5b0b7fc8
Move SegmentInfo out of VchannelInfo, leave Id only to decrease kv size. Get complete SegmentInfo through RPC ( #17441 )
...
Resolves : #17233 #16047
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-06-16 12:00:10 +08:00
bigsheeper
e0cbacba59
Check ignorable error to prevent unnecessary panic ( #17317 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-06-02 10:12:03 +08:00
bigsheeper
a4ea2fb18a
Handle errors in DataNode and QueryNode flow graph ( #17096 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-05-24 21:11:59 +08:00
Xiaofan
89b4a34892
Fix Data race in NodeID ( #16603 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-04-24 22:03:44 +08:00
Ten Thousand Leaves
289e468a7a
Make bulk load fully work ( #16512 )
...
issue: #15604
/kind enhancement
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-20 14:03:40 +08:00
xige-16
99984b88e1
Support delete varChar value ( #16229 )
...
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-04-02 17:43:29 +08:00
zhenshan.cao
f3eeecf146
Reduce the number of metrics ( #16030 )
...
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-03-15 21:51:21 +08:00
Cai Yudong
3e862ecdfb
Merge msgChannelConfig and knowhereConfig into commonConfig ( #15843 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-03-04 11:17:56 +08:00
jaime
7a3a721380
Reconstruct mqstream module ( #15784 )
...
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-03-03 21:57:56 +08:00
Xiaofan
bd7d294c6d
Refine log level in data ( #15821 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-03-02 15:35:55 +08:00
bigsheeper
13177a90af
Add prometheus metrics for DataNode ( #15650 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
2022-02-28 19:11:55 +08:00
Cai Yudong
4bccc85534
Use MsgChannelConfig in GlobalParams for all components ( #15163 )
...
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2022-02-02 00:35:43 +08:00
godchen
98232b4942
[skip e2e]Sort imports ( #14820 )
...
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-01-05 14:25:48 +08:00