wayblink
a1232fafda
feat: Major compaction ( #33620 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-10 21:34:08 +08:00
chyezh
f53ab54c5d
enhance: async cgo utility ( #33133 )
...
issue: #30926 , #33132
- implement future-based cgo utility.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-09 22:55:53 +08:00
jaime
8858fcb40a
fix: fix loaded entity num is inaccurate ( #33521 )
...
issue: #33520
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-04 20:09:54 +08:00
jaime
0d99db23b8
fix: metrics leak on the coord nodes ( #33075 )
...
issue: #32980
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-05-20 22:03:39 +08:00
jaime
f48a7ff8ff
enhance: use Delete instead of DeletePartialMatch to remove metrics ( #33029 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-05-14 18:49:33 +08:00
jaime
ba625835bc
enhance: Add metrics for segment index files size ( #32979 )
...
issue:#32980
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-05-13 17:59:32 +08:00
congqixia
efa0c73c62
fix: Unify querynode metrics cleanup in collection release ( #32805 )
...
Related to #32803
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-07 15:41:29 +08:00
Bingyi Sun
fecd9c21ba
feat: LRU cache implementation ( #32567 )
...
issue: https://github.com/milvus-io/milvus/issues/32783
This pr is the implementation of lru cache on branch lru-dev.
Signed-off-by: sunby <sunbingyi1992@gmail.com>
Co-authored-by: chyezh <chyezh@outlook.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
2024-05-06 20:29:30 +08:00
jaime
3b5cbed86a
enhance: add db label for deletion and bulkinsert operation ( #32611 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-04-26 15:53:29 +08:00
chyezh
2586c2f1b3
enhance: use WalkWithPrefix api for oss, enable piplined file gc ( #31740 )
...
issue: #19095,#29655,#31718
- Change `ListWithPrefix` to `WalkWithPrefix` of OOS into a pipeline
mode.
- File garbage collection is performed in other goroutine.
- Segment Index Recycle clean index file too.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-25 20:41:27 +08:00
congqixia
72c172a7d7
enhance: Remove duplicated collectionID label for task latency ( #32308 )
...
`CollectionID` already exists in channel name, so remove it to save
metrics traffic.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-16 18:55:19 +08:00
SimFG
c012e6786f
feat: support rate limiter based on db and partition levels ( #31070 )
...
issue: https://github.com/milvus-io/milvus/issues/30577
co-author: @jaime0815
---------
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-04-12 16:01:19 +08:00
jaime
d4fd6c7283
enhance: add db label on binlog size metrics ( #32003 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-04-10 21:01:20 +08:00
SimFG
90bed1caf9
enhance: add the related data size for the read apis ( #31816 )
...
issue: #30436
origin pr: #30438
related pr: #31772
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-10 15:07:17 +08:00
chyezh
c9faa6d936
enhance: add more metrics for cache and search ( #31777 )
...
issue: #30931
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-10 10:55:17 +08:00
yihao.dai
23ba2a5388
enhance: Remove deprecated DataCoordNumStoredRowsCounter metric ( #31778 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-04-07 00:25:15 +08:00
jaime
bd853be8c7
enhance: Add db label for some usual metrics ( #30956 )
...
issue: #31782
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-04-02 14:27:13 +08:00
chyezh
1ad5ccc50f
enhance: add rg and db interface for segment and db/rg metric label ( #31715 )
...
issue: #30931
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-01 10:21:21 +08:00
wei liu
7c7375031d
enhance: Add metrics for task latency in querycoord scheduler ( #31405 )
...
This PR add metrics for task latency in querycoord scheduler, so if any
kind of task stuck, it's easy to figure out by metrics
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-20 19:29:06 +08:00
Jiquan Long
dc2cdbe387
enhance: add more metrics ( #31271 )
...
/kind improvement
fix : #31272
This pr add more metrics, which are:
- Slow query count, which the duration considered as slow can be
configurable;
- Number of deleted entities;
- Number of entities imported;
- Number of entities per collection;
- Number of loaded entities per collection;
- Number of indexed entities;
- Number of indexed entities, per collection, per index and whether it's
a vetor index;
- Quota states (LongTimeTickDelay, MemoryExhuasted, DiskQuotaExhuasted)
per database;
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-03-19 15:23:06 +08:00
congqixia
194a611814
enhance: Add metrics for querycoord current target cp lag ( #31391 )
...
See also #31390
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-19 14:07:05 +08:00
wei liu
5016031dc7
enhance: add metrics for load segment progress ( #31005 )
...
This PR add metrics for load segment progress:
1. add metrics for load segment/index concurrency
2. add metrics for load index latency
3. change load segment latency's time unit to ms
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-08 15:39:02 +08:00
yihao.dai
a434d33e75
feat: Add import scheduler and manager ( #29367 )
...
This PR introduces novel managerial roles for importv2:
1. ImportMeta: To manage all the import tasks;
2. ImportScheduler: To process tasks and modify their states;
3. ImportChecker: To ascertain the completion of all tasks and instigate
relevant operations.
issue: https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-01 18:31:02 +08:00
chyezh
211143c5e6
enhance: add basic information of milvus into metrics ( #29665 )
...
add basic build information and runtime component dependency into
metrics.
issue: #29664
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2024-01-29 15:47:02 +08:00
XuanYang-cn
fd19e419f9
fix: Use size bucket for compacted segment size metric ( #30028 )
...
See also: #29204
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-26 10:53:02 +08:00
XuanYang-cn
0ac37a4a0c
fix: Use longTime bucket for longer latency ( #30175 )
...
See also: #30174
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-22 19:18:55 +08:00
yihao.dai
c8a78465b4
enhance: Add a counter monitoring for the rate-limit requests ( #30109 )
...
Add a counter monitoring metric for the ratelimited rpc requests with
labels: proxy nodeID, rpc request type, and state.
issue: https://github.com/milvus-io/milvus/issues/30052
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-21 14:44:59 +08:00
XuanYang-cn
86f48861c1
fix: Add more throughput in related metrics ( #30038 )
...
This PR also fixes bugs in l0 compactor where
l0 results would never be removed from datanode
See also: #30099
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-19 11:34:54 +08:00
xige-16
9702cef2b5
feat: Support multiple vector search ( #29433 )
...
issue #25639
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-01-08 15:34:48 +08:00
XuanYang-cn
bebe1baf18
fix: Reigster CompactionTaskNum metrics ( #29640 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-04 17:16:45 +08:00
smellthemoon
b12c90af72
enhance:Add upsert vector metrics ( #29226 )
...
add some metrics.
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-03 10:06:48 +08:00
XuanYang-cn
4b406e5973
enhance: Add CompactionTaskNum metrics ( #29518 )
...
See also: #27606
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-28 15:46:55 +08:00
XuanYang-cn
fe04598900
enhance: Add compaction type label to metrics ( #29485 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-27 15:56:48 +08:00
XuanYang-cn
7a6aa8552a
fix: add back existing datanode metrics ( #29360 )
...
See also: #29204
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-22 14:20:43 +08:00
wei liu
cb0676153f
enhance: add metrics for stopping querynode balance progress ( #29201 )
...
This PR add three metrics to track the stopping balance progress.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-21 10:14:52 +08:00
aoiasd
010d8362ad
fix: datanode L0 segment num wrong ( #29050 )
...
relate: https://github.com/milvus-io/milvus/issues/27675
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-20 11:20:44 +08:00
congqixia
cbf0f9c527
enhance: change cp metric to absolute unix ts ( #29328 )
...
See also #29327
Change channel checkpoint metrics to unix seconds instead of checkpoint
timestamp lag value
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-20 11:04:45 +08:00
yihao.dai
43abe9ce31
enhance: Delete datanode metric after flowgraph released ( #29003 )
...
`DataNodeFlowGraphBufferDataSize` monitoring metric is leaked. This PR
resolve it.
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-12-07 17:00:35 +08:00
yah01
9b3e06ae86
enhance: add more metrics for level zero segments ( #29029 )
...
- Add SegmentNum metric for level zero segments
- Add level zero segments size metirc
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-07 14:34:35 +08:00
congqixia
dd682658da
enhance: Add channel checkpoint ts lag metric ( #28767 )
...
See also #28765
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-28 07:00:28 +08:00
aoiasd
8a4cfb7d6a
enhance: add l0 metric and fix datacoord no need drop l0 segment when flush ( #28373 )
...
relate: https://github.com/milvus-io/milvus/issues/27675
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-24 15:58:24 +08:00
congqixia
1d76565894
Add metrics for garbage collection ( #27303 )
...
Also fix second metrics usage in compaction
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-22 18:47:25 +08:00
SimFG
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
XuanYang-cn
ea2b909c6d
Fix historm metrics buckets range for index ( #26976 )
...
See also: #26743
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-09-13 17:21:18 +08:00
chyezh
791e6ef6c6
[Improvement] add pulsar metrics and fix timeout ( #26907 )
...
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2023-09-13 12:03:19 +08:00
yihao.dai
c6024a32f5
Add rate limit ratio monitoring ( #26709 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-03 20:47:02 +08:00
XuanYang-cn
ef75784715
Fix LoadSegmentLatency metric p99 ( #26761 )
...
See also: #26743
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-09-01 11:07:07 +08:00
XuanYang-cn
8d54509e54
Fix CompactionLatency metrics ( #26747 )
...
- Refine compactor logs
See also: #26743
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-08-31 17:35:03 +08:00
congqixia
d343888f3c
Fix pkg codecov & static-check script ( #26720 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-31 09:59:01 +08:00
yihao.dai
3c62acde05
Add coll and segState lebels to DataCoordNumStoredRows monitoring ( #26299 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-08-17 15:52:17 +08:00
