yihao.dai
c5088b8527
enhance: Add metrics to monitor import throughput and imported rows ( #36519 ) ( #36588 )
...
issue: https://github.com/milvus-io/milvus/issues/36518
pr: https://github.com/milvus-io/milvus/pull/36519
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-30 10:03:18 +08:00
XuanYang-cn
c1dab50fed
enhance: [cp]Add metrics for Delete entries num of L0seg ( #36227 )
...
- Add metrics *DataCoordL0DeleteEntriesNum*
- Remove metrics *DataCoordRateStoredL0Segment*
See also: #36147
pr: #36175
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-09-14 10:37:08 +08:00
congqixia
1cd8d1bd80
enhance: [2.4] Use stats Handler to record request/response size metrics ( #36107 ) ( #36118 )
...
Cherry-pick from master
pr: #36107
Related to #36102
This PR use newly added `grpcSizeStatsHandler` to reduce calling
`proto.Size` since the request & response size info is recorded by grpc
framework.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-10 17:13:08 +08:00
Gao
1687d64c46
enhance: [2.4] add hit segment num metrics for queryHook ( #35619 )
...
issue: #35576
pr: #35577
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-08-23 12:49:02 +08:00
Jiquan Long
d14d00b0cc
enhance: add metric to record disk quota ( #35305 ) ( #35306 )
...
https://github.com/milvus-io/milvus/issues/35177
pr: https://github.com/milvus-io/milvus/pull/35305
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-07 10:24:27 +08:00
cai.zhang
2534b30e39
enhance: [cherry-pick] Add monitoring metrics for task execution time in datacoord ( #35141 )
...
issue: #35138
master pr: #35139
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-02 19:46:16 +08:00
wei liu
2609e95cb5
enhance: add segment state on data coord bin log size metrics ( #35153 ) ( #35235 )
...
pr: #35153
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-02 18:28:14 +08:00
Jiquan Long
ae3657b23e
enhance: add metric to record maxInsertRate and queryNodeMemoryHighWaterLevel ( #35185 ) ( #35188 )
...
pr: https://github.com/milvus-io/milvus/pull/35185
fix: https://github.com/milvus-io/milvus/issues/35177
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-02 10:16:15 +08:00
PowderLi
fa9d606aa8
enhance: add metrics for restful apis ( #35083 )
...
issue: #34968
pr: #34969
Signed-off-by: PowderLi <min.li@zilliz.com>
2024-07-30 14:13:49 +08:00
Jiquan Long
2822d8729c
feat: record the number of times milvus turns into force-deny-writing states ( #34960 ) ( #34989 )
...
/kind improvement
fix : #34961
pr: #34960
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-07-25 15:43:09 +08:00
Jiquan Long
34873a7f76
feat: record the duration waiting in the proxy queue ( #34744 ) ( #34788 )
...
fix: https://github.com/milvus-io/milvus/issues/34743
pr: #34744
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-07-19 15:37:44 +08:00
wayblink
9dc98ac76d
enhance: [cherry-pick] Refine clustering compaction latency metrics ( #34539 )
...
issue: #34527
pr: #34528
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-11 17:43:43 +08:00
edward.zeng
3d29ebfffb
enhance: Refine default histogram metric bucket ( #34592 )
...
/kind improvement
pr: https://github.com/milvus-io/milvus/pull/34580
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
2024-07-11 16:51:35 +08:00
Chun Han
2f38483418
fix: lose partitionIDs when scalar pruning and refine segment prune ratio metrics( #30376 ) ( #34475 )
...
related: #30376
pr: https://github.com/milvus-io/milvus/pull/34477
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-07-08 19:44:13 +08:00
chyezh
a1a0a56f86
enhance: async search and retrieve in cgo ( #34200 )
...
issue: #33132
pr: #33133
other pr: #33228 , #34084 , #33946
- implement future-based cgo utility
- async search and retrieve in cgo
- modify gc configuration document
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-04 13:02:09 +08:00
Chun Han
e12b701c03
enhance: add metrics for segment prune latnecy( #30376 ) ( #34364 )
...
related: #30376
pr: https://github.com/milvus-io/milvus/pull/34094
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-04 10:14:09 +08:00
wayblink
99586066f5
feat: [cherry-pick] Major compaction ( #34326 )
...
This PR cherry-picks the following commits:
fix: speed up segment lookup via channel name in datacoord (#33530 )
needed by the next commit
feat: Major compaction (#33620 )
issue: #30633
pr: #33620
---------
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: yiwangdr <80064917+yiwangdr@users.noreply.github.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-07-02 18:29:01 +08:00
wei liu
25d8b74f71
enhance: Execute bloom filter apply in parallel to speed up segment predict ( #33793 )
...
issue: #33610
pr: #33792
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-13 14:14:04 +08:00
jaime
0ad55c6c44
fix: fix loaded entity num is inaccurate ( #33522 )
...
issue: #33520
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-04 20:09:54 +08:00
jaime
8990b8b051
fix: correct error of metrics stats ( #33305 )
...
issue: #32980
cherry pick from master
pr: #33075 #33255
---------
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-05-24 09:15:41 +08:00
jaime
f48a7ff8ff
enhance: use Delete instead of DeletePartialMatch to remove metrics ( #33029 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-05-14 18:49:33 +08:00
jaime
ba625835bc
enhance: Add metrics for segment index files size ( #32979 )
...
issue:#32980
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-05-13 17:59:32 +08:00
congqixia
efa0c73c62
fix: Unify querynode metrics cleanup in collection release ( #32805 )
...
Related to #32803
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-07 15:41:29 +08:00
Bingyi Sun
fecd9c21ba
feat: LRU cache implementation ( #32567 )
...
issue: https://github.com/milvus-io/milvus/issues/32783
This pr is the implementation of lru cache on branch lru-dev.
Signed-off-by: sunby <sunbingyi1992@gmail.com>
Co-authored-by: chyezh <chyezh@outlook.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
2024-05-06 20:29:30 +08:00
jaime
3b5cbed86a
enhance: add db label for deletion and bulkinsert operation ( #32611 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-04-26 15:53:29 +08:00
chyezh
2586c2f1b3
enhance: use WalkWithPrefix api for oss, enable piplined file gc ( #31740 )
...
issue: #19095,#29655,#31718
- Change `ListWithPrefix` to `WalkWithPrefix` of OOS into a pipeline
mode.
- File garbage collection is performed in other goroutine.
- Segment Index Recycle clean index file too.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-25 20:41:27 +08:00
congqixia
72c172a7d7
enhance: Remove duplicated collectionID label for task latency ( #32308 )
...
`CollectionID` already exists in channel name, so remove it to save
metrics traffic.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-16 18:55:19 +08:00
SimFG
c012e6786f
feat: support rate limiter based on db and partition levels ( #31070 )
...
issue: https://github.com/milvus-io/milvus/issues/30577
co-author: @jaime0815
---------
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-04-12 16:01:19 +08:00
jaime
d4fd6c7283
enhance: add db label on binlog size metrics ( #32003 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-04-10 21:01:20 +08:00
SimFG
90bed1caf9
enhance: add the related data size for the read apis ( #31816 )
...
issue: #30436
origin pr: #30438
related pr: #31772
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-10 15:07:17 +08:00
chyezh
c9faa6d936
enhance: add more metrics for cache and search ( #31777 )
...
issue: #30931
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-10 10:55:17 +08:00
yihao.dai
23ba2a5388
enhance: Remove deprecated DataCoordNumStoredRowsCounter metric ( #31778 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-04-07 00:25:15 +08:00
jaime
bd853be8c7
enhance: Add db label for some usual metrics ( #30956 )
...
issue: #31782
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-04-02 14:27:13 +08:00
chyezh
1ad5ccc50f
enhance: add rg and db interface for segment and db/rg metric label ( #31715 )
...
issue: #30931
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-01 10:21:21 +08:00
wei liu
7c7375031d
enhance: Add metrics for task latency in querycoord scheduler ( #31405 )
...
This PR add metrics for task latency in querycoord scheduler, so if any
kind of task stuck, it's easy to figure out by metrics
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-20 19:29:06 +08:00
Jiquan Long
dc2cdbe387
enhance: add more metrics ( #31271 )
...
/kind improvement
fix : #31272
This pr add more metrics, which are:
- Slow query count, which the duration considered as slow can be
configurable;
- Number of deleted entities;
- Number of entities imported;
- Number of entities per collection;
- Number of loaded entities per collection;
- Number of indexed entities;
- Number of indexed entities, per collection, per index and whether it's
a vetor index;
- Quota states (LongTimeTickDelay, MemoryExhuasted, DiskQuotaExhuasted)
per database;
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-03-19 15:23:06 +08:00
congqixia
194a611814
enhance: Add metrics for querycoord current target cp lag ( #31391 )
...
See also #31390
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-19 14:07:05 +08:00
wei liu
5016031dc7
enhance: add metrics for load segment progress ( #31005 )
...
This PR add metrics for load segment progress:
1. add metrics for load segment/index concurrency
2. add metrics for load index latency
3. change load segment latency's time unit to ms
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-08 15:39:02 +08:00
yihao.dai
a434d33e75
feat: Add import scheduler and manager ( #29367 )
...
This PR introduces novel managerial roles for importv2:
1. ImportMeta: To manage all the import tasks;
2. ImportScheduler: To process tasks and modify their states;
3. ImportChecker: To ascertain the completion of all tasks and instigate
relevant operations.
issue: https://github.com/milvus-io/milvus/issues/28521
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-01 18:31:02 +08:00
chyezh
211143c5e6
enhance: add basic information of milvus into metrics ( #29665 )
...
add basic build information and runtime component dependency into
metrics.
issue: #29664
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2024-01-29 15:47:02 +08:00
XuanYang-cn
fd19e419f9
fix: Use size bucket for compacted segment size metric ( #30028 )
...
See also: #29204
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-26 10:53:02 +08:00
XuanYang-cn
0ac37a4a0c
fix: Use longTime bucket for longer latency ( #30175 )
...
See also: #30174
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-22 19:18:55 +08:00
yihao.dai
c8a78465b4
enhance: Add a counter monitoring for the rate-limit requests ( #30109 )
...
Add a counter monitoring metric for the ratelimited rpc requests with
labels: proxy nodeID, rpc request type, and state.
issue: https://github.com/milvus-io/milvus/issues/30052
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-21 14:44:59 +08:00
XuanYang-cn
86f48861c1
fix: Add more throughput in related metrics ( #30038 )
...
This PR also fixes bugs in l0 compactor where
l0 results would never be removed from datanode
See also: #30099
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-19 11:34:54 +08:00
xige-16
9702cef2b5
feat: Support multiple vector search ( #29433 )
...
issue #25639
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-01-08 15:34:48 +08:00
XuanYang-cn
bebe1baf18
fix: Reigster CompactionTaskNum metrics ( #29640 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-04 17:16:45 +08:00
smellthemoon
b12c90af72
enhance:Add upsert vector metrics ( #29226 )
...
add some metrics.
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-03 10:06:48 +08:00
XuanYang-cn
4b406e5973
enhance: Add CompactionTaskNum metrics ( #29518 )
...
See also: #27606
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-28 15:46:55 +08:00
XuanYang-cn
fe04598900
enhance: Add compaction type label to metrics ( #29485 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-27 15:56:48 +08:00
XuanYang-cn
7a6aa8552a
fix: add back existing datanode metrics ( #29360 )
...
See also: #29204
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-22 14:20:43 +08:00