Commit Graph

94 Commits

Author SHA1 Message Date
yihao.dai
c5088b8527
enhance: Add metrics to monitor import throughput and imported rows (#36519) (#36588)
issue: https://github.com/milvus-io/milvus/issues/36518

pr: https://github.com/milvus-io/milvus/pull/36519

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-30 10:03:18 +08:00
XuanYang-cn
c1dab50fed
enhance: [cp]Add metrics for Delete entries num of L0seg (#36227)
- Add metrics *DataCoordL0DeleteEntriesNum*
- Remove metrics *DataCoordRateStoredL0Segment*

See also: #36147
pr: #36175

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-09-14 10:37:08 +08:00
congqixia
1cd8d1bd80
enhance: [2.4] Use stats Handler to record request/response size metrics (#36107) (#36118)
Cherry-pick from master
pr: #36107 
Related to #36102

This PR use newly added `grpcSizeStatsHandler` to reduce calling
`proto.Size` since the request & response size info is recorded by grpc
framework.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-10 17:13:08 +08:00
Gao
1687d64c46
enhance: [2.4] add hit segment num metrics for queryHook (#35619)
issue: #35576 
pr: #35577

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-08-23 12:49:02 +08:00
Jiquan Long
d14d00b0cc
enhance: add metric to record disk quota (#35305) (#35306)
https://github.com/milvus-io/milvus/issues/35177
pr: https://github.com/milvus-io/milvus/pull/35305

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-07 10:24:27 +08:00
cai.zhang
2534b30e39
enhance: [cherry-pick] Add monitoring metrics for task execution time in datacoord (#35141)
issue: #35138 

master pr: #35139

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-02 19:46:16 +08:00
wei liu
2609e95cb5
enhance: add segment state on data coord bin log size metrics (#35153) (#35235)
pr: #35153

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-02 18:28:14 +08:00
Jiquan Long
ae3657b23e
enhance: add metric to record maxInsertRate and queryNodeMemoryHighWaterLevel (#35185) (#35188)
pr: https://github.com/milvus-io/milvus/pull/35185
fix: https://github.com/milvus-io/milvus/issues/35177

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-02 10:16:15 +08:00
PowderLi
fa9d606aa8
enhance: add metrics for restful apis (#35083)
issue: #34968
pr: #34969

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-07-30 14:13:49 +08:00
Jiquan Long
2822d8729c
feat: record the number of times milvus turns into force-deny-writing states (#34960) (#34989)
/kind improvement
fix: #34961
pr: #34960 

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-07-25 15:43:09 +08:00
Jiquan Long
34873a7f76
feat: record the duration waiting in the proxy queue (#34744) (#34788)
fix: https://github.com/milvus-io/milvus/issues/34743
pr: #34744

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-07-19 15:37:44 +08:00
wayblink
9dc98ac76d
enhance: [cherry-pick] Refine clustering compaction latency metrics (#34539)
issue: #34527
pr: #34528

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-11 17:43:43 +08:00
edward.zeng
3d29ebfffb
enhance: Refine default histogram metric bucket (#34592)
/kind improvement
pr: https://github.com/milvus-io/milvus/pull/34580

Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
2024-07-11 16:51:35 +08:00
Chun Han
2f38483418
fix: lose partitionIDs when scalar pruning and refine segment prune ratio metrics(#30376) (#34475)
related: #30376
pr: https://github.com/milvus-io/milvus/pull/34477

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-07-08 19:44:13 +08:00
chyezh
a1a0a56f86
enhance: async search and retrieve in cgo (#34200)
issue: #33132
pr: #33133
other pr: #33228, #34084, #33946

- implement future-based cgo utility
- async search and retrieve in cgo
- modify gc configuration document

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-04 13:02:09 +08:00
Chun Han
e12b701c03
enhance: add metrics for segment prune latnecy(#30376) (#34364)
related: #30376
pr: https://github.com/milvus-io/milvus/pull/34094

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-04 10:14:09 +08:00
wayblink
99586066f5
feat: [cherry-pick] Major compaction (#34326)
This PR cherry-picks the following commits:
fix: speed up segment lookup via channel name in datacoord (#33530)
needed by the next commit
  feat: Major compaction (#33620)

issue: #30633
pr: #33620

---------

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: yiwangdr <80064917+yiwangdr@users.noreply.github.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-07-02 18:29:01 +08:00
wei liu
25d8b74f71
enhance: Execute bloom filter apply in parallel to speed up segment predict (#33793)
issue: #33610
pr: #33792

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-13 14:14:04 +08:00
jaime
0ad55c6c44
fix: fix loaded entity num is inaccurate (#33522)
issue: #33520

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-04 20:09:54 +08:00
jaime
8990b8b051
fix: correct error of metrics stats (#33305)
issue: #32980
cherry pick from master
pr:  #33075 #33255

---------

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-05-24 09:15:41 +08:00
jaime
f48a7ff8ff
enhance: use Delete instead of DeletePartialMatch to remove metrics (#33029)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-05-14 18:49:33 +08:00
jaime
ba625835bc
enhance: Add metrics for segment index files size (#32979)
issue:#32980

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-05-13 17:59:32 +08:00
congqixia
efa0c73c62
fix: Unify querynode metrics cleanup in collection release (#32805)
Related to #32803

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-07 15:41:29 +08:00
Bingyi Sun
fecd9c21ba
feat: LRU cache implementation (#32567)
issue: https://github.com/milvus-io/milvus/issues/32783
This pr is the implementation of lru cache on branch lru-dev.

Signed-off-by: sunby <sunbingyi1992@gmail.com>
Co-authored-by: chyezh <chyezh@outlook.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
2024-05-06 20:29:30 +08:00
jaime
3b5cbed86a
enhance: add db label for deletion and bulkinsert operation (#32611)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-04-26 15:53:29 +08:00
chyezh
2586c2f1b3
enhance: use WalkWithPrefix api for oss, enable piplined file gc (#31740)
issue: #19095,#29655,#31718

- Change `ListWithPrefix` to `WalkWithPrefix` of OOS into a pipeline
mode.

- File garbage collection is performed in other goroutine.

- Segment Index Recycle clean index file too.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-25 20:41:27 +08:00
congqixia
72c172a7d7
enhance: Remove duplicated collectionID label for task latency (#32308)
`CollectionID` already exists in channel name, so remove it to save
metrics traffic.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-16 18:55:19 +08:00
SimFG
c012e6786f
feat: support rate limiter based on db and partition levels (#31070)
issue: https://github.com/milvus-io/milvus/issues/30577
co-author: @jaime0815

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-04-12 16:01:19 +08:00
jaime
d4fd6c7283
enhance: add db label on binlog size metrics (#32003)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-04-10 21:01:20 +08:00
SimFG
90bed1caf9
enhance: add the related data size for the read apis (#31816)
issue: #30436
origin pr: #30438
related pr: #31772

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-10 15:07:17 +08:00
chyezh
c9faa6d936
enhance: add more metrics for cache and search (#31777)
issue: #30931

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-10 10:55:17 +08:00
yihao.dai
23ba2a5388
enhance: Remove deprecated DataCoordNumStoredRowsCounter metric (#31778)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-04-07 00:25:15 +08:00
jaime
bd853be8c7
enhance: Add db label for some usual metrics (#30956)
issue: #31782

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-04-02 14:27:13 +08:00
chyezh
1ad5ccc50f
enhance: add rg and db interface for segment and db/rg metric label (#31715)
issue: #30931

Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-01 10:21:21 +08:00
wei liu
7c7375031d
enhance: Add metrics for task latency in querycoord scheduler (#31405)
This PR add metrics for task latency in querycoord scheduler, so if any
kind of task stuck, it's easy to figure out by metrics

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-20 19:29:06 +08:00
Jiquan Long
dc2cdbe387
enhance: add more metrics (#31271)
/kind improvement
fix: #31272 

This pr add more metrics, which are:
- Slow query count, which the duration considered as slow can be
configurable;
- Number of deleted entities;
- Number of entities imported;
- Number of entities per collection;
- Number of loaded entities per collection;
- Number of indexed entities;
- Number of indexed entities, per collection, per index and whether it's
a vetor index;
- Quota states (LongTimeTickDelay, MemoryExhuasted, DiskQuotaExhuasted)
per database;

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-03-19 15:23:06 +08:00
congqixia
194a611814
enhance: Add metrics for querycoord current target cp lag (#31391)
See also #31390

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-19 14:07:05 +08:00
wei liu
5016031dc7
enhance: add metrics for load segment progress (#31005)
This PR add metrics for load segment progress:
1. add metrics for load segment/index concurrency
2. add metrics for load index latency
3. change load segment latency's time unit to ms

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-08 15:39:02 +08:00
yihao.dai
a434d33e75
feat: Add import scheduler and manager (#29367)
This PR introduces novel managerial roles for importv2:
1. ImportMeta: To manage all the import tasks;
2. ImportScheduler: To process tasks and modify their states;
3. ImportChecker: To ascertain the completion of all tasks and instigate
relevant operations.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-01 18:31:02 +08:00
chyezh
211143c5e6
enhance: add basic information of milvus into metrics (#29665)
add basic build information and runtime component dependency into
metrics.

issue: #29664

Signed-off-by: chyezh <ye.zhen@zilliz.com>
2024-01-29 15:47:02 +08:00
XuanYang-cn
fd19e419f9
fix: Use size bucket for compacted segment size metric (#30028)
See also: #29204

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-26 10:53:02 +08:00
XuanYang-cn
0ac37a4a0c
fix: Use longTime bucket for longer latency (#30175)
See also: #30174

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-22 19:18:55 +08:00
yihao.dai
c8a78465b4
enhance: Add a counter monitoring for the rate-limit requests (#30109)
Add a counter monitoring metric for the ratelimited rpc requests with
labels: proxy nodeID, rpc request type, and state.

issue: https://github.com/milvus-io/milvus/issues/30052

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-21 14:44:59 +08:00
XuanYang-cn
86f48861c1
fix: Add more throughput in related metrics (#30038)
This PR also fixes bugs in l0 compactor where
l0 results would never be removed from datanode

See also: #30099

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-19 11:34:54 +08:00
xige-16
9702cef2b5
feat: Support multiple vector search (#29433)
issue #25639 

Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-01-08 15:34:48 +08:00
XuanYang-cn
bebe1baf18
fix: Reigster CompactionTaskNum metrics (#29640)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-04 17:16:45 +08:00
smellthemoon
b12c90af72
enhance:Add upsert vector metrics (#29226)
add some metrics.

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-03 10:06:48 +08:00
XuanYang-cn
4b406e5973
enhance: Add CompactionTaskNum metrics (#29518)
See also: #27606

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-28 15:46:55 +08:00
XuanYang-cn
fe04598900
enhance: Add compaction type label to metrics (#29485)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-27 15:56:48 +08:00
XuanYang-cn
7a6aa8552a
fix: add back existing datanode metrics (#29360)
See also: #29204

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-22 14:20:43 +08:00