test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-03 12:29:36 +08:00
Code Issues Actions 13 Packages Projects Releases Wiki Activity
21,014 Commits 96 Branches 105 Tags 184 MiB
22b917a1e6
Commit Graph

384 Commits

Author SHA1 Message Date
Gao
1d61b604e1
enhance: support retry search when topk is reduced and result not enough (#35645) 
issue: #35576 

This pr is to cover those cases when queryHook optimize search params
and make the result size insufficient, add retry search mechanism and
add related metrics for alarming.

---------

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-10-23 19:19:30 +08:00
jaime
4746f47282
feat: management WebUI homepage (#36822) 
issue: #36784
1. Implement an embedded web server for WebUI access.  
2. Complete the homepage development.

Home page demo:
<img width="2177" alt="iShot_2024-10-10_17 57 34"
src="https://github.com/user-attachments/assets/38539917-ce09-4e54-a5b5-7f4f7eaac353">

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-10-23 11:29:28 +08:00
XuanYang-cn
2e7670908f
enhance: Track complex delete rates (#36957) 
See also: #36953

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-10-18 17:47:30 +08:00
congqixia
e4292569c7
enhance: Unify InboundLabel case using metrics.Label (#36613) 
Previous label case broken by #36107, this PR make all inbound label
using label constants from metrics package.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-30 14:15:17 +08:00
cai.zhang
ecb2b242e2
enhance: Add sorted for segment info (#36469) 
issue: #33744

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-30 10:01:16 +08:00
Ted Xu
e7ea1d7a04
enhance: improve log encoding performance on proxy nodes (#36123) 
See #36122

This PR is designed to enhance log performance through two improvements:

1. Optimize JSON encoding by switching JSON serializer to
`json-iterator`.
2. Adding support of lazy initialization `WithLazy`.

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-09-11 14:51:07 +08:00
congqixia
51870aef08
enhance: Use stats Handler to record request/response size metrics (#36107) 
Related to #36102

This PR use newly added `grpcSizeStatsHandler` to reduce calling
`proto.Size` since the request & response size info is recorded by grpc
framework.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-09 17:13:05 +08:00
jaime
24fb10114b
enhance: remove cooling off in rate limiter for read requests (#35935) 
issue: #35934

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-09-04 14:39:10 +08:00
Zhen Ye
99dff06391
enhance: using streaming service in insert/upsert/flush/delete/querynode (#35406) 
issue: #33285

- using streaming service in insert/upsert/flush/delete/querynode
- fixup flusher bugs and refactor the flush operation
- enable streaming service for dml and ddl
- pass the e2e when enabling streaming service
- pass the integration tst when enabling streaming service

---------

Signed-off-by: chyezh <chyezh@outlook.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-29 10:03:08 +08:00
SimFG
3e1052f889
enhance: use the msg position obj when getting replicate channel position (#35606) 
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-08-27 10:28:59 +08:00
congqixia
2fbc628994
feat: Support field partial load collection (#35416) 
Related to #35415

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-20 16:49:02 +08:00
wei liu
14ef88f423
fix: Fix proxy panic after Import related api failed (#35540) 
issue: #35539

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-19 15:22:55 +08:00
wei liu
1d49358f82
enhance: Add BackupRBAC/RestoreRBAC API to enable rbac backup (#35444) 
issue: #35443

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-16 10:10:53 +08:00
SimFG
b2cc4b0776
feat: add the rbac msg and send them to the replicate channel (#35392) 
- issue: #35391

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-08-15 12:06:52 +08:00
congqixia
de8a266d8a
enhance: Enable linux code checker (#35084) 
See also #34483

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-30 15:53:51 +08:00
wei liu
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 (#34394) 
issue: #34252

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-29 11:31:51 +08:00
congqixia
783f9d9c33
fix: Unify hook singleton implementation in proxy (#34887) 
Related to #34885

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-26 18:07:53 +08:00
congqixia
4ee6c69217
enhance: Add Segment Level in milvus segment info APIs (#34763) 
See also #34746

This PR add segment level field in response of
`GetPersistentSegmentInfo` and `GetQuerySegmentInfo`

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-26 10:01:46 +08:00
SimFG
88c38a3756
fix: execute method should return an error when result is a failure (#34813) 
- issue: #34812

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-07-20 23:13:40 +08:00
yihao.dai
c8bf6c8a46
enhance: Optimize ConsistencyLevel and GuaranteeTimestamp log (#34738) 
/kind improvement

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-18 15:17:41 +08:00
congqixia
b5225df21c
enhance: Use unified requery flag when set metrics (#34632) 
See also #34601

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-15 23:03:38 +08:00
Patrick Weizhi Xu
104d0966b7
feat: support partition key isolation (#34336) 
issue: #34332

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-07-11 19:01:35 +08:00
aoiasd
186757e622
enhance: support mark error as user error (#33498) 
relate: https://github.com/milvus-io/milvus/issues/33492

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-07-01 14:56:12 +08:00
Chun Han
96dcee5dff
fix:load major compaction partial result(#34051) (#34052) 
related: #34051

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-06-28 14:04:05 +08:00
yihao.dai
3540eee977
enhance: Support L0 import (#33514) 
issue: https://github.com/milvus-io/milvus/issues/33157

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-07 14:17:20 +08:00
congqixia
25080bb455
enhance: Print UseDefaultConsistency param in read requests (#33617) 
`UseDefaultConsistency` param is crucial for debugging slow query
problems. It could be confusing when guarantee timestamp is 1 while this
param is not logged

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-05 10:27:50 +08:00
SimFG
2964f60edc
enhance: the proxy metric in the query request (#33307) 
/kind improvement
issue: #33306

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-05-24 14:19:41 +08:00
aoiasd
1b4e28b97f
enhance: Check by proxy rate limiter when delete get data by query. (#30891) 
relate: https://github.com/milvus-io/milvus/issues/30927

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-05-23 20:03:40 +08:00
yihao.dai
9ff023ee35
fix: Fix filtering by partition key fails for importing data (#33274) 
Before executing the import, partition IDs should be reordered according
to partition names. Otherwise, the data might be hashed to the wrong
partition during import. This PR corrects this error.

issue: https://github.com/milvus-io/milvus/issues/33237

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-05-23 11:13:40 +08:00
yihao.dai
6fc00e42d7
fix: Fix binlog import (#33027) 
binlog import should only import into one partition.

issue: https://github.com/milvus-io/milvus/issues/32807

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-05-14 21:45:32 +08:00
wei liu
c35797c399
enhance: expose DescribeDatabase api in proxy (#32732) 
issue: #32707

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-05-09 22:51:30 +08:00
wei liu
ba02d54a30
enhance: update shard leader cache when leader location changed (#32470) 
issue: #32466

this PR enhance that when shard location changed, update proxy's shard
leader cache. in case of query node failover case, proxy can find
replica recover

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-05-08 10:05:29 +08:00
yihao.dai
53874ce245
fix: Fix cannot specify partition name in binlog import (#32730) 
issue: https://github.com/milvus-io/milvus/issues/32807

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-05-07 17:19:30 +08:00
Bingyi Sun
fecd9c21ba
feat: LRU cache implementation (#32567) 
issue: https://github.com/milvus-io/milvus/issues/32783
This pr is the implementation of lru cache on branch lru-dev.

Signed-off-by: sunby <sunbingyi1992@gmail.com>
Co-authored-by: chyezh <chyezh@outlook.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
 2024-05-06 20:29:30 +08:00
wei liu
07720f1a95
enhance: expose alter database api in proxy (#32639) 
issue: #30040

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-04-28 18:15:31 +08:00
jaime
3b5cbed86a
enhance: add db label for deletion and bulkinsert operation (#32611) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-04-26 15:53:29 +08:00
SimFG
8594b55ad5
enhance: add max insert request size and must use partition key configs (#32433) 
issue: https://github.com/milvus-io/milvus/issues/30577
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-04-19 10:31:20 +08:00
SimFG
31a29a2451
enhance: avoid the repeated metric info in the proxy (#32380) 
issue: #30577

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-04-19 10:21:20 +08:00
chyezh
48fe977a9d
enhance: declarative resource group api (#31930) 
issue: #30647

- Add declarative resource group api

- Add config for resource group management

- Resource group recovery enhancement

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-04-15 08:13:19 +08:00
SimFG
c012e6786f
feat: support rate limiter based on db and partition levels (#31070) 
issue: https://github.com/milvus-io/milvus/issues/30577
co-author: @jaime0815

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-04-12 16:01:19 +08:00
zhenshan.cao
0bfe130991
fix: fix metric about ProxySearchVectors (#32152) 
issue :https://github.com/milvus-io/milvus/issues/25639

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-04-11 16:13:17 +08:00
SimFG
90bed1caf9
enhance: add the related data size for the read apis (#31816) 
issue: #30436
origin pr: #30438
related pr: #31772

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-04-10 15:07:17 +08:00
SimFG
a63cf8625c
enhance: support to invalid the database meta cache (#32078) 
issue: #32077
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-04-10 06:45:18 +08:00
zhenshan.cao
089c805e0a
enhance:Refactor hybrid search (#32020) 
issue: https://github.com/milvus-io/milvus/issues/25639
https://github.com/milvus-io/milvus/issues/31368

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-04-09 14:21:18 +08:00
congqixia
7e0d03420e
enhance: Remove duplicated err check code (#32001) 
The `err != nil` check is duplicated here and shall be removed

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-04-08 21:59:17 +08:00
jaime
bd853be8c7
enhance: Add db label for some usual metrics (#30956) 
issue: #31782

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-04-02 14:27:13 +08:00
yihao.dai
417c678754
enhance: Retry on inconsistent requery result (#31713) 
Make retry on ErrInConsistentRequery in proxy rather than in every SDK.

issue: https://github.com/milvus-io/milvus/issues/31642

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-03-30 20:53:19 +08:00
SimFG
b1a1cca10b
feat: add more operation detail info for better allocation (#30438) 
issue: #30436

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-03-28 06:33:11 +08:00
yihao.dai
9a13b9822f
enhance: Return more fields in import progress response (#31539) 
Return more fields in import progress response, include importedRows and
totalRows. Additionally, ensure compatibility with the old import
progress response by retaining fields of create timestamp and row count.

issue: https://github.com/milvus-io/milvus/issues/31448
https://github.com/milvus-io/milvus/issues/31237
https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-03-24 21:57:06 +08:00
yihao.dai
f65a796d18
enhance: Add max file num limit and max file size limit for import (#31497) 
The max number of import files per request should not exceed 1024 by
default (configurable).
The import file size allowed for importing should not exceed 16GB by
default (configurable).

issue: https://github.com/milvus-io/milvus/issues/28521

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-03-22 18:13:06 +08:00