Commit Graph

970 Commits

Author SHA1 Message Date
PowderLi
fa9d606aa8
enhance: add metrics for restful apis (#35083)
issue: #34968
pr: #34969

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-07-30 14:13:49 +08:00
shaoyue
0417463e90
enhance: Gin & restfulv1 handler use sonic json (#35018)
/cc @czs007 @congqixia 
master pr: #35020

Signed-off-by: haorenfsa <haorenfsa@gmail.com>
2024-07-29 00:39:49 +08:00
congqixia
079276c6ff
fix: [2.4] Unify hook singleton implementation in proxy (#34888)
Cherry-pick from master
pr: #34887
Related to #34885

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-26 18:07:53 +08:00
Patrick Weizhi Xu
606bdd7d41
feat: [restful v2][2.4] add partition key isolation prop (#34869)
issue: https://github.com/milvus-io/milvus/issues/34332
pr: https://github.com/milvus-io/milvus/pull/34710

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit 703feaf6b4a94d1747141cea073b5dcfa36eac73)
2024-07-22 11:43:41 +08:00
SimFG
0e226502e4
enhance: [2.4] pick default root password and log level pr (#34777)
default root password
- issue: #33058
- pr: #34752

set log level
- issue: #34756
- pr: #34757

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-18 13:45:43 +08:00
aoiasd
d6fc6a9f41
fix: [Cherry-pick] Restful API use deprecate error code cause access log panic. (#34579)
Avoid panic when use deprecate error code and use merr code replace
deprecate common code.
relate: https://github.com/milvus-io/milvus/issues/34578
pr: https://github.com/milvus-io/milvus/pull/34576

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-07-11 11:58:51 +08:00
jaime
326370c1be
enhance: add disk quota and max collections into db properties (#34386)
issue: https://github.com/milvus-io/milvus/issues/34385
pr: #34368

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-05 18:22:17 +08:00
PowderLi
ba2c232331
fix: [cherry-pick] [restful v2] count(*) & hook (#34433)
issue: #31224 #34374
pr: #34369

for query api:

1. param filter is not requried
2. param limit is useless while outputFields = [count(*)]

add hook about grpc call

---------

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-07-05 09:52:10 +08:00
cai.zhang
f11e421839
enhance: [cherry-pick] Remove compaction plans on the datanode (#33548) (#34312)
issue: #33546

master pr: #33548

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-02 17:39:08 +08:00
aoiasd
bc8ca593bd
enhance: [Cherry-Pick] Update access log (#34295)
Support restful api,print with consistency level and add new write
cache.
relate: https://github.com/milvus-io/milvus/issues/31823
pr: https://github.com/milvus-io/milvus/pull/33155
https://github.com/milvus-io/milvus/pull/33503
https://github.com/milvus-io/milvus/pull/32213

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-07-02 15:30:15 +08:00
cai.zhang
c924b0b502
enhance: [cherry-pick] Refine index code and support analyze data (#34311)
This PR primary picks up the support analyzing functionality, including
the following commits:
- main functionality: https://github.com/milvus-io/milvus/pull/33651
- refine indexnode code: https://github.com/milvus-io/milvus/pull/33458
- related fixes:
  - https://github.com/milvus-io/milvus/pull/33832
  - https://github.com/milvus-io/milvus/pull/33161

issue: #30633 
master prs: #33651 , #33458 , #33832 , #33161

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>
Co-authored-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-07-02 09:50:39 +08:00
yihao.dai
b1e74dc7cb
enhance: [cherry-pick] Decouple compaction from shard (#34157)
This PR cherry-picks the following commits:

- Implement task limit control logic in datanode.
https://github.com/milvus-io/milvus/pull/32881
- Load bf from storage instead of memory during L0 compaction.
https://github.com/milvus-io/milvus/pull/32913
- Remove dependencies on shards (e.g. SyncSegments, injection).
https://github.com/milvus-io/milvus/pull/33138
- Rename Compaction interface to CompactionV2.
https://github.com/milvus-io/milvus/pull/33858
- Remove the unused residual compaction logic.
https://github.com/milvus-io/milvus/pull/33932

issue: https://github.com/milvus-io/milvus/issues/32809

pr: https://github.com/milvus-io/milvus/pull/32881,
https://github.com/milvus-io/milvus/pull/32913,
https://github.com/milvus-io/milvus/pull/33138,
https://github.com/milvus-io/milvus/pull/33858,
https://github.com/milvus-io/milvus/pull/33932

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-25 20:22:03 +08:00
wei liu
4513569207
enhance: add restful api to trigger component stop (#32076) (#33799)
issue: #32698
pr: #32076
This PR add two rest api for component stop and status check:
1. `/management/stop?role=querynode` can stop the specified component
2. `/management/check/ready?role=rootcoord` can check whether the target
component is serviceable

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-17 21:53:59 +08:00
Buqian Zheng
d6a994038c
enhance: [2.4] add sparse float vector support to restful v2 (#33555)
issue: https://github.com/milvus-io/milvus/issues/29419
pr: #33231

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-06-03 19:31:46 +08:00
PowderLi
aaecc92244
fix: [cherry-pick] [restful v2] search result be cut while nq > 1 (#33363)
issue: #33099 #32837 #32419
master pr: #33339

1. len(search result) may be nq * topk, we need return all rather than
topk
2. the  in restful response payload keep the same with milvus error code

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-05-27 00:57:45 +08:00
SimFG
37b2f90c90
enhance: [2.4] the panic when db isn't existed in the rate limit interceptor (#33308)
issue: #33243
pr: #33244

1. fix: the panic when db isn't existed in the rate limit interceptor
#33244
2. enhance: check the auth in some rest v2 api #33256

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-23 17:25:41 +08:00
PowderLi
f10ffb9230
fix: [cherry-pick][restful v2]role operations need dbName (#33291)
issue: #33220
master pr: #33283

use dbName as part of privilege entity, so
1. grant / revoke a privilege need dbName
2. we can describe the privileges of the role which belong to one
special database

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-05-23 09:51:45 +08:00
aoiasd
dc058eaf61
fix: should init access logger before init server (#32976)
relate: https://github.com/milvus-io/milvus/issues/32968

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-14 10:47:32 +08:00
congqixia
c0e62e6629
enhance: Use collection default consistency level for restv2 (#32956)
Set `UseDefaultConsistency` to true so that restv2 read API shall use
collection consistency level setting correctly.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-11 00:47:31 +08:00
wei liu
c35797c399
enhance: expose DescribeDatabase api in proxy (#32732)
issue: #32707

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-09 22:51:30 +08:00
SimFG
8963615b33
enhance: add the cost response for the rest api (#32620)
issue: https://github.com/milvus-io/milvus/issues/30436

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-08 11:53:29 +08:00
wei liu
ba02d54a30
enhance: update shard leader cache when leader location changed (#32470)
issue: #32466

this PR enhance that when shard location changed, update proxy's shard
leader cache. in case of query node failover case, proxy can find
replica recover

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-08 10:05:29 +08:00
yiwangdr
b1eacb2ae8
feat: datacoord/node watch based on rpc (#32036)
issue: https://github.com/milvus-io/milvus/issues/25309

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-05-07 15:49:30 +08:00
wei liu
07720f1a95
enhance: expose alter database api in proxy (#32639)
issue: #30040

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-28 18:15:31 +08:00
Aldrin
51297b51e1
fix: Removed chance for port conflict in service_test (#32626)
issue: https://github.com/milvus-io/milvus/issues/32286

- Avoided chance for server initialisation at port 10000

Signed-off-by: Aldrin <imagesai32@gmail.com>
2024-04-28 14:05:25 +08:00
SimFG
bed6363feb
enhance: update the go-api version for the list api (#32605)
issue: https://github.com/milvus-io/milvus/issues/32550
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-26 17:49:33 +08:00
wei liu
04f355a802
enhance: Enable alter database props in rootcoord (#32458)
issue: #30040

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-25 10:53:25 +08:00
PowderLi
29922829da
fix: [restful v2] parameters of create collection (#32485)
issue: #31176

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-04-24 09:43:29 +08:00
PowderLi
a0a23d0666
fix: [restful]use default search parameter nq: 0 (#32355)
issue: #32225 #31978 #32360

1. v1 can only accept one vector, but v2 accept list of vectors #32225
2. cannot get dbName from AliasReq #31978
3. enhance: [restful v2]support config DB-Name in the http header #32360

---------

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-04-18 14:41:20 +08:00
chyezh
48fe977a9d
enhance: declarative resource group api (#31930)
issue: #30647

- Add declarative resource group api

- Add config for resource group management

- Resource group recovery enhancement

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-15 08:13:19 +08:00
PowderLi
610a65af14
fix: restful v2 (#32144)
issue: #31176

1. cannot get dbName correctly while describe alias #31978
2. return a valid json string even if the user doesn't have the whole
privileges to describe collection #31635
3. rename IndexParam.IndexConfig to IndexParam.Params
4. FieldSchema.ElementTypeParams, IndexParam.Params can not only accept
string

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-04-13 21:55:29 +08:00
SimFG
c012e6786f
feat: support rate limiter based on db and partition levels (#31070)
issue: https://github.com/milvus-io/milvus/issues/30577
co-author: @jaime0815

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-04-12 16:01:19 +08:00
zhenshan.cao
089c805e0a
enhance:Refactor hybrid search (#32020)
issue: https://github.com/milvus-io/milvus/issues/25639
https://github.com/milvus-io/milvus/issues/31368

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-04-09 14:21:18 +08:00
yiwangdr
1cd15d9322
test: support segment release in integration test (#31190)
issue: #29507

Notice that api_testonly.go files should be guarded by compiler tag
`test`, so that production build rules don't compile them and these APIs
don't get misused.

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-04-09 11:39:17 +08:00
Cai Yudong
00438f408f
enhance: Unify data type check APIs for go (#31887)
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-07 14:27:22 +08:00
SimFG
b1a1cca10b
feat: add more operation detail info for better allocation (#30438)
issue: #30436

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-03-28 06:33:11 +08:00
wei liu
92971707de
enhance: Add restful api for devops to execute rolling upgrade (#29998)
issue: #29261
This PR Add restful api for devops to execute rolling upgrade, including
suspend/resume balance and manual transfer segments/channels.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-27 16:15:19 +08:00
yihao.dai
9a13b9822f
enhance: Return more fields in import progress response (#31539)
Return more fields in import progress response, include importedRows and
totalRows. Additionally, ensure compatibility with the old import
progress response by retaining fields of create timestamp and row count.

issue: https://github.com/milvus-io/milvus/issues/31448
https://github.com/milvus-io/milvus/issues/31237
https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-24 21:57:06 +08:00
yihao.dai
0fe5e90e8b
enhance: Remove import v1 (#31403)
Remove all code and logic related to import v1.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-22 15:29:09 +08:00
PowderLi
ee56ae7299
fix: [CHERRY-PICK] [restful v2] add hybrid_search (#31395)
search issue: #30688

2.4 pr: #31344

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-03-21 09:59:08 +08:00
yihao.dai
2b7df47aa9
feat: Add RBAC functionality to ImportV2 (#31298)
issue: https://github.com/milvus-io/milvus/issues/31282,
https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-19 20:01:07 +08:00
PowderLi
ed0526a1a4
fix: [restful v2] create a collection with vector field name (#31317)
fix: [restful v2] create a collection with vector field name

1. quickly create collection, create an index on the vector field #31149
2. valid consistencyLevel is needed, while create a collection
3. show collection's properties and aliases #31180
4. list aliase for one collection, default is the whole database

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-03-17 20:03:07 +08:00
wei liu
d79aa58b37
enhance: Speed up target recovery after query coord restart (#31240)
issue: #28491

after querycoord restart, it will pull a new target, which include
channel and segment list. when segments loaded on querynode has reached
the target, the collection could provide search/query. but if segment
list changes by time, ater querycoord pull a new target, it will takes a
few minutes to catch up the target's segment distribution. and before
that, query/search will fail due to lack of segments.

This PR save the current loaded target to meta storein querycoord's stop
progress, and recover it when query coord starts, to speed up the target
recovery time.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-15 14:19:03 +08:00
jaime
db79be3ae0
fix: ctx cancel should be the last step while stopping server (#31220)
issue: #31219

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-03-15 10:33:05 +08:00
PowderLi
afa4193fc2
fix: [restful v2] create collection & describe collection (#31269)
1. complete the output of describe collection #31180
2. add a new parameter `params` for create collection #31181 #31149
3. create index according to indexConfig while create collection #31199
4. autoID: false while create a collection quickly, take upsert into
consideration
5. polish error information while search without annsField #31010


fix: [restful v1] insert / upsert with default partitionName JIRA-4952

---------

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-03-14 23:27:03 +08:00
yihao.dai
b5c67948b7
enhance: Enhance and modify the return content of ImportV2 (#31192)
1. The Import APIs now provide detailed progress information for each
imported file, including details such as file name, file size, progress,
and more.
2. The APIs now return the collection name and the completion time.
3. Other modifications include changing jobID to jobId and other similar
adjustments.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-13 19:51:03 +08:00
PowderLi
58d7b9f902
fix: [restful v2] search & advanced_search API (#31113)
issue: #30688
former pr:  #30946

1. param `vector` is required #31012
2. param `annsField` is optional, for multiply vector fields #31010
3. support BinaryVector, Float16Vector, BFloat16Vector #31013
4. replace vector with data, to align with pymilvus milvus_client #31093
5. create collection quickly, to align with pymilvus milvus_client
#31149

---------

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-03-11 15:57:01 +08:00
congqixia
d81ba164c8
enhance: Add ListIndexes API from datacoord (#31104)
See also #31103

This PR add `listIndexes` API for datacoor server to list all indexes
for provided collection.
Comparing to the existing `DescribeIndex` API, the new one does NOT
check the segment index building progress to ease the burden when
invoking it

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-07 17:37:01 +08:00
yihao.dai
8ff8fefdc1
fix: Fix errors in the Index service APIs are ignored (#31077)
In Index service APIs, return error if occurs instead of always
returning nil. Additionally, add more tests to cover this scenario.

issue: https://github.com/milvus-io/milvus/issues/31069,
https://github.com/milvus-io/milvus/issues/31027

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-06 22:11:00 +08:00
yihao.dai
3b66c17279
feat: Revise the RESTful bulk insert API (#29698)
Revise the RESTful bulk insert API from version 1 to version 2.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-05 15:03:00 +08:00