Commit Graph

947 Commits

Author SHA1 Message Date
wei liu
07720f1a95
enhance: expose alter database api in proxy (#32639)
issue: #30040

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-28 18:15:31 +08:00
Aldrin
51297b51e1
fix: Removed chance for port conflict in service_test (#32626)
issue: https://github.com/milvus-io/milvus/issues/32286

- Avoided chance for server initialisation at port 10000

Signed-off-by: Aldrin <imagesai32@gmail.com>
2024-04-28 14:05:25 +08:00
SimFG
bed6363feb
enhance: update the go-api version for the list api (#32605)
issue: https://github.com/milvus-io/milvus/issues/32550
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-26 17:49:33 +08:00
wei liu
04f355a802
enhance: Enable alter database props in rootcoord (#32458)
issue: #30040

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-25 10:53:25 +08:00
PowderLi
29922829da
fix: [restful v2] parameters of create collection (#32485)
issue: #31176

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-04-24 09:43:29 +08:00
PowderLi
a0a23d0666
fix: [restful]use default search parameter nq: 0 (#32355)
issue: #32225 #31978 #32360

1. v1 can only accept one vector, but v2 accept list of vectors #32225
2. cannot get dbName from AliasReq #31978
3. enhance: [restful v2]support config DB-Name in the http header #32360

---------

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-04-18 14:41:20 +08:00
chyezh
48fe977a9d
enhance: declarative resource group api (#31930)
issue: #30647

- Add declarative resource group api

- Add config for resource group management

- Resource group recovery enhancement

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-15 08:13:19 +08:00
PowderLi
610a65af14
fix: restful v2 (#32144)
issue: #31176

1. cannot get dbName correctly while describe alias #31978
2. return a valid json string even if the user doesn't have the whole
privileges to describe collection #31635
3. rename IndexParam.IndexConfig to IndexParam.Params
4. FieldSchema.ElementTypeParams, IndexParam.Params can not only accept
string

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-04-13 21:55:29 +08:00
SimFG
c012e6786f
feat: support rate limiter based on db and partition levels (#31070)
issue: https://github.com/milvus-io/milvus/issues/30577
co-author: @jaime0815

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-04-12 16:01:19 +08:00
zhenshan.cao
089c805e0a
enhance:Refactor hybrid search (#32020)
issue: https://github.com/milvus-io/milvus/issues/25639
https://github.com/milvus-io/milvus/issues/31368

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-04-09 14:21:18 +08:00
yiwangdr
1cd15d9322
test: support segment release in integration test (#31190)
issue: #29507

Notice that api_testonly.go files should be guarded by compiler tag
`test`, so that production build rules don't compile them and these APIs
don't get misused.

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-04-09 11:39:17 +08:00
Cai Yudong
00438f408f
enhance: Unify data type check APIs for go (#31887)
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-07 14:27:22 +08:00
SimFG
b1a1cca10b
feat: add more operation detail info for better allocation (#30438)
issue: #30436

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-03-28 06:33:11 +08:00
wei liu
92971707de
enhance: Add restful api for devops to execute rolling upgrade (#29998)
issue: #29261
This PR Add restful api for devops to execute rolling upgrade, including
suspend/resume balance and manual transfer segments/channels.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-27 16:15:19 +08:00
yihao.dai
9a13b9822f
enhance: Return more fields in import progress response (#31539)
Return more fields in import progress response, include importedRows and
totalRows. Additionally, ensure compatibility with the old import
progress response by retaining fields of create timestamp and row count.

issue: https://github.com/milvus-io/milvus/issues/31448
https://github.com/milvus-io/milvus/issues/31237
https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-24 21:57:06 +08:00
yihao.dai
0fe5e90e8b
enhance: Remove import v1 (#31403)
Remove all code and logic related to import v1.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-22 15:29:09 +08:00
PowderLi
ee56ae7299
fix: [CHERRY-PICK] [restful v2] add hybrid_search (#31395)
search issue: #30688

2.4 pr: #31344

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-03-21 09:59:08 +08:00
yihao.dai
2b7df47aa9
feat: Add RBAC functionality to ImportV2 (#31298)
issue: https://github.com/milvus-io/milvus/issues/31282,
https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-19 20:01:07 +08:00
PowderLi
ed0526a1a4
fix: [restful v2] create a collection with vector field name (#31317)
fix: [restful v2] create a collection with vector field name

1. quickly create collection, create an index on the vector field #31149
2. valid consistencyLevel is needed, while create a collection
3. show collection's properties and aliases #31180
4. list aliase for one collection, default is the whole database

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-03-17 20:03:07 +08:00
wei liu
d79aa58b37
enhance: Speed up target recovery after query coord restart (#31240)
issue: #28491

after querycoord restart, it will pull a new target, which include
channel and segment list. when segments loaded on querynode has reached
the target, the collection could provide search/query. but if segment
list changes by time, ater querycoord pull a new target, it will takes a
few minutes to catch up the target's segment distribution. and before
that, query/search will fail due to lack of segments.

This PR save the current loaded target to meta storein querycoord's stop
progress, and recover it when query coord starts, to speed up the target
recovery time.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-15 14:19:03 +08:00
jaime
db79be3ae0
fix: ctx cancel should be the last step while stopping server (#31220)
issue: #31219

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-03-15 10:33:05 +08:00
PowderLi
afa4193fc2
fix: [restful v2] create collection & describe collection (#31269)
1. complete the output of describe collection #31180
2. add a new parameter `params` for create collection #31181 #31149
3. create index according to indexConfig while create collection #31199
4. autoID: false while create a collection quickly, take upsert into
consideration
5. polish error information while search without annsField #31010


fix: [restful v1] insert / upsert with default partitionName JIRA-4952

---------

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-03-14 23:27:03 +08:00
yihao.dai
b5c67948b7
enhance: Enhance and modify the return content of ImportV2 (#31192)
1. The Import APIs now provide detailed progress information for each
imported file, including details such as file name, file size, progress,
and more.
2. The APIs now return the collection name and the completion time.
3. Other modifications include changing jobID to jobId and other similar
adjustments.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-13 19:51:03 +08:00
PowderLi
58d7b9f902
fix: [restful v2] search & advanced_search API (#31113)
issue: #30688
former pr:  #30946

1. param `vector` is required #31012
2. param `annsField` is optional, for multiply vector fields #31010
3. support BinaryVector, Float16Vector, BFloat16Vector #31013
4. replace vector with data, to align with pymilvus milvus_client #31093
5. create collection quickly, to align with pymilvus milvus_client
#31149

---------

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-03-11 15:57:01 +08:00
congqixia
d81ba164c8
enhance: Add ListIndexes API from datacoord (#31104)
See also #31103

This PR add `listIndexes` API for datacoor server to list all indexes
for provided collection.
Comparing to the existing `DescribeIndex` API, the new one does NOT
check the segment index building progress to ease the burden when
invoking it

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-07 17:37:01 +08:00
yihao.dai
8ff8fefdc1
fix: Fix errors in the Index service APIs are ignored (#31077)
In Index service APIs, return error if occurs instead of always
returning nil. Additionally, add more tests to cover this scenario.

issue: https://github.com/milvus-io/milvus/issues/31069,
https://github.com/milvus-io/milvus/issues/31027

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-06 22:11:00 +08:00
yihao.dai
3b66c17279
feat: Revise the RESTful bulk insert API (#29698)
Revise the RESTful bulk insert API from version 1 to version 2.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-05 15:03:00 +08:00
wei liu
fa73520b57
fix: Make datacoord client retry on index api (#30654)
issue: #20553

This PR add retry on all interface which belong to indexcoord in milvus
2.2 and. move to data coord in milvus 2.3, to prevent meet
`unimplemented` error during rolling upgrade from milvus 2.2 to 2.3.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-04 15:32:59 +08:00
PowderLi
e39f46aa38
fix: use GetBinaryField incorrectly (#30969)
issue: #30980 #22837
2.3 pr: #30873

should use GetBFloat16Field, GetBfloat16Field rather than GetBinaryField

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-03-03 17:58:59 +08:00
yihao.dai
a434d33e75
feat: Add import scheduler and manager (#29367)
This PR introduces novel managerial roles for importv2:
1. ImportMeta: To manage all the import tasks;
2. ImportScheduler: To process tasks and modify their states;
3. ImportChecker: To ascertain the completion of all tasks and instigate
relevant operations.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-01 18:31:02 +08:00
PowderLi
4869aaeb94
fix: [restful v2] align search api with milvus client (#30946)
issue: #30688

fix: [restful v2] align search api with milvus client #30688
fix: [restful v2] DescribeIndex return Response{ Status: IndexNotFound
}, nil #30722

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-03-01 11:01:01 +08:00
chyezh
0c7474d7e8
enhance: add graceful stop timeout to avoid node stop hang under extreme cases (#30317)
1. add coordinator graceful stop timeout to 5s
2. change the order of datacoord component while stop
3. change querynode grace stop timeout to 900s, and we should
potentially change this to 600s when graceful stop is smooth

issue: #30310
also see pr: #30306

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-02-29 17:01:50 +08:00
chyezh
e59bc1e526
enhance: use latest milvus proto, add unimplemented server (#30889)
issue: #30647

Signed-off-by: chyezh <chyezh@outlook.com>
2024-02-29 10:13:00 +08:00
PowderLi
50a78b682e
fix: set proxy.http.acceptTypeAllowInt64: true as default (#30720)
issue: #30680

also let the parameter item to be refreshable

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-02-29 09:59:07 +08:00
PowderLi
6548182a4a
fix: [restful]vector field cannot be empty while insert (#30871)
issue: #30870 

did a check whether the vector field is empty in advance

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-02-28 10:36:57 +08:00
PowderLi
19911a411f
fix: [restful v2] bug list (#30740)
bug list
1. element data type is needed while create a collection with an array
field #30638
2. spelling mistake about metricsType #30643
3. new collection enable dynamic field / auto id as user defined #30665
4. convert rowCount from string to int #30661
5. try it's best to create a new collection #30652
6. int64 percious #20415
7. insert into collection which has multi vector fields #30674
8. cannot rename a collection to other database #30700
9. update the request body of "indexes/create" #30769
10. got [] while list indexes of a collection which has no index #30722
11. restful need encode password before call
CreateCredential/UpdateCredential #30730
12. some parameters missed the required label #30737
13.define the field to be or not to be a partition key while create a
collection #30797

enhance: support opentelemetry
enhance: support dataType: Float16Vector & BFloat16Vector #22837
enhance: describe collection will show the field is partition key or not
#30789

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-02-27 16:48:56 +08:00
chyezh
950624d8d3
fix: nil ptr is used as nil interface in grpc client (#30754)
issue: #30715

- Bug: Set nil struct pointer to describe nil interface.
Panic with segment violation when calling method on this nil struct
pointer.

Signed-off-by: chyezh <chyezh@outlook.com>
2024-02-23 10:26:53 +08:00
yiwangdr
c6665c2a4c
test: support multiple data/querynodes in integration test (#30618)
issue: https://github.com/milvus-io/milvus/issues/29507

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-02-21 11:54:53 +08:00
PowderLi
1b87bcd60e
feat: restful support new features (#30485)
feat: restful support new features

1. search with groupingField #25324
2. hybrid search #25639

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-02-07 17:16:47 +08:00
xige-16
060c8603a3
fix: Support mvcc with hybrid serach (#30114)
issue: https://github.com/milvus-io/milvus/issues/29656
/kind bug

Signed-off-by: xige-16 <xi.ge@zilliz.com>

---------

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-02-01 16:03:03 +08:00
PowderLi
5cf9bb236e
enhance: restful support import jobs (#30343)
issue: #28521 #29732

include
1. list collection's import jobs
2. create a new import job
3. get the progress of an import job

fix:
1. mix the order of dbName & collectionName #29728
2. trace log keep the same as v1
3. support traceID
4. azure precheck, blob name cannot end with / #29703

---------

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-31 17:57:04 +08:00
yah01
2f778d9649
fix: proxy may never setup if the port binded (#30035)
the proxy miss-returned nil while failed to listen the port, then the
server continues to run but we can't connect to service
resolve #30034

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2024-01-31 17:37:04 +08:00
PowderLi
6abbab12fa
feat: restful phase two (#29728)
issue: #29732

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-28 16:03:01 +08:00
PowderLi
8fc4ebfa11
fix: empty MetricType (#30216)
issue: #30102 #30225

we should read MetricType from SearchResult,
because query node never
1. read metricType from LoadMeta
2. store to collection
3. set SearchRequest.MetricType

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-28 15:33:02 +08:00
PowderLi
08ca0a2ca5
feat: support etcd authentication (#30226)
issue: #28895
add 3 configuration for ETCD config

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-24 11:35:00 +08:00
SimFG
b62748a1a6
fix: wrong format expr for the delete rest api (#30217)
/kind improvement
issue: #30092

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-01-24 11:27:06 +08:00
PowderLi
4f44942c80
enhance: support dataType: array & json (#30076)
issue: #30075

deal with the array<?> field data correctly

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-21 13:56:55 +08:00
PowderLi
631ebb1f07
fix: CollectionSchema.autoID is deprecated (#30002)
issue: #30000
related to: [milvus-proto
#202](https://github.com/milvus-io/milvus-proto/pull/202)

1. replace collSchema.AutoID with primaryField.AutoID
2. show `enableDynamic` & `enableDynamicField` at the same time
3. avoid data race about the access to metacache

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-18 16:16:52 +08:00
zhenshan.cao
31740445fa
fix: duplicate dynamic field data by mistake (#30042)
issue : #30000

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-01-17 19:02:56 +08:00
wayblink
1df3f90696
feat: Implement DescribeAlias and ListAliases interfaces (#29641)
#22882
/kind feature

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-01-11 19:12:51 +08:00