Commit Graph

134 Commits

Author SHA1 Message Date
wei liu
bd658a6510
enhance: Enable dynamic update replica selection policy (#35860)
issue: #35859

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-13 17:05:15 +08:00
aoiasd
da227ff9a1
feat: Support create collection with functions (#35973)
relate: https://github.com/milvus-io/milvus/issues/35853
Support create collection with functions. Prepare for support bm25
function.

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-12 10:43:06 +08:00
jaime
91d23ecbe1
fix: memory leak in proxy meta cache (#36075)
issue: #36074

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-08 17:33:05 +08:00
congqixia
66ed289a85
enhance: Fix typo of clustering key not loaded msg (#35948)
Related to #35415

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-05 10:49:03 +08:00
congqixia
9d80137698
fix: Check clustering key skip load behavior (#35865)
feature issue: #35415
See also #35861

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-02 11:17:02 +08:00
SimFG
311f860676
enhance: support to drop the role which is related the privilege list (#35727)
- issue: #35545

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-30 15:17:00 +08:00
congqixia
2fbc628994
feat: Support field partial load collection (#35416)
Related to #35415

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-20 16:49:02 +08:00
congqixia
a2b517523d
enhance: Add in-memory cache for casbin enforcer result (#35271)
See also #35270

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-05 18:48:15 +08:00
balloon1995
7306d2d115
fix: fix metaCache cleanup issue when listPolicy failed (#34449)
issue: #34667

---------

Signed-off-by: balloon1995 <hszoe1995@outlook.com>
Co-authored-by: congqixia <congqi.xia@zilliz.com>
2024-07-16 10:03:38 +08:00
Patrick Weizhi Xu
104d0966b7
feat: support partition key isolation (#34336)
issue: #34332

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-07-11 19:01:35 +08:00
jaime
60be454db0
enhance: add disk quota and max collections into db properties (#34368)
issue: #34385

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-05 18:22:17 +08:00
wei liu
8a9a42198d
fix: Proxy crash due to shard leader cache data race (#32971)
issue: #32970
cause InvalidateShardLeaderCache use wrong lock, which may cause data
race in meta cache, then proxy may crash

This PR fixed that use leaderMut when try to access shard leader cache.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-11 14:32:12 +08:00
wei liu
ba02d54a30
enhance: update shard leader cache when leader location changed (#32470)
issue: #32466

this PR enhance that when shard location changed, update proxy's shard
leader cache. in case of query node failover case, proxy can find
replica recover

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-08 10:05:29 +08:00
SimFG
c012e6786f
feat: support rate limiter based on db and partition levels (#31070)
issue: https://github.com/milvus-io/milvus/issues/30577
co-author: @jaime0815

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-04-12 16:01:19 +08:00
SimFG
b1a1cca10b
feat: add more operation detail info for better allocation (#30438)
issue: #30436

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-03-28 06:33:11 +08:00
congqixia
007fab183c
enhance: Skip partition key name check if feature not enabled (#31067)
Skip partition key naming & hash value pre process if collection schema
does not have partition key
The PR removes mislead warning when collection has no partition key

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-06 19:33:00 +08:00
congqixia
9b3005f1be
enhance: Avoid create schema helper for each read task (#30981)
See also #30806

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-04 19:39:00 +08:00
cai.zhang
40ca98f57f
enhance: Skip timestamp allocation when search/query consistency level is eventually (#29773)
issue: #29772 
1. Skip timestamp allocation when search/query consistency level is
eventually.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-02-21 09:52:59 +08:00
cai.zhang
d87726e4c7
enhance: Don't expire ShardLeaders cache actively (#29879)
issue: #29772 

The shardLeaders cache does not actively expire, update the cache when
search/query fails.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-01-26 14:13:01 +08:00
aoiasd
38746dfc2b
fix: Remove useless lock which cause porxy meta cache recursive lock (#30203)
relate: https://github.com/milvus-io/milvus/issues/30193

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-01-23 23:56:55 +08:00
aoiasd
a81d2b4780
enhance: refine proxy metacache for concurrent safe (#29872)
relate: https://github.com/milvus-io/milvus/issues/29675

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-01-22 14:28:55 +08:00
PowderLi
631ebb1f07
fix: CollectionSchema.autoID is deprecated (#30002)
issue: #30000
related to: [milvus-proto
#202](https://github.com/milvus-io/milvus-proto/pull/202)

1. replace collSchema.AutoID with primaryField.AutoID
2. show `enableDynamic` & `enableDynamicField` at the same time
3. avoid data race about the access to metacache

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-18 16:16:52 +08:00
wei liu
797847904c
enhance: Change some frequency log to rated level (#29720)
This PR change some frequency log to rated level

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-11 16:30:50 +08:00
smellthemoon
1c1f2a1371
enhance:change some logs (#29579)
related #29588

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-05 16:12:48 +08:00
congqixia
4f8c540c77
enhance: cache collection schema attributes to reduce proxy cpu (#29668)
See also #29113

The collection schema is crucial when performing search/query but some
of the information is calculated for every request.

This PR change schema field of cached collection info into a utility
`schemaInfo` type to store some stable result, say pk field,
partitionKeyEnabled, etc. And provided field name to id map for
search/query services.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-04 17:28:46 +08:00
congqixia
bcf8f27aa7
enhance: refine proxy meta cache partition logic (#29315)
See also #29113

- Unify partition info refresh logic
- Prevent parse partition names for each partition key search request

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-20 10:02:43 +08:00
congqixia
264505b706
enhance: remove reset seed for every shuffle (#29195)
See also #29113
rand.Seed is deprecated and cost noticable CPU time during heavy payload

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-14 15:46:38 +08:00
congqixia
d0bac9d0bb
enhance: Avoid initializing casbin enforcer for each request (#29117)
See also #29113

This patch:
- Replace plain Enforcer with `casbin.SyncedEnforcer`
- Add implementation of persist.Adapter with `MetaCacheCasbinAdapter`
- Invoke enforcer.LoadPolicy when policy updated

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-12 10:36:43 +08:00
PowderLi
20fc90c591
enhance: find collection schema from cache (#28782)
issue: #28781 #28329

1. There is no need to call `DescribeCollection`, if the collection's
schema is found in the globalMetaCache
2. did `GetProperties` to check the access to Azure Blob Service while
construct the ChunkManager

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-03 19:22:33 +08:00
wei liu
0677d2623d
remove retry in getShards (#28011)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-10-31 16:16:14 +08:00
zhenshan.cao
020ad9a6bc
Rectify wrong exception messages associated with Array datatype (#27769)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2023-10-19 17:24:07 +08:00
jaime
7f7c71ea7d
Decoupling client and server API in types interface (#27186)
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-09-26 09:57:25 +08:00
SimFG
26f06dd732
Format the code (#27275)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
yah01
168e82ee10
Fix panic while handling with the nil status (#27040)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-15 10:09:21 +08:00
yah01
00c65fa0d7
Refine QueryNode errors (#27013)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-12 16:07:18 +08:00
yah01
3349db4aa7
Refine errors to remove changes breaking design (#26521)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-04 09:57:09 +08:00
jaime
d001133bd2
[skip e2e] Refine collection not exists error (#26707)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-08-31 17:29:03 +08:00
smellthemoon
248bbc749d
Fix GetCollectionID to return err (#26239)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-08-11 10:21:29 +08:00
wei liu
f9823e039f
fix describe rg with non exist collection (#26227)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-10 18:43:16 +08:00
SimFG
4e1b65d38f
Refine the rbac cache update process (#26150)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-08-07 11:59:07 +08:00
SimFG
d2649b63db
Delete the user-role mapping info when deleting the user (#25988)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-08-04 18:37:08 +08:00
chyezh
f915f1739a
[Fixup] Short-term fix metacache data race (#25802)
Signed-off-by: chyezh <ye.zhen@zilliz.com>
2023-08-04 17:47:07 +08:00
smellthemoon
9614e61f14
Fix collection and channel not match (#25859)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-08-01 17:33:06 +08:00
Enwei Jiao
66fdc71479
Refactor logs in DataCoord & DataNode (#25574)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-07-14 15:56:31 +08:00
wei liu
986208bca0
fix set wrong consistency level in proxy (#25458)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-07-11 10:06:28 +08:00
xige-16
33c2012675
Add more metrics (#25081)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-06-26 17:52:44 +08:00
jaime
18df2ba6fd
[Cherry-Pick] Support Database (#24769)
Support Database(#23742)
Fix db nonexists error for FlushAll (#24222)
Fix check collection limits fails (#24235)
backward compatibility with empty DB name (#24317)
Fix GetFlushAllState with DB (#24347)
Remove db from global meta cache after drop database (#24474)
Fix db name is empty for describe collection response (#24603)
Add RBAC for Database API (#24653)
Fix miss load the same name collection during recover stage (#24941)

RBAC supports Database validation (#23609)
Fix to list grant with db return empty (#23922)
Optimize PrivilegeAll permission check (#23972)
Add the default db value for the rbac request (#24307)

Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-06-25 17:20:43 +08:00
Enwei Jiao
d143682d7d
Refactor logs in proxy package. (#24936)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-06-19 13:28:41 +08:00
wei liu
a3437e0ab5
refactor replica selection and retry policy on channel (#24367)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-06-13 10:20:37 +08:00
congqixia
864c1bbcee
Use collInfo instead of reading again (#24784)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 20:26:35 +08:00