Commit Graph

18693 Commits

Author SHA1 Message Date
sammy.huang
62fdfe125f
enhance: [skip e2e]to add some comments (#29440)
there are two purposes for this PR
1. to add some comments on variable in the .env file. 
2. avoid git merge conflict when one PR update the value of DATE_VERSION
and LATEST_DATE_VERSION and meanwhile other PR updates the value of
GPU_DATE_VERSION and LATEST_GPU_DATE_VERSION , especailly when github
actions Publish Builder and Publish gpu Builder occur at same time

Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2023-12-25 10:28:47 +08:00
aoiasd
7c916b1035
fix: Remove SetFinalizer which cause proxy msgstream memory leak (#29403)
relate: https://github.com/milvus-io/milvus/issues/28367

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-25 10:16:53 +08:00
PowderLi
13988cbde7
fix: Partition related privileges should belong to Collection (#29428)
issue: #29427

it will try to find privilege_name from the object_type, while grant a
privilege to a role

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-24 20:32:45 +08:00
yah01
aef483806d
enhance: improve the segcore logs (#29372)
- remove the streaming logging
- refine existing logs

fix #29366

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-23 21:52:43 +08:00
congqixia
a937e4c232
fix: segment may never get flushed if sealed before watch (#29436)
See also #29092

`FlushSegments` transfer only `Growing` segment to flushing, if the
segment is in `Sealed` state before Datanode watch channel, the state
will never got satisfied for a segment be selected to be flushed.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-23 21:32:43 +08:00
yah01
1b7f1d7067
enhance: mmap data corrupted after seal the column (#29422)
this bug was introduced in recent changes

Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-23 15:20:43 +08:00
zhagnlu
1cbe3cd5fc
fix: fix memory leak when cancel segcore task (#29431)
#29430

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-12-22 20:28:43 +08:00
SimFG
dd9c61831d
enhance: Support to get the param value in the runtime (#29297)
/kind improvement
issue: #29299

Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-12-22 18:36:44 +08:00
XuanYang-cn
7a6aa8552a
fix: add back existing datanode metrics (#29360)
See also: #29204

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-22 14:20:43 +08:00
SimFG
67ab0e424b
fix: Clean the compaction plan info to avoid the object leak (#29365)
issue: #29296

Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-12-22 12:00:43 +08:00
nico
e75467dbf7
test: modify test cases (#29383)
Signed-off-by: nico <cheng.yuan@zilliz.com>
2023-12-22 09:58:43 +08:00
nico
2f434130f0
enhance: update pymilvus api (#29362)
issue: #17614
update some new api: load_state, num_shards, get_server_type
remove some old api: collection.create_alias, drop_alias

Signed-off-by: nico <cheng.yuan@zilliz.com>
2023-12-22 09:46:43 +08:00
zhagnlu
a6eb7e5f9a
enhance: skip segment when using pk in (..) expr (#29394)
#29293

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-12-21 20:06:42 +08:00
sre-ci-robot
ad71b9aeb2 Update all contributors
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2023-12-21 12:00:40 +00:00
sammy.huang
4ee643a984
enhance: [skip e2e]remove-old-builder (#29384)
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2023-12-21 19:14:42 +08:00
yah01
1997fd150c
enhance: use local path as mmap file path if no mmap path provided (#28992)
the mmap mode isn't controlled by the config anymore, so we don't
require user to set it when they enabled mmap mode.

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-21 18:12:45 +08:00
yah01
a0e1a1eb31
feat: support enable/disable mmap for index (#29005)
support enable/disable mmap for index, the user could alter the index's
mode by `AlterIndex` method
related: https://github.com/milvus-io/milvus/issues/21866

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-21 18:07:24 +08:00
yah01
7a2374e698
enhance: reduce the memory usage of variable length data (#29387)
add all loading data into a buffer and then copy them into the a
fit-in-size memory

---------

Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-21 18:02:42 +08:00
chyezh
be87c18b44
fix: fixup data race at generate binlog index (#29370)
issue: #29339

Signed-off-by: chyezh <ye.zhen@zilliz.com>
2023-12-21 14:58:49 +08:00
XuanYang-cn
48f506b077
fix: Seperate msgstream initctx with lifecycle ctx (#29308)
The close of msgstream should be controlled by Close() not ctx from
outside when init clients

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-21 11:56:45 +08:00
wei liu
fcba1c0d9e
fix: Rename invalid parameterutil package path (#29334)
This PR renames the invalid parameterutil package path

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-21 11:40:42 +08:00
wei liu
cb0676153f
enhance: add metrics for stopping querynode balance progress (#29201)
This PR add three metrics to track the stopping balance progress.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-21 10:14:52 +08:00
congqixia
f699be79f7
fix: grpc client check session skipped due to role not match (#29356)
Related to #28815

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-21 10:12:51 +08:00
yihao.dai
f457b9f7c9
fix: Return time tick delay error and refine quota error messages (#29289)
This pr:
1. Handles the time tick delay error when converting old error codes to
milvus errors.
2. Enhances quota error messages by eliminating "force deny" and
substituting it with "quota exceeded."

issue: https://github.com/milvus-io/milvus/issues/29288

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-12-20 22:52:44 +08:00
congqixia
1eacdc591b
fix: delegator may mark segment offline by mistake (#29343)
See also #29332

The segment may be released before or during the request when delegator
tries to forward delete request to yet. Currently, these two situation
returns different error code.

In this particular case, `ErrSegmentNotLoaded` and `ErrSegmentNotFound`
shall both be ignored preventing return search service unavailable by
mistake.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-20 21:22:43 +08:00
yah01
04b2518ae7
enhance: fix the incorrect init parameter (#29357)
as the `driver_` field is not used so this doesn't matter for now

Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-20 20:50:43 +08:00
aoiasd
46a937ac43
fix: msgstream memory leak caused by config event don't deregister (#29266)
relate: https://github.com/milvus-io/milvus/issues/28620

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-20 20:32:44 +08:00
wei liu
e41fd6fbde
enhance: Move proxy client manager to util package (#28955)
issue:  #28898

This PR move the `ProxyClientManager` to util package, in case of
reusing it's implementation in querycoord

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-20 19:22:42 +08:00
zhuwenxing
1b5ac82c6b
test: [skip e2e] fix getting wrong image tag by short name (#29347)
By adding conjunctions, you can obtain more accurate results.

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2023-12-20 17:34:42 +08:00
cai.zhang
1b4d2674b3
fix: Set the default index name to the name of the existing index (#29275)
issue: #29269

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-12-20 17:22:41 +08:00
XuanYang-cn
7d32861d53
enhance: Add channel check interval params (#29311)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-20 15:06:42 +08:00
Xiaofan
77b291c5dc
fix: Add jitter in GetSyncStaleBufferPolicy (#28626)
related to #28427
Add a jitter in syncStatleBuffer policy so all segments won't flush at
the same time

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2023-12-20 15:00:52 +08:00
Gao
9b52cb6417
enhance: improve reducing results when many segments are filtered (#29073)
Do not fill the invalid ids for the empty results, it will incur useless
memory overhead and reduce overhead when nq and topk is large.

---------

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-12-20 12:56:42 +08:00
aoiasd
010d8362ad
fix: datanode L0 segment num wrong (#29050)
relate:  https://github.com/milvus-io/milvus/issues/27675

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-20 11:20:44 +08:00
congqixia
cbf0f9c527
enhance: change cp metric to absolute unix ts (#29328)
See also #29327

Change channel checkpoint metrics to unix seconds instead of checkpoint
timestamp lag value

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-20 11:04:45 +08:00
yah01
61fc822207
fix: creating growing segments may introduce many threads (#29306)
many growing segments may be created in a short time and there is no
restriction to the process, the CGO call will leave many threads

related: #29282

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-20 10:06:43 +08:00
yah01
8f89e9cf75
enhance: remove all unnecessary string formatting (#29323)
done by two regex expressions:
- `PanicInfo\((.+),[. \n]+fmt::format\(([.\s\S]+?)\)\)`
- `AssertInfo\((.+),[. \n]+fmt::format\(([.\s\S]+?)\)\)`

related: #28811

---------

Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-12-20 10:04:43 +08:00
congqixia
bcf8f27aa7
enhance: refine proxy meta cache partition logic (#29315)
See also #29113

- Unify partition info refresh logic
- Prevent parse partition names for each partition key search request

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-20 10:02:43 +08:00
Bingyi Sun
89b208d27a
enhance: Fix format message (#29159)
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-12-20 09:30:44 +08:00
zhuwenxing
9e846d8db2
test: add bulk insert benchmark for different file size (#29320)
add bulk insert benchmark for different file size

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2023-12-20 09:26:49 +08:00
congqixia
1ee016709d
fix: Unstable TestDataSyncService/TestStartStop unit test (#29291)
fix #29290

Change EXPECT `NotifyCheckpointUpdated` call to `Maybe` expectation

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-19 23:16:42 +08:00
MrPresent-Han
bfca0a7926
fix: refine skipIndex to resolve cyclic dependcy(#29132) (#29189)
related: #29132

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-12-19 10:26:40 +08:00
zhagnlu
56d7225673
fix: Add LD_LIBRARY_PATH when local start milvus (#29287)
#28165

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-12-19 10:08:41 +08:00
nico
4eddcc2113
enhance: update pymilvus version (#29293)
Signed-off-by: nico <cheng.yuan@zilliz.com>
2023-12-19 09:00:42 +08:00
nico
95ada1b679
test: update test cases (#29267)
issue: #29227

Signed-off-by: nico <cheng.yuan@zilliz.com>
2023-12-18 19:28:42 +08:00
aoiasd
011a9cee79
test: remove outdated e2e test case beacause disknn support search with search list num large than 200 or 65535 (#28949)
relate: https://github.com/milvus-io/milvus/issues/28745

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-18 19:26:52 +08:00
PowderLi
bcd6865b29
enhance: add 3 builtin roles (#28961)
issue: #28960 [milvus-proto
#212](https://github.com/milvus-io/milvus-proto/issues/212)

add new configuration: builtinRoles
user can define roles in config file: `milvus.yaml`

there is an example:
1. db_ro, only have read privileges, include load
2. db_rw, read and write privileges, include create/drop/rename
collection
3. db_admin, not only read and write privileges, but also user
administration

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-18 14:28:41 +08:00
zhagnlu
a602171d06
enhance: Refactor runtime and expr framework (#28166)
#28165

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-12-18 12:04:42 +08:00
zhuwenxing
438f39e268
test: fix gen array field (#29233)
fix gen array field

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2023-12-18 11:30:46 +08:00
wei liu
820ee692fc
enhance: Add config for querycoord auto balance channel (#29231)
issue: #23726
This PR add control config to querycoord's background auto balance
channel operation

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-18 10:00:40 +08:00