congqixia
1833913f44
enhance: Add streaming forward policy switch for delegator ( #36330 )
...
Related to #35303
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-23 18:01:12 +08:00
smellthemoon
6e880d19a8
enhance: add search params in search request in restful ( #36304 )
...
https://github.com/milvus-io/milvus/issues/36321
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-23 16:51:12 +08:00
SimFG
c50fe71163
fix: long buffering causes mq to be unable to receive messages. ( #36420 )
...
- issue: #36397
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-23 16:33:18 +08:00
Chun Han
eb23e23cd2
enhance: refine parameter relationship for hybridsearch_group_by( #35096 ) ( #36289 )
...
related: #35096
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-20 14:55:11 +08:00
cai.zhang
4b077e1bd2
fix: Fix the compatibility bug between stats task and segment ( #36359 )
...
issue: #33744
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-20 14:33:11 +08:00
congqixia
d2c774fb6d
fix: Return all compactTo segments after support split ( #36361 )
...
Related to #36360
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-20 14:11:11 +08:00
wei liu
3b10085f61
enhance: Optimize workload based replica selection policy ( #36181 )
...
issue: #35859
This PR introduce two new param: toleranceFactor and checkRequestNum,
after every checkRequestNum request has been assigned, try to compute
querynode's workload score.
if the diff is less than the toleranceFactor, replica selection policy
will fallback to round_robin, which reduce the average cost to about
500ns.
if the diff is larger than the toleranceFactor, replica selection policy
will compute querynode's score to select the target node with smallest
score in every assigment.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-20 12:33:11 +08:00
Ted Xu
363004fd44
enhance: simplify reduction on single search result ( #36334 )
...
See: #36122
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-09-20 11:59:10 +08:00
smellthemoon
89397d1e66
enhance: adjust parquet reader type check with null type ( #36266 )
...
#36252
remove no need type check. if users use null type writer to write
parquet, hope it successfully.
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-19 18:43:10 +08:00
smellthemoon
e84c4f2a92
fix: to fill default value when nullable and default value are both enable ( #36030 )
...
#36003
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-19 17:35:11 +08:00
Zhen Ye
f65261215b
fix: streaming node health check panic ( #36336 )
...
issue: #36335
Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-19 17:11:12 +08:00
aoiasd
139787371e
feat: support embedding bm25 sparse vector and flush bm25 stats log ( #36036 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-19 10:57:12 +08:00
sre-ci-robot
167e4fb10d
[automated] Update Knowhere Commit ( #36352 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-09-19 01:01:10 +08:00
Zhen Ye
47da9023a6
fix: add future stateful lock ( #36332 )
...
issue: #36323
Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-18 20:15:11 +08:00
Bingyi Sun
23b95aeba3
fix: remove element type check ( #35828 )
...
https://github.com/milvus-io/milvus/issues/36275
Array's element type is not same with schema's. It is INT32 for INT16
and INT8
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-09-18 11:37:10 +08:00
zhenshan.cao
dcd904d2fa
fix: keep inner topK to avoid exceeding efSearch ( #36284 )
...
issue: #https://github.com/milvus-io/milvus/issues/36243
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-09-16 11:01:25 +08:00
yihao.dai
763fd0dfc5
enhance: Use a separate mmap config for chunk cache ( #36276 )
...
issue: https://github.com/milvus-io/milvus/issues/35273
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-15 16:23:09 +08:00
cai.zhang
517f8b3755
enhance: Refine the code for returning error ( #36103 )
...
issue: #36023
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-15 15:23:14 +08:00
cai.zhang
e486442ca6
enhance: Disallow the keywords as a field name or dynamic field name ( #36101 )
...
issue: #35873
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-15 15:19:14 +08:00
yihao.dai
a61668c77e
feat: Introduce stats task for import ( #35868 )
...
This PR introduce stats task for import:
1. Define new `Stats` and `IndexBuilding` states for importJob
2. Add new stats step to the import process: trigger the stats task and
wait for its completion
3. Abort stats task if import job failed
issue: https://github.com/milvus-io/milvus/issues/33744
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-15 15:17:08 +08:00
jaime
2ff3765058
enhance: catch std::stoi exception and improve error msg ( #36267 )
...
issue: #36255
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-14 16:17:08 +08:00
congqixia
3352030a84
enhance: Graceful stop flowgraph manager when stopping datanode ( #36229 )
...
Flowgraph manager is not stopped durong datanode stopping procedure
which may lead to unexpect flowgraph behavior during/after datanode stop
progress.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-14 15:55:08 +08:00
Chun Han
b8b4aea4f5
enhance: restrict max group size( #33544 ) ( #36223 )
...
related: #33544
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-14 15:45:08 +08:00
smellthemoon
fc1bdd4c84
fix: to forbid bulk insert with nullable field in numpy files ( #36246 )
...
#36241
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-14 15:35:07 +08:00
wei liu
329fb421cd
fix: fix search/query/count may access same growing and sealed segment ( #36258 )
...
issue: #36257
during syncTargetVersion, sealed segment should be excluded, to avoid
it's growing segment be conusmed from stream again.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-14 14:21:07 +08:00
congqixia
d07338b4b9
enhance: Ignore index check for non-loaded vector field ( #36170 )
...
Related to #35996
For `Field Partial Load` feature, Milvus shall ignore index check for
non-loaded vector field.
Also, this PR unifies the logic of index check for load collection and
load partitions tasks.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-14 11:01:08 +08:00
smellthemoon
2d05b7f219
fix: to add check in partition key field when set nullable==true ( #36218 )
...
#36213
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-14 10:53:08 +08:00
congqixia
3bc7d63be9
fix: overwrite correct selection when pk duplicated ( #35826 )
...
Related to #35505
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-14 10:27:08 +08:00
zhenshan.cao
9d8d332c88
fix: Fix improper use of offset in HybridSearch ( #36244 )
...
issue :https://github.com/milvus-io/milvus/issues/36243
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-09-13 22:05:15 +08:00
zhagnlu
489087d18b
enhance: refactor executor framework V2 ( #35251 )
...
#32636
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-13 20:57:09 +08:00
jaime
22cce44afc
fix: metrics stored_index_files_size is never cleared ( #36160 )
...
issue: #36159
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-13 20:09:15 +08:00
congqixia
58d3200986
enhance: Filter out non-hit delete records during load delta ( #36207 )
...
Related to #35303
This PR utilizes pk index in segment to exclude non-hit delete record
during load delete records. This ability is crucial when l0/delete
forward policy only replies on segment itself(without BF filtering).
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-13 19:05:08 +08:00
wei liu
f7d950d465
fix: [skip e2e] Fix unstable ut TestCollectionObserver ( #36231 )
...
issue: #36237
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-13 19:01:09 +08:00
wei liu
fb2a41a94c
fix: Clean dirty segment/channel on querynode ( #36202 )
...
issue: #36201
after querynode has been remove from replica, all dirty segment/channel
on it should be released.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-13 18:15:08 +08:00
Zhen Ye
b4c1603a57
fix: skip recover flusher on streaming node when channel is dropped ( #36149 )
...
issue: #36016,#36230
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-13 17:19:08 +08:00
wei liu
bd658a6510
enhance: Enable dynamic update replica selection policy ( #35860 )
...
issue: #35859
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-13 17:05:15 +08:00
Zhen Ye
c03eb6f664
fix: streaming node consume blocks if recv message is too large ( #36151 )
...
issue: #36081
Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-13 16:41:08 +08:00
aoiasd
c22a2cebb6
fix: split stream query result to avoid grpc response too large error ( #36090 )
...
relate: https://github.com/milvus-io/milvus/issues/36089
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-13 15:07:09 +08:00
Jiquan Long
f0f2fb4cf0
enhance: span tracing of c++ part ( #36205 )
...
fix: https://github.com/milvus-io/milvus/issues/36204
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-09-13 11:19:09 +08:00
wei liu
06a706e5f0
fix: fix ReadWrite privilege group deny all global API ( #36144 )
...
issue: #35471
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-13 10:33:08 +08:00
congqixia
c6eb6c7cb2
enhance: Add error handler for write buffer ( #36216 )
...
Related to #36215
This PR add error handler setting option providing the possibility to
change error handling behavior other than panicking.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-13 10:11:09 +08:00
Ted Xu
b9c037f558
feat: adding cache to expression parse ( #36185 )
...
See #36122
This PR improves the proxy node performance by adding cache to
expression parse.
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-09-13 10:03:09 +08:00
cai.zhang
8395c8a8db
enhance: Update stats task to optional ( #35947 )
...
issue: #33744
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-12 20:37:08 +08:00
XuanYang-cn
e8840a1b41
enhance: Add metrics for Delete entries num of L0seg ( #36175 )
...
- Add metrics *DataCoordL0DeleteEntriesNum*
- Remove metrics *DataCoordRateStoredL0Segment*
See also: #36147
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-09-12 18:07:08 +08:00
zhagnlu
5e5e87cc2f
enhance: rename some params and reduce default bitmapCardinalityLimit… ( #36138 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-12 12:09:08 +08:00
congqixia
11dbe1e755
enhance: Add L0 forward policy to support remote load ( #36189 )
...
Related to #35303
This PR add a param item to support change l0 forward behavior from bf
filtering and forward to remote load.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-12 12:01:08 +08:00
aoiasd
da227ff9a1
feat: Support create collection with functions ( #35973 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
Support create collection with functions. Prepare for support bm25
function.
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-12 10:43:06 +08:00
congqixia
700a443e85
enhance: Check ANN field loaded in proxy ( #36172 )
...
Related to #35415
Currently ANN field is loaded in Parital load is not performed in proxy,
this cause error message not clear and no error returned when collection
is empty
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-11 17:43:06 +08:00
Ted Xu
e7ea1d7a04
enhance: improve log encoding performance on proxy nodes ( #36123 )
...
See #36122
This PR is designed to enhance log performance through two improvements:
1. Optimize JSON encoding by switching JSON serializer to
`json-iterator`.
2. Adding support of lazy initialization `WithLazy`.
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-09-11 14:51:07 +08:00
smellthemoon
3f75bf1f20
fix: clustering compact not support null ( #36152 )
...
#36055
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-11 14:49:06 +08:00
