yihao.dai
b1e74dc7cb
enhance: [cherry-pick] Decouple compaction from shard ( #34157 )
...
This PR cherry-picks the following commits:
- Implement task limit control logic in datanode.
https://github.com/milvus-io/milvus/pull/32881
- Load bf from storage instead of memory during L0 compaction.
https://github.com/milvus-io/milvus/pull/32913
- Remove dependencies on shards (e.g. SyncSegments, injection).
https://github.com/milvus-io/milvus/pull/33138
- Rename Compaction interface to CompactionV2.
https://github.com/milvus-io/milvus/pull/33858
- Remove the unused residual compaction logic.
https://github.com/milvus-io/milvus/pull/33932
issue: https://github.com/milvus-io/milvus/issues/32809
pr: https://github.com/milvus-io/milvus/pull/32881 ,
https://github.com/milvus-io/milvus/pull/32913 ,
https://github.com/milvus-io/milvus/pull/33138 ,
https://github.com/milvus-io/milvus/pull/33858 ,
https://github.com/milvus-io/milvus/pull/33932
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-25 20:22:03 +08:00
sammy.huang
2ecfe1599c
[automated] Update cpu Builder image changes ( #34079 )
...
Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
2024-06-25 19:02:03 +08:00
yihao.dai
3ae1865ffb
fix: Prevent import from generating orphaned files ( #34070 ) ( #34071 )
...
When import failed, mark the import segment as dropped instead of drop
it directly to prevent generating orphaned files.
issue: https://github.com/milvus-io/milvus/issues/34068
pr: https://github.com/milvus-io/milvus/pull/34070
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-25 17:00:04 +08:00
cai.zhang
65050b3117
fix: [pick] Fallback field type when it isn't in request ( #33832 ) ( #34133 )
...
issue: #33432
pr: #33832
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-06-25 14:56:04 +08:00
cqy123456
f5344abdaf
enhance: [cherry-pick]growing segment support mmap ( #34110 )
...
issue: issue: https://github.com/milvus-io/milvus/issues/32984
related pr: https://github.com/milvus-io/milvus/pull/32633 ,
https://github.com/milvus-io/milvus/pull/33951 ,
https://github.com/milvus-io/milvus/pull/33993
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-25 14:52:07 +08:00
sammy.huang
bd8ad106b9
enhance: revert dead try on uploading codecov ( #34131 )
...
This reverts commit 5a14f65ee4
2024-06-25 11:51:35 +08:00
XuanYang-cn
a33b68678d
enhance: [cherry-pick] Move compactor into sub package ( #34098 )
...
This PR consists of the following commits:
- enhance: Tidy compactor and remove dup codes (#32198 )
- fix: Fix l0 compactor may cause DN from OOM (#33554 )
- enhance: Add deltaRowCount in l0 compaction (#33997 )
- enhance: enable stream writer in compactions (#32612 )
- fix: turn on compression on stream writers (#34067 )
- fix: adding blob memory size in binlog serde (#33324 )
See also: #32451 , #33547 , #33998 , #31679
pr: #32198 , #33554 , #33997 , #32612
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
2024-06-25 11:16:02 +08:00
Patrick Weizhi Xu
1168e8c107
enhance: [2.4] support integral type for MV and skip MV if there is only one category ( #34005 )
...
pr: #33161
issue: #29892
---------
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-06-24 19:12:02 +08:00
GenkenWei
059aaad3bb
fix: update ubuntu base image version ( #33927 )
...
related to #33945
FIX CVEs of milvus base image: LOW: 32, MEDIUM: 50, **Total FIX: 82**
Signed-off-by: weizhenkun <zhenkun.wei@qq.com>
Co-authored-by: weizhenkun <zhenkun.wei@qq.com>
2024-06-24 14:16:02 +08:00
yanliang567
59d910320d
test:[cherry-pick]Update tests for range search and add test for query with dup ids ( #34069 )
...
related issue: https://github.com/milvus-io/milvus/issues/33883
pr: #34057
Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2024-06-24 11:34:03 +08:00
Jiquan Long
22e6807e9a
feat: support inverted index for array ( #33452 ) ( #34053 )
...
pr: https://github.com/milvus-io/milvus/pull/33184
pr: https://github.com/milvus-io/milvus/pull/33452
pr: https://github.com/milvus-io/milvus/pull/33633
issue: https://github.com/milvus-io/milvus/issues/27704
Co-authored-by: xiaocai2333 <cai.zhang@zilliz.com>
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
2024-06-24 10:50:03 +08:00
zhuwenxing
630a726f35
test: refine restful testcases trace ( #34065 )
...
pr: https://github.com/milvus-io/milvus/pull/34066
---------
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-06-24 10:40:03 +08:00
sre-ci-robot
f4debe5e5e
[automated] Update gpu Builder image changes ( #34031 )
...
Update gpu Builder image changes
See changes:
b3d425f50asre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-06-24 10:38:13 +08:00
sre-ci-robot
63c9e6e023
[automated] Update cpu Builder image changes ( #34030 )
...
Update cpu Builder image changes
See changes:
b3d425f50asre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-06-24 10:36:25 +08:00
sammy.huang
9ea4aa9bc0
enhance: get environment variable from .env ( #34080 )
...
Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
2024-06-23 09:34:14 +08:00
yihao.dai
2fda43e49f
fix: Do compressBinlog to fix logID 0 ( #34060 ) ( #34062 )
...
Do compressBinlog to ensure that reloadFromKV will fill binlogs' logID
after datacoord restarts.
issue: https://github.com/milvus-io/milvus/issues/34059
pr: https://github.com/milvus-io/milvus/pull/34060
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-22 19:42:02 +08:00
sammy.huang
83ff759195
enhance: upgrade build-env to ubuntu 22.04 and gcc12 ( #33961 )
...
pr: #33959
Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
2024-06-22 19:40:05 +08:00
wei liu
061a00c58f
enhance: Enable database level replica num and resource groups for loading collection ( #33052 ) ( #33981 )
...
pr: #33052
issue: #30040
This PR introduce two database level props:
1. database.replica.number
2. database.resource_groups
User can set those two database props by AlterDatabase API, then can
load collection without specified replica_num and resource groups. then
it will use database level load param when try to load collections.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-21 16:56:02 +08:00
zhuwenxing
89461db5f3
test: update the lib of bf16 ( #34044 )
...
pr: https://github.com/milvus-io/milvus/pull/34043
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-06-21 14:20:08 +08:00
congqixia
e02a95e3c2
fix: [2.4] Return record with largest timestamp for entires with same PK ( #33936 ) ( #34024 )
...
Cherry-pick from master
pr: #33936
See also #33883
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-21 14:14:01 +08:00
congqixia
891a94ad9e
fix: [2.4] Check nodeID wildcard when removing pkOracle ( #33895 ) ( #34020 )
...
Cherry-pick from master
pr: #33895
See also #33894
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-21 12:04:00 +08:00
jaime
c219dca001
fix: metrics database_num is 0 after restarting rootcoord ( #34010 )
...
issue: https://github.com/milvus-io/milvus/issues/34041
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-21 11:46:02 +08:00
wei liu
7d1d5a838a
fix: Fix GetReplicas API return nil status ( #33715 ) ( #34019 )
...
issue: #33702
pr: #33715
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-21 10:26:02 +08:00
wei liu
fbc8fb3cb2
enhance: Skip return data distribution if no change happen ( #32814 ) ( #33985 )
...
issue: #32813
pr: #32814
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-21 10:24:12 +08:00
elstic
5952c09925
test: [cherry-pick] optimizing variable names ( #34036 )
...
pr: https://github.com/milvus-io/milvus/pull/34035
Signed-off-by: elstic <hao.wang@zilliz.com>
2024-06-21 10:12:01 +08:00
XuanYang-cn
e55fee6b04
enhance: Add deltaRowCount in l0 compaction ( #33843 )
...
See also: #33998
pr: #33997
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-06-20 19:32:02 +08:00
sre-ci-robot
8f6f6dc29f
[automated] Bump milvus version to v2.4.5 ( #34028 )
...
Bump milvus version to v2.4.5
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-06-20 17:38:01 +08:00
shaoting-huang
b3d425f50a
enhance: Upgrade go version from 1.20 to 1.21 ( #33940 )
...
issue #32982
related pr in master:
pr: #33047 #33150 #33176 #33351 #33202 #33192
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-06-20 14:52:00 +08:00
congqixia
4424c9e5e7
fix: [2.4] Remove loopclosure issue in ChannelManagerImplV2 ( #33989 ) ( #34004 )
...
Cherry-pick from master
pr: #33989
See also #33987
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-20 14:42:00 +08:00
wei liu
a7ae45c91c
enhance: Add trace for bf cost in l0 compactor ( #33898 )
...
pr: #33860
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-20 14:40:15 +08:00
elstic
e0e2ed073b
test: [cherry-pick] change the number of insertion vectors ( #34015 )
...
pr: https://github.com/milvus-io/milvus/pull/34014
Signed-off-by: elstic <hao.wang@zilliz.com>
2024-06-20 13:58:05 +08:00
congqixia
c28b69384a
fix: [2.4] Check the correct return error in MultiRemove ( #33926 ) ( #33966 )
...
Cherry-pick from master
pr: #33926
See also #33925
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-20 11:28:02 +08:00
Jiquan Long
032685239f
enhance: fix copying hits of inverted index twice ( #33968 ) ( #33975 )
...
issue: https://github.com/milvus-io/milvus/issues/29793
pr: https://github.com/milvus-io/milvus/pull/33968
The custom `VecCollector` have already transformed the results into
vector of offsets, no need to copy them twice.
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-06-20 10:06:06 +08:00
wei liu
87508c3390
enhance: Avoid to iterate whole segment list for each task's process( #33943 ) ( #33976 )
...
pr: #33943
when querycoord process segment task, it will try to iterate whole
segment list to checke whether segment is loaded, which cost too much
cpu if there has thousands of segments.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-20 10:00:05 +08:00
zhuwenxing
c61740d407
test: [cherry-pick]add test case for bulkwriter ( #33914 )
...
pr: https://github.com/milvus-io/milvus/pull/33879
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-06-20 09:22:18 +08:00
presburger
30b70b05d0
fix: fix milvus gpu compile error ( #33990 )
...
NVIDIA RAFT uses customized versions of spdlog and fmt, which conflict
with the versions provided by Conan. Therefore, it is necessary to
isolate the build environment for knowhere to avoid these conflicts.
issue: #33991
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
2024-06-20 09:20:00 +08:00
nico
27905de192
test: update test cases ( #33971 )
...
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-06-19 16:48:01 +08:00
elstic
caa1b0f06e
test: [cherry-pick] add more sparse test cases ( #33918 )
...
issue: https://github.com/milvus-io/milvus/issues/31483
pr: https://github.com/milvus-io/milvus/pull/33916
Signed-off-by: elstic <hao.wang@zilliz.com>
2024-06-19 15:24:09 +08:00
wei liu
2d3c5191b8
enhance: Refine name rule check error msg ( #33815 ) ( #33900 )
...
pr: #33815
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-19 14:26:01 +08:00
congqixia
60695bdb44
enhance: [2.4] Set maxPartitionNum default value to 1024 ( #33950 )
...
Cherry-pick from master
pr: #33949
See also #30059
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-18 18:42:00 +08:00
presburger
79546a6c25
fix: fix not find arrow::parquet lib ( #33941 )
...
Upgrade the Arrow version and remove unnecessary library link
dependencies.
issue: #33942
pr: https://github.com/milvus-io/milvus/pull/33928
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
2024-06-18 13:09:59 +08:00
nico
530756fe42
test: update test cases ( #33899 )
...
pr: #33096
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-06-18 11:35:59 +08:00
wei liu
aafa86095f
enhance: Force to reset coord connection for unavailable error ( #33910 )
...
pr: #33908
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-18 11:15:58 +08:00
wei liu
4513569207
enhance: add restful api to trigger component stop ( #32076 ) ( #33799 )
...
issue: #32698
pr: #32076
This PR add two rest api for component stop and status check:
1. `/management/stop?role=querynode` can stop the specified component
2. `/management/check/ready?role=rootcoord` can check whether the target
component is serviceable
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-17 21:53:59 +08:00
Gao
08c096cf55
enhance: Use primitive type for vectorType ( #33911 )
...
issue: #22837
pr: #33868
Use primitive type instead of proto enum type for queryHook to recognize
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-06-17 19:05:58 +08:00
congqixia
e8071830fa
fix: [2.4] Prevent use captured iteration variable partitionID ( #33907 )
...
Cherry-pick from master
pr: #33906
See also #33902
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-17 19:02:00 +08:00
Gao
5fc1370f6f
enhance: [2.4] autoindex for multi data type ( #33867 )
...
issue: #22837
pr: https://github.com/milvus-io/milvus/pull/33868
- opensource autoindex support
- metric type check for different data types
- autoindex data type for search param
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-06-14 23:26:00 +08:00
congqixia
59144d8246
enhance: Bump milvus & proto version to v2.4.5 ( #33852 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-14 22:27:59 +08:00
cqy123456
562369627d
enhance: [cherry-pick]check index with data type in knowhere api ( #33878 )
...
issue: https://github.com/milvus-io/milvus/issues/22837
related: https://github.com/milvus-io/milvus/pull/33880
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-14 19:45:58 +08:00
jaime
fd1c7b1a1c
enhance: enable flush rate limiter of collection level ( #33864 )
...
pr: #33837
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-14 16:27:57 +08:00
