test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-02 20:09:57 +08:00
Code Issues Actions 12 Packages Projects Releases Wiki Activity
19,997 Commits 94 Branches 105 Tags 184 MiB
fc6bd387b8
Commit Graph

19997 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
zhagnlu
fc6bd387b8
fix: expr add skip using index while index exists (#34202) 
This PR cherry-pick part from commit:

-  enhance: add skip using array index when some situation #33947 
-  fix: [ut] regex query under unsupported index  #34087

pr: #33947, #34087

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-30 20:03:13 +08:00
Bingyi Sun
460815ceab
fix: fix partition loaded num metric (#33316) (#34195) 
issue: https://github.com/milvus-io/milvus/issues/32108
related pr: #33316

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-06-28 14:58:05 +08:00
cai.zhang
85aa929749
enhance: Set the timeout for SyncSegments based on the number of segments (#34228) 
issue: #32809 

master pr: #34226

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-28 14:18:06 +08:00
zhuwenxing
9cb3658d74
test: [cherry-pick]remove dup code (#34243) 
pr: https://github.com/milvus-io/milvus/pull/34242

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-28 10:44:05 +08:00
sammy.huang
7b4c1a0a96
enhance: [skip e2e] use disk ann when os is ubuntu, rocky or amazon (#34244) 
pr: https://github.com/milvus-io/milvus/pull/34236

Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
 2024-06-27 20:50:13 +08:00
cai.zhang
1c6e850f73
enhance: [cherry-pick] Periodically synchronize segments to datanode watcher (#33420) (#34186) 
This PR primary picks up the SyncSegments functionality, including the
following commits:
- main functionality: https://github.com/milvus-io/milvus/pull/33420
- related fixes:
  - https://github.com/milvus-io/milvus/pull/33664
  - https://github.com/milvus-io/milvus/pull/33829
  - https://github.com/milvus-io/milvus/pull/34056
  - https://github.com/milvus-io/milvus/pull/34156

issue: #32809 
master pr: #33420, #33664, #33829, #34056, #34156

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-27 11:24:05 +08:00
cai.zhang
42096a2337
enhance: [cherry-pick] Reduce the frequency of index ralted logs (#33050) (#34198) 
issue:  #33001


pr: #33050 , #33212 , #33321

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-27 11:08:05 +08:00
congqixia
88766bb714
fix: [2.4] Pass otlpSecure config when setup segcore tracing (#34193) (#34210) 
Cherry-pick from master
pr: #34193
See also #34192

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-27 10:22:06 +08:00
congqixia
f741bb7526
enhance: [2.4] Avoid merging insert data when buffering insert msgs (#34205) 
Cherry-pick from master
pr: #33526 #33817
See also #33561

This PR:
- Use zero copy when buffering insert messages
- Make `storage.InsertCodec` support serialize multiple insert data
chunk into same batch binlog files

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-27 10:14:05 +08:00
congqixia
aae94d7c40
enhance: [2.4] Unify DeleteLog parsing code (#34188) 
Cherry-pick from master
pr: #34009 
See also #33787

The parsing delete log is distributed in lots of places, which is not
recommended and hard to maintain.

This PR abstract common parsing logic into DeleteLog.Parse method to
unify implementation and make it easier to replace json parsing lib.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-27 10:12:13 +08:00
cqy123456
0bd93de627
enhance: [cherry-pick]remove CheckVecIndexWithDataTypeExist function in pkg (#34104) 
issue: https://github.com/milvus-io/milvus/issues/22837
related pr: https://github.com/milvus-io/milvus/pull/34102

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-26 21:14:26 +08:00
jaime
7f636dbdf6
enhance: refine error message for invalid limit parameter (#34179) 
pr: #34178

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-26 20:46:06 +08:00
congqixia
07a05db51b
fix: [2.4] Wrap init segcore tracing with golang timeout (#33494) (#34191) 
Cherry-pick from master
pr: #33494
See also #33483

Wrap `C.InitTrace` & `C.SetTrace` with timeout preventing otlp
initializtion hangs forever when endpoint is not set correctly

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-26 19:44:05 +08:00
wei liu
18a0efe737
enhance: Avoid search querynode return nil status in response (#34100) (#34189) 
pr: #34100

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-26 18:52:04 +08:00
sre-ci-robot
a9a0981157
[automated] Update cpu Builder image changes (#34167) 
Update cpu Builder image changes
See changes:
2ecfe1599c
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2024-06-26 17:56:05 +08:00
jaime
6423b6c718
enhance: move rocksmq from internal to pkg (#34165) 
pr:  https://github.com/milvus-io/milvus/pull/33881
issue:  https://github.com/milvus-io/milvus/issues/33956

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-26 13:36:05 +08:00
congqixia
3cf526e0cc
fix: [2.4] Deep copy ImportTask.segmentsInfo to prevent data race (#34090) (#34126) 
Cherry-pick from master
pr: #34090
See also #34089

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-26 11:50:11 +08:00
congqixia
26b2e1d43c
fix: [2.4] Make querycoord panick when rg metastore sync fail (#34106) (#34127) 
Cherry-pick from master
pr: #34106
See also #34047

When `unassignNode` sync resource group with node removed failed

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-26 10:04:03 +08:00
cqy123456
3e51256fb4
enhance: [cherry-pick]update new mmap config parmeters version (#34145) 
issue: https://github.com/milvus-io/milvus/issues/32984
related pr: https://github.com/milvus-io/milvus/pull/34143

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-25 21:24:05 +08:00
yihao.dai
b1e74dc7cb
enhance: [cherry-pick] Decouple compaction from shard (#34157) 
This PR cherry-picks the following commits:

- Implement task limit control logic in datanode.
https://github.com/milvus-io/milvus/pull/32881
- Load bf from storage instead of memory during L0 compaction.
https://github.com/milvus-io/milvus/pull/32913
- Remove dependencies on shards (e.g. SyncSegments, injection).
https://github.com/milvus-io/milvus/pull/33138
- Rename Compaction interface to CompactionV2.
https://github.com/milvus-io/milvus/pull/33858
- Remove the unused residual compaction logic.
https://github.com/milvus-io/milvus/pull/33932

issue: https://github.com/milvus-io/milvus/issues/32809

pr: https://github.com/milvus-io/milvus/pull/32881,
https://github.com/milvus-io/milvus/pull/32913,
https://github.com/milvus-io/milvus/pull/33138,
https://github.com/milvus-io/milvus/pull/33858,
https://github.com/milvus-io/milvus/pull/33932

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-25 20:22:03 +08:00
sammy.huang
2ecfe1599c
[automated] Update cpu Builder image changes (#34079) 
Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
 2024-06-25 19:02:03 +08:00
yihao.dai
3ae1865ffb
fix: Prevent import from generating orphaned files (#34070) (#34071) 
When import failed, mark the import segment as dropped instead of drop
it directly to prevent generating orphaned files.

issue: https://github.com/milvus-io/milvus/issues/34068

pr: https://github.com/milvus-io/milvus/pull/34070

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-25 17:00:04 +08:00
cai.zhang
65050b3117
fix: [pick] Fallback field type when it isn't in request (#33832) (#34133) 
issue: #33432
pr: #33832

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-25 14:56:04 +08:00
cqy123456
f5344abdaf
enhance: [cherry-pick]growing segment support mmap (#34110) 
issue: issue: https://github.com/milvus-io/milvus/issues/32984
related pr: https://github.com/milvus-io/milvus/pull/32633,
https://github.com/milvus-io/milvus/pull/33951,
https://github.com/milvus-io/milvus/pull/33993

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-25 14:52:07 +08:00
sammy.huang
bd8ad106b9
enhance: revert dead try on uploading codecov (#34131) 
This reverts commit 5a14f65ee4.

Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
 2024-06-25 11:51:35 +08:00
XuanYang-cn
a33b68678d
enhance: [cherry-pick] Move compactor into sub package (#34098) 
This PR consists of the following commits:

- enhance: Tidy compactor and remove dup codes (#32198)
- fix: Fix l0 compactor may cause DN from OOM (#33554)
- enhance: Add deltaRowCount in l0 compaction (#33997)
- enhance: enable stream writer in compactions (#32612)
- fix: turn on compression on stream writers (#34067)
- fix: adding blob memory size in binlog serde (#33324)

See also: #32451, #33547, #33998, #31679
pr: #32198, #33554, #33997, #32612

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
 2024-06-25 11:16:02 +08:00
Patrick Weizhi Xu
1168e8c107
enhance: [2.4] support integral type for MV and skip MV if there is only one category (#34005) 
pr: #33161
issue: #29892

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-06-24 19:12:02 +08:00
GenkenWei
059aaad3bb
fix: update ubuntu base image version (#33927) 
related to #33945
FIX CVEs of milvus base image: LOW: 32,  MEDIUM: 50, **Total FIX: 82**

![image](https://github.com/milvus-io/milvus/assets/27683687/020fbb73-9a1f-42cb-8224-a595179b3533)

![image](https://github.com/milvus-io/milvus/assets/27683687/fef403a5-c658-4ee6-baf4-ace3a0387799)

Signed-off-by: weizhenkun <zhenkun.wei@qq.com>
Co-authored-by: weizhenkun <zhenkun.wei@qq.com>
 2024-06-24 14:16:02 +08:00
yanliang567
59d910320d
test:[cherry-pick]Update tests for range search and add test for query with dup ids (#34069) 
related issue: https://github.com/milvus-io/milvus/issues/33883
pr: #34057

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
 2024-06-24 11:34:03 +08:00
Jiquan Long
22e6807e9a
feat: support inverted index for array (#33452) (#34053) 
pr: https://github.com/milvus-io/milvus/pull/33184
pr: https://github.com/milvus-io/milvus/pull/33452
pr: https://github.com/milvus-io/milvus/pull/33633
issue: https://github.com/milvus-io/milvus/issues/27704
Co-authored-by: xiaocai2333 <cai.zhang@zilliz.com>

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
 2024-06-24 10:50:03 +08:00
zhuwenxing
630a726f35
test: refine restful testcases trace (#34065) 
pr: https://github.com/milvus-io/milvus/pull/34066

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-24 10:40:03 +08:00
sre-ci-robot
f4debe5e5e
[automated] Update gpu Builder image changes (#34031) 
Update gpu Builder image changes
See changes:
b3d425f50a
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2024-06-24 10:38:13 +08:00
sre-ci-robot
63c9e6e023
[automated] Update cpu Builder image changes (#34030) 
Update cpu Builder image changes
See changes:
b3d425f50a
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2024-06-24 10:36:25 +08:00
sammy.huang
9ea4aa9bc0
enhance: get environment variable from .env (#34080) 
Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
 2024-06-23 09:34:14 +08:00
yihao.dai
2fda43e49f
fix: Do compressBinlog to fix logID 0 (#34060) (#34062) 
Do compressBinlog to ensure that reloadFromKV will fill binlogs' logID
after datacoord restarts.

issue: https://github.com/milvus-io/milvus/issues/34059

pr: https://github.com/milvus-io/milvus/pull/34060

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-22 19:42:02 +08:00
sammy.huang
83ff759195
enhance: upgrade build-env to ubuntu 22.04 and gcc12 (#33961) 
pr: #33959

Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
 2024-06-22 19:40:05 +08:00
wei liu
061a00c58f
enhance: Enable database level replica num and resource groups for loading collection (#33052) (#33981) 
pr: #33052

issue: #30040

This PR introduce two database level props:
1. database.replica.number
2. database.resource_groups

User can set those two database props by AlterDatabase API, then can
load collection without specified replica_num and resource groups. then
it will use database level load param when try to load collections.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-21 16:56:02 +08:00
zhuwenxing
89461db5f3
test: update the lib of bf16 (#34044) 
pr: https://github.com/milvus-io/milvus/pull/34043

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-21 14:20:08 +08:00
congqixia
e02a95e3c2
fix: [2.4] Return record with largest timestamp for entires with same PK (#33936) (#34024) 
Cherry-pick from master
pr: #33936
See also #33883

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-21 14:14:01 +08:00
congqixia
891a94ad9e
fix: [2.4] Check nodeID wildcard when removing pkOracle (#33895) (#34020) 
Cherry-pick from master
pr: #33895
See also #33894

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-21 12:04:00 +08:00
jaime
c219dca001
fix: metrics database_num is 0 after restarting rootcoord (#34010) 
issue: https://github.com/milvus-io/milvus/issues/34041

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-21 11:46:02 +08:00
wei liu
7d1d5a838a
fix: Fix GetReplicas API return nil status (#33715) (#34019) 
issue: #33702
pr: #33715

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-21 10:26:02 +08:00
wei liu
fbc8fb3cb2
enhance: Skip return data distribution if no change happen (#32814) (#33985) 
issue: #32813
pr: #32814

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-21 10:24:12 +08:00
elstic
5952c09925
test: [cherry-pick] optimizing variable names (#34036) 
pr: https://github.com/milvus-io/milvus/pull/34035

Signed-off-by: elstic <hao.wang@zilliz.com>
 2024-06-21 10:12:01 +08:00
XuanYang-cn
e55fee6b04
enhance: Add deltaRowCount in l0 compaction (#33843) 
See also: #33998 
pr: #33997

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-20 19:32:02 +08:00
sre-ci-robot
8f6f6dc29f
[automated] Bump milvus version to v2.4.5 (#34028) 
Bump milvus version to v2.4.5
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2024-06-20 17:38:01 +08:00
shaoting-huang
b3d425f50a
enhance: Upgrade go version from 1.20 to 1.21 (#33940) 
issue #32982

related pr in master:
pr: #33047 #33150 #33176 #33351 #33202 #33192

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
 2024-06-20 14:52:00 +08:00
congqixia
4424c9e5e7
fix: [2.4] Remove loopclosure issue in ChannelManagerImplV2 (#33989) (#34004) 
Cherry-pick from master
pr: #33989
See also #33987

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-20 14:42:00 +08:00
wei liu
a7ae45c91c
enhance: Add trace for bf cost in l0 compactor (#33898) 
pr: #33860

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-20 14:40:15 +08:00
elstic
e0e2ed073b
test: [cherry-pick] change the number of insertion vectors (#34015) 
pr: https://github.com/milvus-io/milvus/pull/34014

Signed-off-by: elstic <hao.wang@zilliz.com>
 2024-06-20 13:58:05 +08:00