test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-11 09:46:26 +08:00
Code Issues Actions 16 Packages Projects Releases Wiki Activity
20,092 Commits 97 Branches 105 Tags 189 MiB
9bcab2ebee
Commit Graph

20092 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
cai.zhang
1c6e850f73
enhance: [cherry-pick] Periodically synchronize segments to datanode watcher (#33420) (#34186) 
This PR primary picks up the SyncSegments functionality, including the
following commits:
- main functionality: https://github.com/milvus-io/milvus/pull/33420
- related fixes:
  - https://github.com/milvus-io/milvus/pull/33664
  - https://github.com/milvus-io/milvus/pull/33829
  - https://github.com/milvus-io/milvus/pull/34056
  - https://github.com/milvus-io/milvus/pull/34156

issue: #32809 
master pr: #33420, #33664, #33829, #34056, #34156

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-27 11:24:05 +08:00
cai.zhang
42096a2337
enhance: [cherry-pick] Reduce the frequency of index ralted logs (#33050) (#34198) 
issue:  #33001


pr: #33050 , #33212 , #33321

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-27 11:08:05 +08:00
congqixia
88766bb714
fix: [2.4] Pass otlpSecure config when setup segcore tracing (#34193) (#34210) 
Cherry-pick from master
pr: #34193
See also #34192

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-27 10:22:06 +08:00
congqixia
f741bb7526
enhance: [2.4] Avoid merging insert data when buffering insert msgs (#34205) 
Cherry-pick from master
pr: #33526 #33817
See also #33561

This PR:
- Use zero copy when buffering insert messages
- Make `storage.InsertCodec` support serialize multiple insert data
chunk into same batch binlog files

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-27 10:14:05 +08:00
congqixia
aae94d7c40
enhance: [2.4] Unify DeleteLog parsing code (#34188) 
Cherry-pick from master
pr: #34009 
See also #33787

The parsing delete log is distributed in lots of places, which is not
recommended and hard to maintain.

This PR abstract common parsing logic into DeleteLog.Parse method to
unify implementation and make it easier to replace json parsing lib.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-27 10:12:13 +08:00
cqy123456
0bd93de627
enhance: [cherry-pick]remove CheckVecIndexWithDataTypeExist function in pkg (#34104) 
issue: https://github.com/milvus-io/milvus/issues/22837
related pr: https://github.com/milvus-io/milvus/pull/34102

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-26 21:14:26 +08:00
jaime
7f636dbdf6
enhance: refine error message for invalid limit parameter (#34179) 
pr: #34178

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-26 20:46:06 +08:00
congqixia
07a05db51b
fix: [2.4] Wrap init segcore tracing with golang timeout (#33494) (#34191) 
Cherry-pick from master
pr: #33494
See also #33483

Wrap `C.InitTrace` & `C.SetTrace` with timeout preventing otlp
initializtion hangs forever when endpoint is not set correctly

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-26 19:44:05 +08:00
wei liu
18a0efe737
enhance: Avoid search querynode return nil status in response (#34100) (#34189) 
pr: #34100

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-26 18:52:04 +08:00
sre-ci-robot
a9a0981157
[automated] Update cpu Builder image changes (#34167) 
Update cpu Builder image changes
See changes:
2ecfe1599c
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2024-06-26 17:56:05 +08:00
jaime
6423b6c718
enhance: move rocksmq from internal to pkg (#34165) 
pr:  https://github.com/milvus-io/milvus/pull/33881
issue:  https://github.com/milvus-io/milvus/issues/33956

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-26 13:36:05 +08:00
congqixia
3cf526e0cc
fix: [2.4] Deep copy ImportTask.segmentsInfo to prevent data race (#34090) (#34126) 
Cherry-pick from master
pr: #34090
See also #34089

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-26 11:50:11 +08:00
congqixia
26b2e1d43c
fix: [2.4] Make querycoord panick when rg metastore sync fail (#34106) (#34127) 
Cherry-pick from master
pr: #34106
See also #34047

When `unassignNode` sync resource group with node removed failed

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-26 10:04:03 +08:00
cqy123456
3e51256fb4
enhance: [cherry-pick]update new mmap config parmeters version (#34145) 
issue: https://github.com/milvus-io/milvus/issues/32984
related pr: https://github.com/milvus-io/milvus/pull/34143

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-25 21:24:05 +08:00
yihao.dai
b1e74dc7cb
enhance: [cherry-pick] Decouple compaction from shard (#34157) 
This PR cherry-picks the following commits:

- Implement task limit control logic in datanode.
https://github.com/milvus-io/milvus/pull/32881
- Load bf from storage instead of memory during L0 compaction.
https://github.com/milvus-io/milvus/pull/32913
- Remove dependencies on shards (e.g. SyncSegments, injection).
https://github.com/milvus-io/milvus/pull/33138
- Rename Compaction interface to CompactionV2.
https://github.com/milvus-io/milvus/pull/33858
- Remove the unused residual compaction logic.
https://github.com/milvus-io/milvus/pull/33932

issue: https://github.com/milvus-io/milvus/issues/32809

pr: https://github.com/milvus-io/milvus/pull/32881,
https://github.com/milvus-io/milvus/pull/32913,
https://github.com/milvus-io/milvus/pull/33138,
https://github.com/milvus-io/milvus/pull/33858,
https://github.com/milvus-io/milvus/pull/33932

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-25 20:22:03 +08:00
sammy.huang
2ecfe1599c
[automated] Update cpu Builder image changes (#34079) 
Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
 2024-06-25 19:02:03 +08:00
yihao.dai
3ae1865ffb
fix: Prevent import from generating orphaned files (#34070) (#34071) 
When import failed, mark the import segment as dropped instead of drop
it directly to prevent generating orphaned files.

issue: https://github.com/milvus-io/milvus/issues/34068

pr: https://github.com/milvus-io/milvus/pull/34070

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-25 17:00:04 +08:00
cai.zhang
65050b3117
fix: [pick] Fallback field type when it isn't in request (#33832) (#34133) 
issue: #33432
pr: #33832

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-25 14:56:04 +08:00
cqy123456
f5344abdaf
enhance: [cherry-pick]growing segment support mmap (#34110) 
issue: issue: https://github.com/milvus-io/milvus/issues/32984
related pr: https://github.com/milvus-io/milvus/pull/32633,
https://github.com/milvus-io/milvus/pull/33951,
https://github.com/milvus-io/milvus/pull/33993

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-25 14:52:07 +08:00
sammy.huang
bd8ad106b9
enhance: revert dead try on uploading codecov (#34131) 
This reverts commit 5a14f65ee4.

Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
 2024-06-25 11:51:35 +08:00
XuanYang-cn
a33b68678d
enhance: [cherry-pick] Move compactor into sub package (#34098) 
This PR consists of the following commits:

- enhance: Tidy compactor and remove dup codes (#32198)
- fix: Fix l0 compactor may cause DN from OOM (#33554)
- enhance: Add deltaRowCount in l0 compaction (#33997)
- enhance: enable stream writer in compactions (#32612)
- fix: turn on compression on stream writers (#34067)
- fix: adding blob memory size in binlog serde (#33324)

See also: #32451, #33547, #33998, #31679
pr: #32198, #33554, #33997, #32612

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
 2024-06-25 11:16:02 +08:00
Patrick Weizhi Xu
1168e8c107
enhance: [2.4] support integral type for MV and skip MV if there is only one category (#34005) 
pr: #33161
issue: #29892

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-06-24 19:12:02 +08:00
GenkenWei
059aaad3bb
fix: update ubuntu base image version (#33927) 
related to #33945
FIX CVEs of milvus base image: LOW: 32,  MEDIUM: 50, **Total FIX: 82**

![image](https://github.com/milvus-io/milvus/assets/27683687/020fbb73-9a1f-42cb-8224-a595179b3533)

![image](https://github.com/milvus-io/milvus/assets/27683687/fef403a5-c658-4ee6-baf4-ace3a0387799)

Signed-off-by: weizhenkun <zhenkun.wei@qq.com>
Co-authored-by: weizhenkun <zhenkun.wei@qq.com>
 2024-06-24 14:16:02 +08:00
yanliang567
59d910320d
test:[cherry-pick]Update tests for range search and add test for query with dup ids (#34069) 
related issue: https://github.com/milvus-io/milvus/issues/33883
pr: #34057

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
 2024-06-24 11:34:03 +08:00
Jiquan Long
22e6807e9a
feat: support inverted index for array (#33452) (#34053) 
pr: https://github.com/milvus-io/milvus/pull/33184
pr: https://github.com/milvus-io/milvus/pull/33452
pr: https://github.com/milvus-io/milvus/pull/33633
issue: https://github.com/milvus-io/milvus/issues/27704
Co-authored-by: xiaocai2333 <cai.zhang@zilliz.com>

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
 2024-06-24 10:50:03 +08:00
zhuwenxing
630a726f35
test: refine restful testcases trace (#34065) 
pr: https://github.com/milvus-io/milvus/pull/34066

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-24 10:40:03 +08:00
sre-ci-robot
f4debe5e5e
[automated] Update gpu Builder image changes (#34031) 
Update gpu Builder image changes
See changes:
b3d425f50a
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2024-06-24 10:38:13 +08:00
sre-ci-robot
63c9e6e023
[automated] Update cpu Builder image changes (#34030) 
Update cpu Builder image changes
See changes:
b3d425f50a
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2024-06-24 10:36:25 +08:00
sammy.huang
9ea4aa9bc0
enhance: get environment variable from .env (#34080) 
Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
 2024-06-23 09:34:14 +08:00
yihao.dai
2fda43e49f
fix: Do compressBinlog to fix logID 0 (#34060) (#34062) 
Do compressBinlog to ensure that reloadFromKV will fill binlogs' logID
after datacoord restarts.

issue: https://github.com/milvus-io/milvus/issues/34059

pr: https://github.com/milvus-io/milvus/pull/34060

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-22 19:42:02 +08:00
sammy.huang
83ff759195
enhance: upgrade build-env to ubuntu 22.04 and gcc12 (#33961) 
pr: #33959

Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
 2024-06-22 19:40:05 +08:00
wei liu
061a00c58f
enhance: Enable database level replica num and resource groups for loading collection (#33052) (#33981) 
pr: #33052

issue: #30040

This PR introduce two database level props:
1. database.replica.number
2. database.resource_groups

User can set those two database props by AlterDatabase API, then can
load collection without specified replica_num and resource groups. then
it will use database level load param when try to load collections.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-21 16:56:02 +08:00
zhuwenxing
89461db5f3
test: update the lib of bf16 (#34044) 
pr: https://github.com/milvus-io/milvus/pull/34043

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-21 14:20:08 +08:00
congqixia
e02a95e3c2
fix: [2.4] Return record with largest timestamp for entires with same PK (#33936) (#34024) 
Cherry-pick from master
pr: #33936
See also #33883

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-21 14:14:01 +08:00
congqixia
891a94ad9e
fix: [2.4] Check nodeID wildcard when removing pkOracle (#33895) (#34020) 
Cherry-pick from master
pr: #33895
See also #33894

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-21 12:04:00 +08:00
jaime
c219dca001
fix: metrics database_num is 0 after restarting rootcoord (#34010) 
issue: https://github.com/milvus-io/milvus/issues/34041

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-21 11:46:02 +08:00
wei liu
7d1d5a838a
fix: Fix GetReplicas API return nil status (#33715) (#34019) 
issue: #33702
pr: #33715

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-21 10:26:02 +08:00
wei liu
fbc8fb3cb2
enhance: Skip return data distribution if no change happen (#32814) (#33985) 
issue: #32813
pr: #32814

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-21 10:24:12 +08:00
elstic
5952c09925
test: [cherry-pick] optimizing variable names (#34036) 
pr: https://github.com/milvus-io/milvus/pull/34035

Signed-off-by: elstic <hao.wang@zilliz.com>
 2024-06-21 10:12:01 +08:00
XuanYang-cn
e55fee6b04
enhance: Add deltaRowCount in l0 compaction (#33843) 
See also: #33998 
pr: #33997

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-20 19:32:02 +08:00
sre-ci-robot
8f6f6dc29f
[automated] Bump milvus version to v2.4.5 (#34028) 
Bump milvus version to v2.4.5
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2024-06-20 17:38:01 +08:00
shaoting-huang
b3d425f50a
enhance: Upgrade go version from 1.20 to 1.21 (#33940) 
issue #32982

related pr in master:
pr: #33047 #33150 #33176 #33351 #33202 #33192

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
 2024-06-20 14:52:00 +08:00
congqixia
4424c9e5e7
fix: [2.4] Remove loopclosure issue in ChannelManagerImplV2 (#33989) (#34004) 
Cherry-pick from master
pr: #33989
See also #33987

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-20 14:42:00 +08:00
wei liu
a7ae45c91c
enhance: Add trace for bf cost in l0 compactor (#33898) 
pr: #33860

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-20 14:40:15 +08:00
elstic
e0e2ed073b
test: [cherry-pick] change the number of insertion vectors (#34015) 
pr: https://github.com/milvus-io/milvus/pull/34014

Signed-off-by: elstic <hao.wang@zilliz.com>
 2024-06-20 13:58:05 +08:00
congqixia
c28b69384a
fix: [2.4] Check the correct return error in MultiRemove (#33926) (#33966) 
Cherry-pick from master
pr: #33926
See also #33925

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-20 11:28:02 +08:00
Jiquan Long
032685239f
enhance: fix copying hits of inverted index twice (#33968) (#33975) 
issue: https://github.com/milvus-io/milvus/issues/29793
pr: https://github.com/milvus-io/milvus/pull/33968
The custom `VecCollector` have already transformed the results into
vector of offsets, no need to copy them twice.

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-06-20 10:06:06 +08:00
wei liu
87508c3390
enhance: Avoid to iterate whole segment list for each task's process(#33943) (#33976) 
pr: #33943

when querycoord process segment task, it will try to iterate whole
segment list to checke whether segment is loaded, which cost too much
cpu if there has thousands of segments.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-20 10:00:05 +08:00
zhuwenxing
c61740d407
test: [cherry-pick]add test case for bulkwriter (#33914) 
pr: https://github.com/milvus-io/milvus/pull/33879

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-20 09:22:18 +08:00
presburger
30b70b05d0
fix: fix milvus gpu compile error (#33990) 
NVIDIA RAFT uses customized versions of spdlog and fmt, which conflict
with the versions provided by Conan. Therefore, it is necessary to
isolate the build environment for knowhere to avoid these conflicts.
issue: #33991

Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
 2024-06-20 09:20:00 +08:00