test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-04 12:59:23 +08:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity
20,028 Commits 96 Branches 105 Tags 185 MiB
b3c5eb29ed
Commit Graph

2870 Commits

Author SHA1 Message Date
wayblink
c62bf8a0b0
fix: [Cherry-pick]Pick major compaction fixs and optimizations (#34360) 
This PR cherry-picks the following commits:

- fix: sync partitiion stats blocking balance task #33742
- fix: Fix meta prefix overlap bug #33830
- fix: Small fixs of major compaction #33929 
- fix: Fix memory buffer error & some renaming #33850
- fix: sync part stats task cannot be finished #34027 
- Add an option to enable/disable vector field clustering key #34097
- fix: fix error ignore in compactor #34169
- fix:load major compaction partial result #34052
- Use new stream segment reader in clustering compaction #34232

issue: #30633
pr: #33742 #33830 #33929 #33850 #34027 #34097 #34169 #34052 #34232

---------

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Chun Han <116052805+MrPresent-Han@users.noreply.github.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-07-03 09:53:37 +08:00
elstic
fb88267855
test: [cherry-pick] update test case (#34109) 
pr: https://github.com/milvus-io/milvus/pull/34108

Signed-off-by: elstic <hao.wang@zilliz.com>
 2024-07-02 22:04:08 +08:00
wayblink
99586066f5
feat: [cherry-pick] Major compaction (#34326) 
This PR cherry-picks the following commits:
fix: speed up segment lookup via channel name in datacoord (#33530)
needed by the next commit
  feat: Major compaction (#33620)

issue: #30633
pr: #33620

---------

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: yiwangdr <80064917+yiwangdr@users.noreply.github.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
 2024-07-02 18:29:01 +08:00
zhenshan.cao
14a11e379c
enhance: Refactor Compaction to enable persistence(#33265) (#34268) 
pr : #33265 

issue #33586

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-07-01 19:32:07 +08:00
zhuwenxing
9cb3658d74
test: [cherry-pick]remove dup code (#34243) 
pr: https://github.com/milvus-io/milvus/pull/34242

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-28 10:44:05 +08:00
yihao.dai
b1e74dc7cb
enhance: [cherry-pick] Decouple compaction from shard (#34157) 
This PR cherry-picks the following commits:

- Implement task limit control logic in datanode.
https://github.com/milvus-io/milvus/pull/32881
- Load bf from storage instead of memory during L0 compaction.
https://github.com/milvus-io/milvus/pull/32913
- Remove dependencies on shards (e.g. SyncSegments, injection).
https://github.com/milvus-io/milvus/pull/33138
- Rename Compaction interface to CompactionV2.
https://github.com/milvus-io/milvus/pull/33858
- Remove the unused residual compaction logic.
https://github.com/milvus-io/milvus/pull/33932

issue: https://github.com/milvus-io/milvus/issues/32809

pr: https://github.com/milvus-io/milvus/pull/32881,
https://github.com/milvus-io/milvus/pull/32913,
https://github.com/milvus-io/milvus/pull/33138,
https://github.com/milvus-io/milvus/pull/33858,
https://github.com/milvus-io/milvus/pull/33932

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-25 20:22:03 +08:00
sammy.huang
2ecfe1599c
[automated] Update cpu Builder image changes (#34079) 
Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
 2024-06-25 19:02:03 +08:00
yanliang567
59d910320d
test:[cherry-pick]Update tests for range search and add test for query with dup ids (#34069) 
related issue: https://github.com/milvus-io/milvus/issues/33883
pr: #34057

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
 2024-06-24 11:34:03 +08:00
Jiquan Long
22e6807e9a
feat: support inverted index for array (#33452) (#34053) 
pr: https://github.com/milvus-io/milvus/pull/33184
pr: https://github.com/milvus-io/milvus/pull/33452
pr: https://github.com/milvus-io/milvus/pull/33633
issue: https://github.com/milvus-io/milvus/issues/27704
Co-authored-by: xiaocai2333 <cai.zhang@zilliz.com>

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
 2024-06-24 10:50:03 +08:00
zhuwenxing
630a726f35
test: refine restful testcases trace (#34065) 
pr: https://github.com/milvus-io/milvus/pull/34066

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-24 10:40:03 +08:00
wei liu
061a00c58f
enhance: Enable database level replica num and resource groups for loading collection (#33052) (#33981) 
pr: #33052

issue: #30040

This PR introduce two database level props:
1. database.replica.number
2. database.resource_groups

User can set those two database props by AlterDatabase API, then can
load collection without specified replica_num and resource groups. then
it will use database level load param when try to load collections.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-21 16:56:02 +08:00
zhuwenxing
89461db5f3
test: update the lib of bf16 (#34044) 
pr: https://github.com/milvus-io/milvus/pull/34043

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-21 14:20:08 +08:00
elstic
5952c09925
test: [cherry-pick] optimizing variable names (#34036) 
pr: https://github.com/milvus-io/milvus/pull/34035

Signed-off-by: elstic <hao.wang@zilliz.com>
 2024-06-21 10:12:01 +08:00
elstic
e0e2ed073b
test: [cherry-pick] change the number of insertion vectors (#34015) 
pr: https://github.com/milvus-io/milvus/pull/34014

Signed-off-by: elstic <hao.wang@zilliz.com>
 2024-06-20 13:58:05 +08:00
zhuwenxing
c61740d407
test: [cherry-pick]add test case for bulkwriter (#33914) 
pr: https://github.com/milvus-io/milvus/pull/33879

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-20 09:22:18 +08:00
nico
27905de192
test: update test cases (#33971) 
Signed-off-by: nico <cheng.yuan@zilliz.com>
 2024-06-19 16:48:01 +08:00
elstic
caa1b0f06e
test: [cherry-pick] add more sparse test cases (#33918) 
issue: https://github.com/milvus-io/milvus/issues/31483
pr: https://github.com/milvus-io/milvus/pull/33916

Signed-off-by: elstic <hao.wang@zilliz.com>
 2024-06-19 15:24:09 +08:00
congqixia
60695bdb44
enhance: [2.4] Set maxPartitionNum default value to 1024 (#33950) 
Cherry-pick from master
pr: #33949
See also #30059

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-18 18:42:00 +08:00
nico
530756fe42
test: update test cases (#33899) 
pr: #33096

Signed-off-by: nico <cheng.yuan@zilliz.com>
 2024-06-18 11:35:59 +08:00
zhuwenxing
5b847e93fc
test: [cherry-pick]update import test case to support different dim (#33709) (#33866) 
pr: #33709

add test case for https://github.com/milvus-io/milvus/issues/33681

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
(cherry picked from commit ca1f7ab019)
 2024-06-14 16:23:57 +08:00
zhuwenxing
f06be79a4d
test: [cherry-pick]refine load state check in restful test (#33862) 
pr: https://github.com/milvus-io/milvus/pull/33861

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-14 16:22:10 +08:00
Cai Yudong
ebd0af14f4
enhance: Handle Float16Vector/BFloat16Vector numpy bulk insert as same as BinaryVector (#33760) (#33788) 
pr: #33760
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-06-13 10:49:57 +08:00
yihao.dai
396f8608dd
fix: Fix multiple vector fields import (#33723) (#33724) 
1. Fix dim mismatch with multi-vector fields and JSON import
2. Enhance: do not display file ID in GetImportResponse.

issue: https://github.com/milvus-io/milvus/issues/33681,
https://github.com/milvus-io/milvus/issues/33682

pr: https://github.com/milvus-io/milvus/pull/33723

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-10 21:55:55 +08:00
yihao.dai
ed1dee9e38
enhance: Support L0 import (#33514) (#33712) 
issue: https://github.com/milvus-io/milvus/issues/33157

pr: https://github.com/milvus-io/milvus/pull/33514

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-08 11:17:52 +08:00
zhuwenxing
29efd69857
test: refine import test (#33691) 
pr: #33600 #33356
 2024-06-07 09:21:53 +08:00
zhuwenxing
b78d7edca6
test: add coo format sparse vector in restful test (#33689) 
pr: https://github.com/milvus-io/milvus/pull/33677

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-07 09:19:58 +08:00
zhuwenxing
c39d7c3f49
test: wait all tasks finished in import test (#33629) 
pr: None

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-05 15:51:52 +08:00
zhuwenxing
6344b788fd
test: [cherry-pick]add sparse vector datatype for import test (#33166) (#33436) 
pr: #33166

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-04 08:59:47 +08:00
Buqian Zheng
d6a994038c
enhance: [2.4] add sparse float vector support to restful v2 (#33555) 
issue: https://github.com/milvus-io/milvus/issues/29419
pr: #33231

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-06-03 19:31:46 +08:00
yihao.dai
8ff5d2793c
fix: Fill stats log id and check validity (#33477) (#33478) 
1. Fill log ID of stats log from import
2. Add a check to validate the log ID before writing to meta

issue: https://github.com/milvus-io/milvus/issues/33476

pr: https://github.com/milvus-io/milvus/pull/33477

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-05-31 14:13:46 +08:00
zhuwenxing
6de80a0c19
test: add different nq for restful v2 test (#33469) 
pr: https://github.com/milvus-io/milvus/pull/33456

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-05-31 13:54:07 +08:00
Cai Yudong
68e2d532d8
enhance: Cherry-pick following SparseFloatVector bulk insert PRs to Milvus2.4 (#33391) 
Cherry pick from master
pr: #33064 #33101 #33187 #33259 #33224
#33064 Support readable JSON file import for
Float16/BFloat16/SparseFloat
  #33101 Store SparseFloatVector into parquet as JSON string
  #33187 Fix SparseFloatVector data parse error for parquet
  #33259 Fix SparseFloatVector data parse error for json
  #33224 Optimize bulk insert unittest

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-05-30 10:31:45 +08:00
PowderLi
aaecc92244
fix: [cherry-pick] [restful v2] search result be cut while nq > 1 (#33363) 
issue: #33099 #32837 #32419
master pr: #33339

1. len(search result) may be nq * topk, we need return all rather than
topk
2. the  in restful response payload keep the same with milvus error code

Signed-off-by: PowderLi <min.li@zilliz.com>
 2024-05-27 00:57:45 +08:00
nico
ecdaeda6a8
enhance: update nightly groovy (#33362) 
Signed-off-by: nico <cheng.yuan@zilliz.com>
 2024-05-24 18:43:41 +08:00
zhuwenxing
512f57e65a
test: [cherry-pick]add check for partition key for import test (#33323) 
pr:https://github.com/milvus-io/milvus/pull/33253

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-05-24 10:57:40 +08:00
yanliang567
8c9afd5f07
test: [cherry pick] Add index name tests for multiple vectors (#33315) 
Related issue: https://github.com/milvus-io/milvus/issues/32653
Related pr: #33250

---------

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
 2024-05-23 17:17:41 +08:00
yihao.dai
ad4c1975bd
fix: Fix filtering by partition key fails for importing data (#33274) (#33277) 
Before executing the import, partition IDs should be reordered according
to partition names. Otherwise, the data might be hashed to the wrong
partition during import. This PR corrects this error.

issue: https://github.com/milvus-io/milvus/issues/33237

pr: https://github.com/milvus-io/milvus/pull/33274

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-05-23 11:25:40 +08:00
sammy.huang
7013d2f906
feat: support arm-based image build and pull request (#33225) 
Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
 2024-05-21 16:54:43 +08:00
sre-ci-robot
780f3137f3
[automated] Update Pytest image changes (#33091) 
Update Pytest image changes
See changes:
9536e79363
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2024-05-17 15:09:36 +08:00
binbin
9536e79363
test: update pymilvus version (#33077) 
issue: #17614 
pr: #33079

Signed-off-by: binbin lv <binbin.lv@zilliz.com>
 2024-05-16 14:51:34 +08:00
yihao.dai
6fc00e42d7
fix: Fix binlog import (#33027) 
binlog import should only import into one partition.

issue: https://github.com/milvus-io/milvus/issues/32807

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-05-14 21:45:32 +08:00
yanliang567
ba3b2a91a0
test: Remove useless common types and refine error assert in negative cases (#33023) 
Related issue: #32653 

1. Remove some meaningless common types
2. Refine error assertion in negative cases
3. Remove some dup tests

---------

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
 2024-05-14 15:03:33 +08:00
Cai Yudong
4fc7915c70
enhance: unify data generation test APIs (#32955) 
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-05-14 14:33:33 +08:00
zhuwenxing
3871009cf5
test: add timeout for GetMetrics grpc interface (#33024) 
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-05-14 10:29:32 +08:00
zhuwenxing
9e3f3d99f7
test: add timeout for import testcases (#33006) 
add timeout for import testcases

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-05-13 19:35:32 +08:00
SimFG
4031abd2fa
enhance: change default partition num to 16 when using partition key (#32950) 
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-05-13 14:19:31 +08:00
wei liu
e2332bdc17
enhance: Enable channel exclusive balance policy (#32911) 
issue: #32910  
* split replica's node list to channels when create replicas
 * balance nodes among channels when node change happens
 * implement channel level balance, let balance happens in channel level

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-05-10 17:27:31 +08:00
Cai Yudong
dc89c6f810
enhance: remove duplicated data generation APIs for bulk insert test (#32889) 
Issue: #22837

including following changes:
1. Add API CreateInsertData() and BuildArrayData() in
internal/util/testutil
2. Remove duplicated test APIs from importutilv2 unittest and bulk
insert integration test

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-05-10 15:27:31 +08:00
yanliang567
82a1e1ebe1
test: Update insert data tests and refactor some data gen functions (#32924) 
related issue: #32653

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
 2024-05-10 14:57:32 +08:00
zhuwenxing
e776c807c4
test: add more json datatype in restful test cases (#32930) 
add more json datatype in restful test cases

related to https://github.com/milvus-io/milvus/issues/31051

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-05-10 14:11:30 +08:00