test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-11-30 10:59:32 +08:00
Code Issues Actions 11 Packages Projects Releases Wiki Activity
19,549 Commits 94 Branches 105 Tags 184 MiB
2944971507
Commit Graph

19549 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Xiaofan
19ee8810e8
Update README.md 2024-03-25 12:35:11 -07:00
yihao.dai
31cf849f68
enhance: Support retriving file size from importutilv2.Reader (#31533) 
To reduce the overhead caused by listing the S3 objects, add an
interface to importutil.Reader to retrieve file sizes.

issue: https://github.com/milvus-io/milvus/issues/31532,
https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-03-25 20:29:07 +08:00
sre-ci-robot
6bb654f23f Update all contributors 
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
 2024-03-25 12:00:51 +00:00
cqy123456
e3d78f55b3
fix: add some check fieldata dim (#31564) 
issue:https://github.com/milvus-io/milvus/issues/30138

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-03-25 19:19:07 +08:00
wei liu
6438d65459
fix: Grow task stuck at stopping node (#31487) 
issue: #30816
this PR fix that grow task stuck at stopping node

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-03-25 18:57:07 +08:00
John
ced7629d7d
doc: remove repetitive words (#31105) 
Signed-off-by: hishope <csqiye@126.com>
 2024-03-25 17:27:09 +08:00
Bingyi Sun
0ac9bb4a9c
enhance: add mmap migration tool (#30909) 
issue: #30908

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-03-25 15:51:09 +08:00
Bingyi Sun
8e661f791a
fix: lazy load index data in cache (#31094) 
issue: https://github.com/milvus-io/milvus/issues/31571

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-03-25 15:43:07 +08:00
presburger
fe1961ff14
enhance: add comments for gpu mem pool setting (#31231) 
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
 2024-03-25 14:41:07 +08:00
Bingyi Sun
0c868b654d
fix: fix mmap enabled check in resource estimation (#31536) 
issue: https://github.com/milvus-io/milvus/issues/31569

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-03-25 14:29:07 +08:00
yihao.dai
9a13b9822f
enhance: Return more fields in import progress response (#31539) 
Return more fields in import progress response, include importedRows and
totalRows. Additionally, ensure compatibility with the old import
progress response by retaining fields of create timestamp and row count.

issue: https://github.com/milvus-io/milvus/issues/31448
https://github.com/milvus-io/milvus/issues/31237
https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-03-24 21:57:06 +08:00
Alexander Guzhva
c4b37fb285
enhance: Custom bitset and bitsetview prototypes (#30454) 
Issue: #31285 

Basically, I've replaced `FixedVector<bool>` and `boost::dynamic_bitset`
with custom bitset and bitsetview in order to reduce the memory
bandwidth & increase performance for the filtering.

This PR is for internal use only. 

Current progress (numbers are for GCC 9.5.0 on Ubuntu 22.04 LTS;
clang-17 produces better performance numbers):
Baseline:
```
[ RUN      ] CApiTest.AssembeChunkPerfTest
start test
cost: 17903us
[       OK ] CApiTest.AssembeChunkPerfTest (183 ms)

[ RUN      ] Expr.TestMultiLogicalExprsOptimization
cost: 1391us
cost: 5us
cost: 4us
cost: 4us
cost: 6us
cost: 4us
cost: 4us
cost: 4us
cost: 4us
cost: 4us
143
cost: 10us
cost: 8us
cost: 10us
cost: 8us
cost: 8us
cost: 8us
cost: 8us
cost: 8us
cost: 8us
cost: 9us
8
/home/ubuntu/zilliz/milvus4/milvus/internal/core/unittest/test_expr.cpp:1561: Failure
Expected: (cost_op) < (cost_no_op), actual: 143 vs 8
[  FAILED  ] Expr.TestMultiLogicalExprsOptimization (7 ms)
[ RUN      ] Expr.TestExprs
start test
3cost: 889us
start test
10cost: 2us
start test
20cost: 2us
start test
30cost: 2us
start test
50cost: 3us
start test
100cost: 7us
start test
200cost: 16us
[       OK ] Expr.TestExprs (9 ms)

[ RUN      ] Expr.TestUnaryBenchTest
start test type:2
 cost: 124.8us
start test type:3
 cost: 163.1us
start test type:4
 cost: 275.9us
start test type:5
 cost: 590.9us
start test type:10
 cost: 62.7us
start test type:11
 cost: 65.9us
[       OK ] Expr.TestUnaryBenchTest (1153 ms)
[ RUN      ] Expr.TestBinaryRangeBenchTest
start test type:2
 cost: 151.4us
start test type:3
 cost: 198.4us
start test type:4
 cost: 361.9us
start test type:5
 cost: 753.9us
start test type:10
 cost: 64.6us
start test type:11
 cost: 62.2us
[       OK ] Expr.TestBinaryRangeBenchTest (1151 ms)
[ RUN      ] Expr.TestLogicalUnaryBenchTest
start test type:2
 cost: 121.14us
start test type:3
 cost: 156.84us
start test type:4
 cost: 249.76us
start test type:5
 cost: 534.44us
start test type:10
 cost: 82.2us
start test type:11
 cost: 83.52us
[       OK ] Expr.TestLogicalUnaryBenchTest (1202 ms)
[ RUN      ] Expr.TestBinaryLogicalBenchTest
start test type:2
 cost: 80.64us
start test type:3
 cost: 78.22us
start test type:4
 cost: 255.76us
start test type:5
 cost: 532.04us
start test type:10
 cost: 89.26us
start test type:11
 cost: 90us
[       OK ] Expr.TestBinaryLogicalBenchTest (1198 ms)
[ RUN      ] Expr.TestBinaryArithOpEvalRangeBenchExpr
start test type:2
 cost: 401.7us
start test type:3
 cost: 420.96us
start test type:4
 cost: 418.04us
start test type:5
 cost: 470.54us
start test type:10
 cost: 250.32us
start test type:11
 cost: 850.08us
[       OK ] Expr.TestBinaryArithOpEvalRangeBenchExpr (1273 ms)
[ RUN      ] Expr.TestCompareExprBenchTest
start test type:2
 cost: 162us
start test type:3
 cost: 142us
start test type:4
 cost: 374us
start test type:5
 cost: 674us
start test type:10
 cost: 366us
start test type:11
 cost: 645us
[       OK ] Expr.TestCompareExprBenchTest (1214 ms)
[ RUN      ] Expr.TestRefactorExprs
start test
3cost: 1253us
start test
10cost: 1060us
start test
20cost: 681us
start test
30cost: 522us
start test
50cost: 511us
start test
100cost: 506us
start test
200cost: 497us
[       OK ] Expr.TestRefactorExprs (1142 ms)

```

Candidate:
```
[ RUN      ] CApiTest.AssembeChunkPerfTest
start test
cost: 6099us
[       OK ] CApiTest.AssembeChunkPerfTest (153 ms)

[ RUN      ] Expr.TestMultiLogicalExprsOptimization
cost: 42us
cost: 15us
cost: 15us
cost: 14us
cost: 15us
cost: 15us
cost: 15us
cost: 15us
cost: 15us
cost: 15us
17
cost: 41us
cost: 39us
cost: 33us
cost: 33us
cost: 33us
cost: 33us
cost: 34us
cost: 41us
cost: 34us
cost: 34us
35
[       OK ] Expr.TestMultiLogicalExprsOptimization (6 ms)
[ RUN      ] Expr.TestExprs
start test
3cost: 20us
start test
10cost: 2us
start test
20cost: 2us
start test
30cost: 2us
start test
50cost: 4us
start test
100cost: 8us
start test
200cost: 15us
[       OK ] Expr.TestExprs (8 ms)

[ RUN      ] Expr.TestUnaryBenchTest
start test type:2
 cost: 55.7us
start test type:3
 cost: 79.8us
start test type:4
 cost: 177.6us
start test type:5
 cost: 337.2us
start test type:10
 cost: 16.9us
start test type:11
 cost: 15.7us
[       OK ] Expr.TestUnaryBenchTest (1140 ms)
[ RUN      ] Expr.TestBinaryRangeBenchTest
start test type:2
 cost: 57.1us
start test type:3
 cost: 87us
start test type:4
 cost: 177.5us
start test type:5
 cost: 342.7us
start test type:10
 cost: 17.9us
start test type:11
 cost: 16.7us
[       OK ] Expr.TestBinaryRangeBenchTest (1152 ms)
[ RUN      ] Expr.TestLogicalUnaryBenchTest
start test type:2
 cost: 34.58us
start test type:3
 cost: 68.86us
start test type:4
 cost: 151.38us
start test type:5
 cost: 286.8us
start test type:10
 cost: 16.54us
start test type:11
 cost: 16.7us
[       OK ] Expr.TestLogicalUnaryBenchTest (1165 ms)
[ RUN      ] Expr.TestBinaryLogicalBenchTest
start test type:2
 cost: 20us
start test type:3
 cost: 17.1us
start test type:4
 cost: 154.12us
start test type:5
 cost: 286.1us
start test type:10
 cost: 19.6us
start test type:11
 cost: 19.24us
[       OK ] Expr.TestBinaryLogicalBenchTest (1188 ms)
[ RUN      ] Expr.TestBinaryArithOpEvalRangeBenchExpr
start test type:2
 cost: 125.7us
start test type:3
 cost: 111.34us
start test type:4
 cost: 148.02us
start test type:5
 cost: 306.7us
start test type:10
 cost: 149.3us
start test type:11
 cost: 282.94us
[       OK ] Expr.TestBinaryArithOpEvalRangeBenchExpr (1221 ms)
[ RUN      ] Expr.TestCompareExprBenchTest
start test type:2
 cost: 89us
start test type:3
 cost: 79us
start test type:4
 cost: 323us
start test type:5
 cost: 629us
start test type:10
 cost: 313us
start test type:11
 cost: 591us
[       OK ] Expr.TestCompareExprBenchTest (1228 ms)
[ RUN      ] Expr.TestRefactorExprs
start test
3cost: 874us
start test
10cost: 611us
start test
20cost: 290us
start test
30cost: 294us
start test
50cost: 272us
start test
100cost: 278us
start test
200cost: 279us
[       OK ] Expr.TestRefactorExprs (1149 ms)

```

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
 2024-03-24 21:49:07 +08:00
sre-ci-robot
42f2425822 Update all contributors 
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
 2024-03-24 00:01:15 +00:00
congqixia
4d2142d041
fix: Check latest leader exists before using it (#31500) 
See also #31495

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-03-22 18:25:07 +08:00
yihao.dai
f65a796d18
enhance: Add max file num limit and max file size limit for import (#31497) 
The max number of import files per request should not exceed 1024 by
default (configurable).
The import file size allowed for importing should not exceed 16GB by
default (configurable).

issue: https://github.com/milvus-io/milvus/issues/28521

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-03-22 18:13:06 +08:00
yihao.dai
0fe5e90e8b
enhance: Remove import v1 (#31403) 
Remove all code and logic related to import v1.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-03-22 15:29:09 +08:00
congqixia
7b38274cce
fix: Use mutex protection when accessing pks of l0 segments (#31505) 
See also #31504

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-03-22 14:11:06 +08:00
Chun Han
c3264ca3e3
feat: support segment pruner (#31003) 
related: #30376
 2024-03-22 13:57:06 +08:00
sre-ci-robot
c27db43ba7
[automated] Update cpu Builder image changes (#31477) 
Update cpu Builder image changes
See changes:
1b31824f83
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2024-03-22 11:39:10 +08:00
congqixia
4da8b6607d
enhance: Add scripts to use gotestsum to execute integration test (#31490) 
See also #31489

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-03-22 10:29:06 +08:00
congqixia
d90e01532f
enhance: Release level zero segments when channel unsub (#31486) 
Related to #27349
See also #30816

Level zero is not allowed to balance among delegators, they shall always
serve current delegator. This PR releases all level zero segments after
channel is unsubscribed and preventing level zero segment blocking
graceful stop.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-03-22 10:27:17 +08:00
congqixia
5c5f53d11b
fix: Check nodeID before update channel checkpoint (#31473) 
See also #31470

This PR adds nodeID assignment verification before updating channel
checkpoints.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-03-22 10:07:06 +08:00
zhuwenxing
be68b84088
test: adjust index timing (#31492) 
index before insert

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-03-22 09:09:11 +08:00
Ted Xu
09281a07f4
enhance: disk cache to enable size based eviction (#31469) 
See #31262

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-03-21 18:57:07 +08:00
wei liu
03eaa5d478
fix: Load segment task promote failed (#31430) 
issue: #30816

pr #31319 introduce the logic that segment checker need to load level
zero segment which only exist in current target.

This PR fix load segment task promote failed when segment only belongs
to current target

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-03-21 18:09:07 +08:00
yanliang567
a2fdebd90d
test: Add tests for more json forms support (#31474) 
related issue: #31051

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
 2024-03-21 17:53:07 +08:00
sammy.huang
1b31824f83
enhance: add openblas dependency (#31437) 
Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
 2024-03-21 15:59:12 +08:00
zhuwenxing
f2c2877609
test: update restful v2 testcases (#31404) 
* update restful v2 test cases
* add case to ci

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-03-21 15:31:09 +08:00
Ted Xu
c13c96e321
feat: adding cache scavenger (#31264) 
See #31262

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-03-21 14:33:06 +08:00
chyezh
9f9ef8ac32
enhance: transfer resource group and dbname to querynode when load (#30936) 
issue: #30931

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-03-21 11:59:12 +08:00
Patrick Weizhi Xu
982dd2834b
enhance: add materialized view search info (#30888) 
issue: #29892 

This PR
1. Pass Materialized View (MV) search information obtained from the
expression parsing planning procedure to Knowhere. It only performs when
MV is enabled and the partition key is involved in the expression. The
search information includes:
1. Touched field_id and the count of related categories in the
expression. E.g., `color == red && color == blue` yields `field_id ->
2`.
2. Whether the expression only includes AND (&&) logical operator,
default `true`.
    3. Whether the expression has NOT (!) operator, default `false`.
4. Store if turning on MV on the proxy to eliminate reading from
paramtable for every search request.
5. Renames to MV.

## Rebuttals
1. Did not write in `ExtractInfoPlanNodeVisitor` since the new scalar
framework was introduced and this part might be removed in the future.
2. Currently only interested in `==` and `in` expression, `string` data
type, anything else is a bonus.
3. Leave handling expressions like `F == A || F == A` for future works
of the optimizer.

## Detailed MV Info

![image](https://github.com/milvus-io/milvus/assets/6563846/b27c08a0-9fd3-4474-8897-30a3d6d6b36f)

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-03-21 11:19:07 +08:00
groot
c81909bfab
enhance: Support MinIO TLS connection (#31311) 
issue: https://github.com/milvus-io/milvus/issues/30709
pr: #31292

Signed-off-by: yhmo <yihua.mo@zilliz.com>
Co-authored-by: Chen Rao <chenrao317328@163.com>
 2024-03-21 11:15:20 +08:00
zhagnlu
cf5109ec17
fix: fix mmap failed when string field all value is empty (#31406) 
#31162

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-03-21 10:41:07 +08:00
cai.zhang
1f43be4a3c
enhance: Support auto index for scalar index (#31255) 
issue: #29309 
reopen pr : #29310

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-03-21 10:37:08 +08:00
yihao.dai
c725f95885
enhance: Enhance the load balancing of import tasks (#31423) 
Optimize the load balancing of import tasks.

issue: https://github.com/milvus-io/milvus/issues/31232

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-03-21 10:21:06 +08:00
zhuwenxing
d0dd962a2f
test: update bulk insert bench (#31457) 
use restful api to do bulk insert

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-03-21 10:11:08 +08:00
PowderLi
ee56ae7299
fix: [CHERRY-PICK] [restful v2] add hybrid_search (#31395) 
search issue: #30688

2.4 pr: #31344

Signed-off-by: PowderLi <min.li@zilliz.com>
 2024-03-21 09:59:08 +08:00
sammy.huang
b63c29b4fa
enhance: change config for etcd in nightly one pod (#31367) 
Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
 2024-03-21 09:17:06 +08:00
congqixia
3f7c77481d
fix: Use correct ticker for gc work defer (#31456) 
See also #31362

Fix the second defer using same ticker var.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-03-20 22:17:06 +08:00
pingliu
d5ba221b25
doc: change milvus docker image version to v2.4.0-rc.1 (#31461) 
Signed-off-by: ping.liu <ping.liu@zilliz.com>
 2024-03-20 22:15:05 +08:00
zhuwenxing
ede5269221
enhance:[skip e2e]remove some custom deploy config (#31440) 
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-03-20 20:51:26 +08:00
sammy.huang
c895093959
enhance: add some parameters in docker-compose (#31454) 
Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
 2024-03-20 20:45:11 +08:00
sre-ci-robot
d4917b7d51 Update all contributors 
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
 2024-03-20 12:01:08 +00:00
wei liu
7c7375031d
enhance: Add metrics for task latency in querycoord scheduler (#31405) 
This PR add metrics for task latency in querycoord scheduler, so if any
kind of task stuck, it's easy to figure out by metrics

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-03-20 19:29:06 +08:00
Bingyi Sun
0fe2ac4c3a
fix: fix mmap enable log (#31439) 
issue: https://github.com/milvus-io/milvus/issues/31443

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-03-20 19:17:05 +08:00
congqixia
a647b84f3e
enhance: Add AllPartitionsID const to replace InvalidPartitionID (#31438) 
"-1" as `InvalidPartitionID` previously used as All partition place
holder in delete cases. It's confusing and hard to maintain when a const
var has more than one meaning.

This PR add `AllPartitionsID` to replace these usages in delete
scenarios.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-03-20 19:01:05 +08:00
sammy.huang
88213030a0
enhance: [skip e2e]change cache path (#31445) 
Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
 2024-03-20 18:03:11 +08:00
Bingyi Sun
9dbd67879f
enhance: use mmap prefix to define all mmap related configs (#31436) 
Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-03-20 17:55:08 +08:00
aoiasd
0c153a5820
enhance: Rename update segment operator (#31121) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-03-20 17:53:14 +08:00
cai.zhang
4dca57535f
fix: Fix bug for get segment index state (#31427) 
issue: #31361

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-03-20 14:45:10 +08:00