test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-04 04:49:08 +08:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity
20,092 Commits 96 Branches 105 Tags 185 MiB
9bcab2ebee
Commit Graph

20092 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
congqixia
084140b093
enhance: [2.4] Make hasMoreResult accurate when hit number larger than limit(#33609) (#33642) 
Cherry-pick from master
pr: #33609

See also milvus-io/milvus-sdk-go#756

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-06 11:45:51 +08:00
wei liu
0c6354018b
enhance: Avoid load bf in delegator when qn worker has no more memory(#33557) (#33650) 
pr: #33557

query coord send load request to delegator, delegator load bf first,
then forward load request to qn worker. but when qn worker has no more
memory, it will return load failed immediatelly. then delegator roll
back the loaded bf. query coord wil retry the load request, and
delegator will load and roll back bf again and again.

this PR delay the loading bf step until load segment succeed in worker.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-06 10:41:52 +08:00
wei liu
f2917f5bdf
enhance: Remove StringPrimaryKey to reduce unnecessary copy and function call cost (#33486) (#33649) 
issue: #33497
pr: #33486

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-06 10:40:01 +08:00
foxspy
58a7111599
enhance: [cherry-pick] add autoindex mapping for binary/sparse datatype (#33625) 
issue: #22837 
pr: #33624

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
 2024-06-06 10:33:52 +08:00
cqy123456
443197bdfc
enhance: [cherry-pick]disk index support binary_vector (#33575) 
issue:https://github.com/milvus-io/milvus/issues/22837
related https://github.com/milvus-io/milvus/pull/33631

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-05 19:37:57 +08:00
zhagnlu
f94a72252c
enhance: optimize some cache to reduce memory usage (#33560) 
pr: https://github.com/milvus-io/milvus/pull/33534
pr: #33599

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-05 19:21:52 +08:00
Chun Han
627b787aed
fix: query iterator lack results(#33137) (#33422) (#33506) 
related: #33137 
pr: https://github.com/milvus-io/milvus/pull/33422

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-06-05 18:51:52 +08:00
yihao.dai
e81ae1e5a4
fix: Fix import segment size is uneven (#33605) (#33634) 
The data coordinator computed the appropriate number of import segments,
thus when importing in the data node, one can randomly select a segment.

issue: https://github.com/milvus-io/milvus/issues/33604

pr: https://github.com/milvus-io/milvus/pull/33605

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-05 18:49:52 +08:00
yihao.dai
68e5e6434b
enhance: Print more disk quota info (#33596) (#33597) 
/kind enhancement

pr: https://github.com/milvus-io/milvus/pull/33596

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-05 16:51:56 +08:00
zhuwenxing
c39d7c3f49
test: wait all tasks finished in import test (#33629) 
pr: None

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-05 15:51:52 +08:00
Gao
b32c14f96b
fix: [2.4] correct get vector data size for bf16/fp16/binary vector (#33488) 
issue: #22837 
pr: #33377

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-06-05 14:31:57 +08:00
yihao.dai
e282e1408e
enhance: Abstract Execute interface for import/preimport task (#33234) (#33607) 
Abstract Execute interface for import/preimport task, simplify import
scheduler.

issue: https://github.com/milvus-io/milvus/issues/33157

pr: https://github.com/milvus-io/milvus/pull/33234

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-05 11:17:56 +08:00
XuanYang-cn
95582b0208
fix: [2.4] L0 compactor may cause DN OOM (#33564) 
See also: #33547
pr: #33554

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-05 10:51:50 +08:00
aoiasd
aaf6c85095
fix: paramtable cache cause dynamic config non-dynamic (#33473) (#33590) 
relate: https://github.com/milvus-io/milvus/issues/33461
pr: https://github.com/milvus-io/milvus/pull/33473

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-06-05 10:15:51 +08:00
SimFG
766f47c9ca
fix: [2.4] reset the RootCoordQuotaStates metric before recording this metric (#33601) 
- issue: #33539
- pr: #33553

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-06-04 20:17:51 +08:00
jaime
0ad55c6c44
fix: fix loaded entity num is inaccurate (#33522) 
issue: #33520

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-04 20:09:54 +08:00
congqixia
44e97b7cda
enhance: [2.4] Use map PK to timestamp in buffer insert (#33566) (#33582) 
Cherry-pick from master
pr: #33566 
Related to #27675

Store pk to minimal timestamp in `inData` instead of bloom filter to
check whether some delete entry hit current insert batch

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-04 19:21:54 +08:00
yihao.dai
3a037884ac
fix: Fix exception info is missing (#33393) (#33396) 
Replace based std::exception to prevent "object slicing"

issue: https://github.com/milvus-io/milvus/issues/33392

pr: https://github.com/milvus-io/milvus/pull/33393

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-04 15:15:48 +08:00
zhuwenxing
6344b788fd
test: [cherry-pick]add sparse vector datatype for import test (#33166) (#33436) 
pr: #33166

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-04 08:59:47 +08:00
Buqian Zheng
d6a994038c
enhance: [2.4] add sparse float vector support to restful v2 (#33555) 
issue: https://github.com/milvus-io/milvus/issues/29419
pr: #33231

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-06-03 19:31:46 +08:00
congqixia
c9a1acc31e
fix: [2.4] Remove group checker when closing qn pipeline (#33443) (#33485) 
Cherry-pick from master
pr: #33443
See also #33442

This fix shall prevent group checker keep printing "some node(s) haven't
received input" err message after collection released

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-03 19:25:51 +08:00
yihao.dai
30fd4a955a
enhance: Avoid redundant meta operations of import (#33518) (#33519) 
issue: https://github.com/milvus-io/milvus/issues/33513

pr: https://github.com/milvus-io/milvus/pull/33518

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-03 14:15:53 +08:00
congqixia
b9b76ee9a4
fix: [2.4]Use localStorage path to check disk cap for indexnode (#33450) (#33505) 
Cherry-pick from master
pr: #33450
See also #30943 #30944

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-03 10:21:47 +08:00
sre-ci-robot
e68f0bf787
[automated] Bump milvus version to v2.4.4 (#33525) 
Bump milvus version to v2.4.4
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2024-05-31 18:55:50 +08:00
congqixia
8e7f36d928
enhance: Bump milvus version v2.4.4 (#33504) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-31 14:55:44 +08:00
yihao.dai
8ff5d2793c
fix: Fill stats log id and check validity (#33477) (#33478) 
1. Fill log ID of stats log from import
2. Add a check to validate the log ID before writing to meta

issue: https://github.com/milvus-io/milvus/issues/33476

pr: https://github.com/milvus-io/milvus/pull/33477

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-05-31 14:13:46 +08:00
zhuwenxing
6de80a0c19
test: add different nq for restful v2 test (#33469) 
pr: https://github.com/milvus-io/milvus/pull/33456

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-05-31 13:54:07 +08:00
Buqian Zheng
38e5e33196
enhance: [2.4] update knowhere version (#33491) 
issue: https://github.com/milvus-io/milvus/issues/33489
pr: #33490
update knowhere version to latest. remove usage of seed_ef as it be
replaced by existing ef.

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-05-31 10:17:50 +08:00
Cai Yudong
68e2d532d8
enhance: Cherry-pick following SparseFloatVector bulk insert PRs to Milvus2.4 (#33391) 
Cherry pick from master
pr: #33064 #33101 #33187 #33259 #33224
#33064 Support readable JSON file import for
Float16/BFloat16/SparseFloat
  #33101 Store SparseFloatVector into parquet as JSON string
  #33187 Fix SparseFloatVector data parse error for parquet
  #33259 Fix SparseFloatVector data parse error for json
  #33224 Optimize bulk insert unittest

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-05-30 10:31:45 +08:00
SimFG
23b0731a6a
fix: [2.4] show empty collection when has granted the all privilege (#33454) 
- issue: #33382
- pr: #33445

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-05-29 20:49:45 +08:00
Alexander Guzhva
713c9ce4c7
fix: [Cherry-pick][2.4] make milvus compilable using gcc-13 (#33441) 
issue: https://github.com/milvus-io/milvus/issues/30190
pr: https://github.com/milvus-io/milvus/pull/30149

add a missing header

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
 2024-05-29 16:19:45 +08:00
Alexander Guzhva
5a668a17a9
enhance: [Cherry-pick][2.4] Upgrade bitset for ARM SVE (#33440) 
issue: https://github.com/milvus-io/milvus/issues/32826
pr: https://github.com/milvus-io/milvus/pull/32718
improve ARM SVE performance for internal/core/src/bitset

Baseline timings for gcc 11.4 + Graviton 3 + manually enabled SVE:
https://gist.github.com/alexanderguzhva/a974b50134c8bb9255fb15f144e5ac83

Candidate timings for gcc 11.4 + Graviton 3 + manually enabled SVE:
https://gist.github.com/alexanderguzhva/19fc88f4ad3757e05e0f7feaf563b3d3

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
 2024-05-29 16:17:51 +08:00
Alexander Guzhva
2638735d05
fix: [Cherry-pick][2.4] Download and install cmake for the current platform, not x86_64 only (#33439) 
issue https://github.com/milvus-io/milvus/issues/32476
pr: https://github.com/milvus-io/milvus/pull/32548
tested on x86_64 and aarch64. I'm not sure what needs to be done on some
exotic architectures.

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
 2024-05-29 16:16:00 +08:00
sre-ci-robot
8006b32b1e
[automated] Bump milvus version to v2.4.3 (#33448) 
Bump milvus version to v2.4.3
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2024-05-29 14:15:44 +08:00
yihao.dai
7384bfe3f8
fix: use seperate warmup pool and disable warmup by default (#33348) (#33349) 
1. use a small warmup pool to reduce the impact of warmup
2. change the warmup pool to nonblocking mode
3. disable warmup by default
4. remove the maximum size limit of 16 for the load pool

issue: https://github.com/milvus-io/milvus/issues/32772

pr: https://github.com/milvus-io/milvus/pull/33348

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: xiaofanluan <xiaofan.luan@zilliz.com>
 2024-05-28 19:27:43 +08:00
Bingyi Sun
9c0076cc3b
fix: fix wrong default local storage path (#33425) 
issue: https://github.com/milvus-io/milvus/issues/33427
pr: https://github.com/milvus-io/milvus/pull/33389

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-05-28 17:05:44 +08:00
congqixia
ef9c1191ae
enhance: Bump milvus version to v2.4.3 (#33408) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-28 10:11:43 +08:00
XuanYang-cn
0c5aaeb34c
enhance: Make channel meta able to writer 200k plus segments (#33300) 
See also: #33125
pr: #33279

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-05-28 10:09:48 +08:00
SimFG
6585227c9c
fix: [2.4] not found database name in the datacoord meta object (#33412) 
- issue: #33410
- pr: #33411

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-05-28 10:07:43 +08:00
XuanYang-cn
07b995fea4
fix: [2.4]Sync dropped segment for dropped partition (#33332) 
See also: #33330
pr: #33331

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-05-27 17:57:43 +08:00
SimFG
cbeb6c169d
fix: [2.4] quotaCenter collect metrics failed because the wrong param (#33399) 
- issue: #33397
- pr: #33398

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-05-27 17:05:42 +08:00
wayblink
deebae70a7
fix:[cherry-pick]Panic if ProcessActiveStandBy returns error (#33372) 
pr:#33369
issue:#33368

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-05-27 10:13:59 +08:00
yihao.dai
b205d7e29f
fix: Fix frequent 'failed to get request info' log (#33334) (#33337) 
issue: issue: https://github.com/milvus-io/milvus/issues/33333

pr: https://github.com/milvus-io/milvus/pull/33334

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-05-27 01:37:41 +08:00
PowderLi
aaecc92244
fix: [cherry-pick] [restful v2] search result be cut while nq > 1 (#33363) 
issue: #33099 #32837 #32419
master pr: #33339

1. len(search result) may be nq * topk, we need return all rather than
topk
2. the  in restful response payload keep the same with milvus error code

Signed-off-by: PowderLi <min.li@zilliz.com>
 2024-05-27 00:57:45 +08:00
congqixia
e2626c7b9e
fix: [2.4]Allocate new slice for each batch in streaming reader (#33360) 
Cherry-pick from master
pr: #33359
Related to #33268

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-24 18:59:42 +08:00
nico
ecdaeda6a8
enhance: update nightly groovy (#33362) 
Signed-off-by: nico <cheng.yuan@zilliz.com>
 2024-05-24 18:43:41 +08:00
zhuwenxing
512f57e65a
test: [cherry-pick]add check for partition key for import test (#33323) 
pr:https://github.com/milvus-io/milvus/pull/33253

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-05-24 10:57:40 +08:00
wei liu
36172492f7
enhance: Decrease bloom filter fp rate to reduce delete impact (#33301) (#33329) 
pr: #33301
when milvus process delete record, it need to find record's corresponded
segment by bloom filter, and higher bloom filter fp rate will cause
delete record forwards to wrong segments.

This PR Decrease bloom filter's default fp to 0.001.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-05-24 10:27:41 +08:00
jaime
8990b8b051
fix: correct error of metrics stats (#33305) 
issue: #32980
cherry pick from master
pr:  #33075 #33255

---------

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-05-24 09:15:41 +08:00
Bingyi Sun
d4a146ef1a
enhance: mmap load raw data if scalar index does not have raw data (#… (#33317) 
pr: #33175

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-05-24 09:09:42 +08:00