test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-05 05:18:52 +08:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity
19,885 Commits 96 Branches 105 Tags 186 MiB
e81ae1e5a4
Commit Graph

8792 Commits

Author SHA1 Message Date
yihao.dai
e81ae1e5a4
fix: Fix import segment size is uneven (#33605) (#33634) 
The data coordinator computed the appropriate number of import segments,
thus when importing in the data node, one can randomly select a segment.

issue: https://github.com/milvus-io/milvus/issues/33604

pr: https://github.com/milvus-io/milvus/pull/33605

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-05 18:49:52 +08:00
yihao.dai
68e5e6434b
enhance: Print more disk quota info (#33596) (#33597) 
/kind enhancement

pr: https://github.com/milvus-io/milvus/pull/33596

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-05 16:51:56 +08:00
Gao
b32c14f96b
fix: [2.4] correct get vector data size for bf16/fp16/binary vector (#33488) 
issue: #22837 
pr: #33377

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-06-05 14:31:57 +08:00
yihao.dai
e282e1408e
enhance: Abstract Execute interface for import/preimport task (#33234) (#33607) 
Abstract Execute interface for import/preimport task, simplify import
scheduler.

issue: https://github.com/milvus-io/milvus/issues/33157

pr: https://github.com/milvus-io/milvus/pull/33234

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-05 11:17:56 +08:00
XuanYang-cn
95582b0208
fix: [2.4] L0 compactor may cause DN OOM (#33564) 
See also: #33547
pr: #33554

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-05 10:51:50 +08:00
SimFG
766f47c9ca
fix: [2.4] reset the RootCoordQuotaStates metric before recording this metric (#33601) 
- issue: #33539
- pr: #33553

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-06-04 20:17:51 +08:00
jaime
0ad55c6c44
fix: fix loaded entity num is inaccurate (#33522) 
issue: #33520

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-04 20:09:54 +08:00
congqixia
44e97b7cda
enhance: [2.4] Use map PK to timestamp in buffer insert (#33566) (#33582) 
Cherry-pick from master
pr: #33566 
Related to #27675

Store pk to minimal timestamp in `inData` instead of bloom filter to
check whether some delete entry hit current insert batch

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-04 19:21:54 +08:00
yihao.dai
3a037884ac
fix: Fix exception info is missing (#33393) (#33396) 
Replace based std::exception to prevent "object slicing"

issue: https://github.com/milvus-io/milvus/issues/33392

pr: https://github.com/milvus-io/milvus/pull/33393

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-04 15:15:48 +08:00
Buqian Zheng
d6a994038c
enhance: [2.4] add sparse float vector support to restful v2 (#33555) 
issue: https://github.com/milvus-io/milvus/issues/29419
pr: #33231

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-06-03 19:31:46 +08:00
congqixia
c9a1acc31e
fix: [2.4] Remove group checker when closing qn pipeline (#33443) (#33485) 
Cherry-pick from master
pr: #33443
See also #33442

This fix shall prevent group checker keep printing "some node(s) haven't
received input" err message after collection released

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-03 19:25:51 +08:00
yihao.dai
30fd4a955a
enhance: Avoid redundant meta operations of import (#33518) (#33519) 
issue: https://github.com/milvus-io/milvus/issues/33513

pr: https://github.com/milvus-io/milvus/pull/33518

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-03 14:15:53 +08:00
yihao.dai
8ff5d2793c
fix: Fill stats log id and check validity (#33477) (#33478) 
1. Fill log ID of stats log from import
2. Add a check to validate the log ID before writing to meta

issue: https://github.com/milvus-io/milvus/issues/33476

pr: https://github.com/milvus-io/milvus/pull/33477

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-05-31 14:13:46 +08:00
Buqian Zheng
38e5e33196
enhance: [2.4] update knowhere version (#33491) 
issue: https://github.com/milvus-io/milvus/issues/33489
pr: #33490
update knowhere version to latest. remove usage of seed_ef as it be
replaced by existing ef.

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-05-31 10:17:50 +08:00
Cai Yudong
68e2d532d8
enhance: Cherry-pick following SparseFloatVector bulk insert PRs to Milvus2.4 (#33391) 
Cherry pick from master
pr: #33064 #33101 #33187 #33259 #33224
#33064 Support readable JSON file import for
Float16/BFloat16/SparseFloat
  #33101 Store SparseFloatVector into parquet as JSON string
  #33187 Fix SparseFloatVector data parse error for parquet
  #33259 Fix SparseFloatVector data parse error for json
  #33224 Optimize bulk insert unittest

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-05-30 10:31:45 +08:00
SimFG
23b0731a6a
fix: [2.4] show empty collection when has granted the all privilege (#33454) 
- issue: #33382
- pr: #33445

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-05-29 20:49:45 +08:00
Alexander Guzhva
713c9ce4c7
fix: [Cherry-pick][2.4] make milvus compilable using gcc-13 (#33441) 
issue: https://github.com/milvus-io/milvus/issues/30190
pr: https://github.com/milvus-io/milvus/pull/30149

add a missing header

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
 2024-05-29 16:19:45 +08:00
Alexander Guzhva
5a668a17a9
enhance: [Cherry-pick][2.4] Upgrade bitset for ARM SVE (#33440) 
issue: https://github.com/milvus-io/milvus/issues/32826
pr: https://github.com/milvus-io/milvus/pull/32718
improve ARM SVE performance for internal/core/src/bitset

Baseline timings for gcc 11.4 + Graviton 3 + manually enabled SVE:
https://gist.github.com/alexanderguzhva/a974b50134c8bb9255fb15f144e5ac83

Candidate timings for gcc 11.4 + Graviton 3 + manually enabled SVE:
https://gist.github.com/alexanderguzhva/19fc88f4ad3757e05e0f7feaf563b3d3

Signed-off-by: Alexandr Guzhva <alexanderguzhva@gmail.com>
 2024-05-29 16:17:51 +08:00
yihao.dai
7384bfe3f8
fix: use seperate warmup pool and disable warmup by default (#33348) (#33349) 
1. use a small warmup pool to reduce the impact of warmup
2. change the warmup pool to nonblocking mode
3. disable warmup by default
4. remove the maximum size limit of 16 for the load pool

issue: https://github.com/milvus-io/milvus/issues/32772

pr: https://github.com/milvus-io/milvus/pull/33348

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: xiaofanluan <xiaofan.luan@zilliz.com>
 2024-05-28 19:27:43 +08:00
XuanYang-cn
0c5aaeb34c
enhance: Make channel meta able to writer 200k plus segments (#33300) 
See also: #33125
pr: #33279

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-05-28 10:09:48 +08:00
SimFG
6585227c9c
fix: [2.4] not found database name in the datacoord meta object (#33412) 
- issue: #33410
- pr: #33411

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-05-28 10:07:43 +08:00
XuanYang-cn
07b995fea4
fix: [2.4]Sync dropped segment for dropped partition (#33332) 
See also: #33330
pr: #33331

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-05-27 17:57:43 +08:00
SimFG
cbeb6c169d
fix: [2.4] quotaCenter collect metrics failed because the wrong param (#33399) 
- issue: #33397
- pr: #33398

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-05-27 17:05:42 +08:00
wayblink
deebae70a7
fix:[cherry-pick]Panic if ProcessActiveStandBy returns error (#33372) 
pr:#33369
issue:#33368

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-05-27 10:13:59 +08:00
yihao.dai
b205d7e29f
fix: Fix frequent 'failed to get request info' log (#33334) (#33337) 
issue: issue: https://github.com/milvus-io/milvus/issues/33333

pr: https://github.com/milvus-io/milvus/pull/33334

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-05-27 01:37:41 +08:00
PowderLi
aaecc92244
fix: [cherry-pick] [restful v2] search result be cut while nq > 1 (#33363) 
issue: #33099 #32837 #32419
master pr: #33339

1. len(search result) may be nq * topk, we need return all rather than
topk
2. the  in restful response payload keep the same with milvus error code

Signed-off-by: PowderLi <min.li@zilliz.com>
 2024-05-27 00:57:45 +08:00
congqixia
e2626c7b9e
fix: [2.4]Allocate new slice for each batch in streaming reader (#33360) 
Cherry-pick from master
pr: #33359
Related to #33268

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-24 18:59:42 +08:00
jaime
8990b8b051
fix: correct error of metrics stats (#33305) 
issue: #32980
cherry pick from master
pr:  #33075 #33255

---------

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-05-24 09:15:41 +08:00
Bingyi Sun
d4a146ef1a
enhance: mmap load raw data if scalar index does not have raw data (#… (#33317) 
pr: #33175

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-05-24 09:09:42 +08:00
Bingyi Sun
e3f1331c47
fix: check array field data is nil before getting the type (#33114) (#33311) 
issue: https://github.com/milvus-io/milvus/issues/33074
pr: #33114

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-05-24 09:07:11 +08:00
yihao.dai
e8c9bdfa5c
fix: Fix global rate limit is not working (#33335) (#33336) 
If the request is limited by rate limiter, limiter should not "Cancel".
This is because, if limited, tokens are not deducted; instead, "Cancel"
operation would increase the token count.

issue: https://github.com/milvus-io/milvus/issues/31705

pr: https://github.com/milvus-io/milvus/pull/33335

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-05-24 09:05:47 +08:00
yihao.dai
62663ac2e2
fix: Fix printing type of request (#33289) (#33319) 
issue: https://github.com/milvus-io/milvus/issues/31705

pr: https://github.com/milvus-io/milvus/pull/33289

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-05-23 21:14:31 +08:00
cai.zhang
ef10d15658
fix: [2.4]Throw an exception after all the threads in thread pool finished (#32810) (#33314) 
issue: #32487
master pr: #32810

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-05-23 17:55:40 +08:00
Buqian Zheng
ec33024637
fix: [2.4] in segcore validate search sparse vector to make sure it is legal (#33312) 
pr: #32856

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-05-23 17:35:40 +08:00
SimFG
37b2f90c90
enhance: [2.4] the panic when db isn't existed in the rate limit interceptor (#33308) 
issue: #33243
pr: #33244

1. fix: the panic when db isn't existed in the rate limit interceptor
#33244
2. enhance: check the auth in some rest v2 api #33256

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-05-23 17:25:41 +08:00
foxspy
2b62324d7b
enhance: Update Knowhere version (#33310) 
/kind branch-feature

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
 2024-05-23 16:53:40 +08:00
Jiquan Long
d98e1f6ff5
fix: two-phase retrieval on lru-segment (#32945) (#33313) 
Cherry-pick from master
pr: #32945 
issue: #31822

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-05-23 16:25:40 +08:00
wei liu
32bfd9befa
enhance: Enable to dynamic update balancer policy in querycoord (#33037) (#33272) 
issue: #33036
pr: #33037
This PR enable to dynamic update balancer policy without restart
querycoord.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-05-23 15:43:41 +08:00
congqixia
3bd8137062
enhance: [2.4] Use pre-built logger for write buffer frequent ops (#33273) (#33304) 
Cherry-pick from master
pr: #33273
See also #33266

Each `WriteBuffer` shall have same channel/collection id attribute, so
use same logger will do and reduce logger allocation & frequent name
composition

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-23 15:19:41 +08:00
congqixia
2f3b377479
fix: [2.4] Remove task from syncmgr after task done (#33303) 
Cherry-pick from master
pr: #33302
See also #33247
Introduced in PR #32865

Remove task after task done to keep checkpoint sound and safe

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-23 14:51:39 +08:00
yihao.dai
ad4c1975bd
fix: Fix filtering by partition key fails for importing data (#33274) (#33277) 
Before executing the import, partition IDs should be reordered according
to partition names. Otherwise, the data might be hashed to the wrong
partition during import. This PR corrects this error.

issue: https://github.com/milvus-io/milvus/issues/33237

pr: https://github.com/milvus-io/milvus/pull/33274

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-05-23 11:25:40 +08:00
PowderLi
f10ffb9230
fix: [cherry-pick][restful v2]role operations need dbName (#33291) 
issue: #33220
master pr: #33283

use dbName as part of privilege entity, so
1. grant / revoke a privilege need dbName
2. we can describe the privileges of the role which belong to one
special database

Signed-off-by: PowderLi <min.li@zilliz.com>
 2024-05-23 09:51:45 +08:00
wei liu
a988e7cabc
enhance: Reduce bloom filter lock contention between insert and delete in query coord (#32643) (#33284) 
issue: #32530
pr: #32643 

cause ProcessDelete need to check whether pk exist in bloom filter, and
ProcessInsert need to update pk to bloom filter, when execute
ProcessInsert and ProcessDelete in parallel, it will cause race
condition in segment's bloom filter

This PR execute ProcessInsert and ProcessDelete in serial to avoid block
each other

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-05-22 20:53:40 +08:00
congqixia
3c31499bbc
enhance: [2.4]Assert insert data length not overflow int (#33248) (#33271) 
Cherry-pick from master
pr: #33248 
When InsertData is too large for cpp proto unmarshalling, the error
message is confusing since the length is overflowed

This PR adds assertion for insert data length.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-22 19:07:40 +08:00
XuanYang-cn
e5ca1f7c45
fix: Return error when startup Delete/AddNode fail (#33193) (#33258) 
See also: #33151, #33149
pr: #33193

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-05-22 14:49:40 +08:00
wei liu
4b8680894f
fix: Clean offline node from resource group after qc restart (#33233) 
issue: #33200 #33207
pr: #33232
pr#33104 causes the offline node will be kept in resource group after qc
recover, and offline node will be assign to new replica as rwNode, then
request send to those node will fail by NodeNotFound.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-05-22 14:07:39 +08:00
SimFG
2a38f5b598
enhance: [2.4] add config to control whether to init public role permissions (#33174) 
issue: #33164
pr: #33165

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-05-21 22:39:46 +08:00
XuanYang-cn
00b05fcc02
fix: Remove L0 compactor in completedCompactor (#33169) (#33216) 
See also: #33168
pr: #33169

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-05-21 19:07:39 +08:00
congqixia
e5db2f7e77
fix: [2.4] Check schema without vector field in proxy (#33211) (#33218) 
Cherry-pick from master
pr: #33211
Related to #33199

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-21 16:37:44 +08:00
XuanYang-cn
b2f7d7ba4e
fix: [2.4]channel manager's goroutine run order (#33121) 
See also: #33117
pr: #33118

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-05-21 14:31:39 +08:00