test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-04 12:59:23 +08:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity
20,007 Commits 96 Branches 105 Tags 185 MiB
a3fdd02586
Commit Graph

8878 Commits

Author SHA1 Message Date
smellthemoon
a3fdd02586
enhance: do check when add not empty logpath(#33640) (#34212) 
pr: #33640

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-07-01 18:10:08 +08:00
smellthemoon
af442b936c
enhance:change wrong log(#33447) (#34213) 
pr: #33447

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-07-01 18:02:07 +08:00
smellthemoon
064afc9dcb
enhance: use the only MaxEtcdTxnNum(#33070) (#34215) 
pr: #33070

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-07-01 18:00:07 +08:00
wei liu
92b7eebb53
enhance: Skip update index for L0 segment (#34099) (#34280) 
pr: #34280
try to update index for l0 segment, will failed by `index not found`

This PR skip update index for l0 segment

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-01 16:32:07 +08:00
wei liu
c344083f22
enhance: Optimize grow slice cost during query (#34254) 
issue: #32252
pr: #34253

This PR try to pre-allocate FieldData for Reduce operations in the Query
chain using typeutil.PrepareResultFieldData to avoid the overhead of
dynamically growing the slice during appendFieldData process.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-01 15:18:11 +08:00
xige-16
06d9d258c4
fix: [cherry-pick]Fix datacoord's properties overwriten (#34240) 
issue: https://github.com/milvus-io/milvus/issues/34144
pr: https://github.com/milvus-io/milvus/pull/34146
/kind bug

Signed-off-by: xige-16 <xige2016@gmail.com>
 2024-07-01 13:08:07 +08:00
wei liu
b18de95817
enhance: Avoid assign too much segment/channels to new querynode (#34096) (#34245) 
issue: #34095
pr: #34096

When a new query node comes online, the segment_checker,
channel_checker, and balance_checker simultaneously attempt to allocate
segments to it. If this occurs during the execution of a load task and
the distribution of the new query node hasn't been updated, the query
coordinator may mistakenly view the new query node as empty. As a
result, it assigns segments or channels to it, potentially overloading
the new query node with more segments or channels than expected.

This PR measures the workload of the executing tasks on the target query
node to prevent assigning an excessive number of segments to it.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-01 10:32:06 +08:00
jaime
b37d6fa0f9
enhance: decrease cpu overhead during filter segments on datacoord (#34231) 
issue: https://github.com/milvus-io/milvus/issues/33129
pr: #33130 
pr: #33373

---------

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-01 10:20:08 +08:00
jaime
0992f10694
enhance: improve check health (#34265) 
issue: https://github.com/milvus-io/milvus/issues/34264
pr: #33800

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-01 10:18:07 +08:00
zhagnlu
fc6bd387b8
fix: expr add skip using index while index exists (#34202) 
This PR cherry-pick part from commit:

-  enhance: add skip using array index when some situation #33947 
-  fix: [ut] regex query under unsupported index  #34087

pr: #33947, #34087

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-30 20:03:13 +08:00
Bingyi Sun
460815ceab
fix: fix partition loaded num metric (#33316) (#34195) 
issue: https://github.com/milvus-io/milvus/issues/32108
related pr: #33316

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-06-28 14:58:05 +08:00
cai.zhang
85aa929749
enhance: Set the timeout for SyncSegments based on the number of segments (#34228) 
issue: #32809 

master pr: #34226

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-28 14:18:06 +08:00
cai.zhang
1c6e850f73
enhance: [cherry-pick] Periodically synchronize segments to datanode watcher (#33420) (#34186) 
This PR primary picks up the SyncSegments functionality, including the
following commits:
- main functionality: https://github.com/milvus-io/milvus/pull/33420
- related fixes:
  - https://github.com/milvus-io/milvus/pull/33664
  - https://github.com/milvus-io/milvus/pull/33829
  - https://github.com/milvus-io/milvus/pull/34056
  - https://github.com/milvus-io/milvus/pull/34156

issue: #32809 
master pr: #33420, #33664, #33829, #34056, #34156

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-27 11:24:05 +08:00
cai.zhang
42096a2337
enhance: [cherry-pick] Reduce the frequency of index ralted logs (#33050) (#34198) 
issue:  #33001


pr: #33050 , #33212 , #33321

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-27 11:08:05 +08:00
congqixia
88766bb714
fix: [2.4] Pass otlpSecure config when setup segcore tracing (#34193) (#34210) 
Cherry-pick from master
pr: #34193
See also #34192

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-27 10:22:06 +08:00
congqixia
f741bb7526
enhance: [2.4] Avoid merging insert data when buffering insert msgs (#34205) 
Cherry-pick from master
pr: #33526 #33817
See also #33561

This PR:
- Use zero copy when buffering insert messages
- Make `storage.InsertCodec` support serialize multiple insert data
chunk into same batch binlog files

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-27 10:14:05 +08:00
congqixia
aae94d7c40
enhance: [2.4] Unify DeleteLog parsing code (#34188) 
Cherry-pick from master
pr: #34009 
See also #33787

The parsing delete log is distributed in lots of places, which is not
recommended and hard to maintain.

This PR abstract common parsing logic into DeleteLog.Parse method to
unify implementation and make it easier to replace json parsing lib.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-27 10:12:13 +08:00
cqy123456
0bd93de627
enhance: [cherry-pick]remove CheckVecIndexWithDataTypeExist function in pkg (#34104) 
issue: https://github.com/milvus-io/milvus/issues/22837
related pr: https://github.com/milvus-io/milvus/pull/34102

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-26 21:14:26 +08:00
jaime
7f636dbdf6
enhance: refine error message for invalid limit parameter (#34179) 
pr: #34178

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-26 20:46:06 +08:00
congqixia
07a05db51b
fix: [2.4] Wrap init segcore tracing with golang timeout (#33494) (#34191) 
Cherry-pick from master
pr: #33494
See also #33483

Wrap `C.InitTrace` & `C.SetTrace` with timeout preventing otlp
initializtion hangs forever when endpoint is not set correctly

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-26 19:44:05 +08:00
wei liu
18a0efe737
enhance: Avoid search querynode return nil status in response (#34100) (#34189) 
pr: #34100

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-26 18:52:04 +08:00
jaime
6423b6c718
enhance: move rocksmq from internal to pkg (#34165) 
pr:  https://github.com/milvus-io/milvus/pull/33881
issue:  https://github.com/milvus-io/milvus/issues/33956

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-26 13:36:05 +08:00
congqixia
3cf526e0cc
fix: [2.4] Deep copy ImportTask.segmentsInfo to prevent data race (#34090) (#34126) 
Cherry-pick from master
pr: #34090
See also #34089

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-26 11:50:11 +08:00
congqixia
26b2e1d43c
fix: [2.4] Make querycoord panick when rg metastore sync fail (#34106) (#34127) 
Cherry-pick from master
pr: #34106
See also #34047

When `unassignNode` sync resource group with node removed failed

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-26 10:04:03 +08:00
yihao.dai
b1e74dc7cb
enhance: [cherry-pick] Decouple compaction from shard (#34157) 
This PR cherry-picks the following commits:

- Implement task limit control logic in datanode.
https://github.com/milvus-io/milvus/pull/32881
- Load bf from storage instead of memory during L0 compaction.
https://github.com/milvus-io/milvus/pull/32913
- Remove dependencies on shards (e.g. SyncSegments, injection).
https://github.com/milvus-io/milvus/pull/33138
- Rename Compaction interface to CompactionV2.
https://github.com/milvus-io/milvus/pull/33858
- Remove the unused residual compaction logic.
https://github.com/milvus-io/milvus/pull/33932

issue: https://github.com/milvus-io/milvus/issues/32809

pr: https://github.com/milvus-io/milvus/pull/32881,
https://github.com/milvus-io/milvus/pull/32913,
https://github.com/milvus-io/milvus/pull/33138,
https://github.com/milvus-io/milvus/pull/33858,
https://github.com/milvus-io/milvus/pull/33932

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-25 20:22:03 +08:00
yihao.dai
3ae1865ffb
fix: Prevent import from generating orphaned files (#34070) (#34071) 
When import failed, mark the import segment as dropped instead of drop
it directly to prevent generating orphaned files.

issue: https://github.com/milvus-io/milvus/issues/34068

pr: https://github.com/milvus-io/milvus/pull/34070

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-25 17:00:04 +08:00
cai.zhang
65050b3117
fix: [pick] Fallback field type when it isn't in request (#33832) (#34133) 
issue: #33432
pr: #33832

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-25 14:56:04 +08:00
cqy123456
f5344abdaf
enhance: [cherry-pick]growing segment support mmap (#34110) 
issue: issue: https://github.com/milvus-io/milvus/issues/32984
related pr: https://github.com/milvus-io/milvus/pull/32633,
https://github.com/milvus-io/milvus/pull/33951,
https://github.com/milvus-io/milvus/pull/33993

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-25 14:52:07 +08:00
XuanYang-cn
a33b68678d
enhance: [cherry-pick] Move compactor into sub package (#34098) 
This PR consists of the following commits:

- enhance: Tidy compactor and remove dup codes (#32198)
- fix: Fix l0 compactor may cause DN from OOM (#33554)
- enhance: Add deltaRowCount in l0 compaction (#33997)
- enhance: enable stream writer in compactions (#32612)
- fix: turn on compression on stream writers (#34067)
- fix: adding blob memory size in binlog serde (#33324)

See also: #32451, #33547, #33998, #31679
pr: #32198, #33554, #33997, #32612

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
 2024-06-25 11:16:02 +08:00
Patrick Weizhi Xu
1168e8c107
enhance: [2.4] support integral type for MV and skip MV if there is only one category (#34005) 
pr: #33161
issue: #29892

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-06-24 19:12:02 +08:00
Jiquan Long
22e6807e9a
feat: support inverted index for array (#33452) (#34053) 
pr: https://github.com/milvus-io/milvus/pull/33184
pr: https://github.com/milvus-io/milvus/pull/33452
pr: https://github.com/milvus-io/milvus/pull/33633
issue: https://github.com/milvus-io/milvus/issues/27704
Co-authored-by: xiaocai2333 <cai.zhang@zilliz.com>

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
 2024-06-24 10:50:03 +08:00
yihao.dai
2fda43e49f
fix: Do compressBinlog to fix logID 0 (#34060) (#34062) 
Do compressBinlog to ensure that reloadFromKV will fill binlogs' logID
after datacoord restarts.

issue: https://github.com/milvus-io/milvus/issues/34059

pr: https://github.com/milvus-io/milvus/pull/34060

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-22 19:42:02 +08:00
wei liu
061a00c58f
enhance: Enable database level replica num and resource groups for loading collection (#33052) (#33981) 
pr: #33052

issue: #30040

This PR introduce two database level props:
1. database.replica.number
2. database.resource_groups

User can set those two database props by AlterDatabase API, then can
load collection without specified replica_num and resource groups. then
it will use database level load param when try to load collections.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-21 16:56:02 +08:00
congqixia
e02a95e3c2
fix: [2.4] Return record with largest timestamp for entires with same PK (#33936) (#34024) 
Cherry-pick from master
pr: #33936
See also #33883

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-21 14:14:01 +08:00
congqixia
891a94ad9e
fix: [2.4] Check nodeID wildcard when removing pkOracle (#33895) (#34020) 
Cherry-pick from master
pr: #33895
See also #33894

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-21 12:04:00 +08:00
jaime
c219dca001
fix: metrics database_num is 0 after restarting rootcoord (#34010) 
issue: https://github.com/milvus-io/milvus/issues/34041

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-21 11:46:02 +08:00
wei liu
7d1d5a838a
fix: Fix GetReplicas API return nil status (#33715) (#34019) 
issue: #33702
pr: #33715

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-21 10:26:02 +08:00
wei liu
fbc8fb3cb2
enhance: Skip return data distribution if no change happen (#32814) (#33985) 
issue: #32813
pr: #32814

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-21 10:24:12 +08:00
XuanYang-cn
e55fee6b04
enhance: Add deltaRowCount in l0 compaction (#33843) 
See also: #33998 
pr: #33997

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-20 19:32:02 +08:00
congqixia
4424c9e5e7
fix: [2.4] Remove loopclosure issue in ChannelManagerImplV2 (#33989) (#34004) 
Cherry-pick from master
pr: #33989
See also #33987

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-20 14:42:00 +08:00
wei liu
a7ae45c91c
enhance: Add trace for bf cost in l0 compactor (#33898) 
pr: #33860

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-20 14:40:15 +08:00
congqixia
c28b69384a
fix: [2.4] Check the correct return error in MultiRemove (#33926) (#33966) 
Cherry-pick from master
pr: #33926
See also #33925

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-20 11:28:02 +08:00
Jiquan Long
032685239f
enhance: fix copying hits of inverted index twice (#33968) (#33975) 
issue: https://github.com/milvus-io/milvus/issues/29793
pr: https://github.com/milvus-io/milvus/pull/33968
The custom `VecCollector` have already transformed the results into
vector of offsets, no need to copy them twice.

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-06-20 10:06:06 +08:00
wei liu
87508c3390
enhance: Avoid to iterate whole segment list for each task's process(#33943) (#33976) 
pr: #33943

when querycoord process segment task, it will try to iterate whole
segment list to checke whether segment is loaded, which cost too much
cpu if there has thousands of segments.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-20 10:00:05 +08:00
presburger
30b70b05d0
fix: fix milvus gpu compile error (#33990) 
NVIDIA RAFT uses customized versions of spdlog and fmt, which conflict
with the versions provided by Conan. Therefore, it is necessary to
isolate the build environment for knowhere to avoid these conflicts.
issue: #33991

Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
 2024-06-20 09:20:00 +08:00
wei liu
2d3c5191b8
enhance: Refine name rule check error msg (#33815) (#33900) 
pr: #33815

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-19 14:26:01 +08:00
presburger
79546a6c25
fix: fix not find arrow::parquet lib (#33941) 
Upgrade the Arrow version and remove unnecessary library link
dependencies.
issue: #33942
pr: https://github.com/milvus-io/milvus/pull/33928

Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
 2024-06-18 13:09:59 +08:00
wei liu
aafa86095f
enhance: Force to reset coord connection for unavailable error (#33910) 
pr: #33908

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-18 11:15:58 +08:00
wei liu
4513569207
enhance: add restful api to trigger component stop (#32076) (#33799) 
issue: #32698
pr: #32076
This PR add two rest api for component stop and status check:
1. `/management/stop?role=querynode` can stop the specified component
2. `/management/check/ready?role=rootcoord` can check whether the target
component is serviceable

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-17 21:53:59 +08:00
Gao
08c096cf55
enhance: Use primitive type for vectorType (#33911) 
issue: #22837 
pr: #33868 

Use primitive type instead of proto enum type for queryHook to recognize

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-06-17 19:05:58 +08:00