test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-01 11:29:48 +08:00
Code Issues Actions 11 Packages Projects Releases Wiki Activity
20,127 Commits 94 Branches 105 Tags 184 MiB
d1e0d4ea4a
Commit Graph

20127 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
wayblink
c62bf8a0b0
fix: [Cherry-pick]Pick major compaction fixs and optimizations (#34360) 
This PR cherry-picks the following commits:

- fix: sync partitiion stats blocking balance task #33742
- fix: Fix meta prefix overlap bug #33830
- fix: Small fixs of major compaction #33929 
- fix: Fix memory buffer error & some renaming #33850
- fix: sync part stats task cannot be finished #34027 
- Add an option to enable/disable vector field clustering key #34097
- fix: fix error ignore in compactor #34169
- fix:load major compaction partial result #34052
- Use new stream segment reader in clustering compaction #34232

issue: #30633
pr: #33742 #33830 #33929 #33850 #34027 #34097 #34169 #34052 #34232

---------

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Chun Han <116052805+MrPresent-Han@users.noreply.github.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-07-03 09:53:37 +08:00
zhenshan.cao
760b3fafd5
enhance: Refine compaction (#33982) (#34363) 
This PR cherry-picks the following commits related to data compaction:
- enhance: Refine compaction.
[#33982](https://github.com/milvus-io/milvus/pull/33982)
- fix l0 compaction may miss some sealed segments.
[#33838](https://github.com/milvus-io/milvus/pull/33980)

issue : https://github.com/milvus-io/milvus/issues/32939
https://github.com/milvus-io/milvus/issues/33955

pr : https://github.com/milvus-io/milvus/pull/33982

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-07-03 09:52:49 +08:00
elstic
fb88267855
test: [cherry-pick] update test case (#34109) 
pr: https://github.com/milvus-io/milvus/pull/34108

Signed-off-by: elstic <hao.wang@zilliz.com>
 2024-07-02 22:04:08 +08:00
cai.zhang
6cb0f1ff74
fix: [cherry-pick] Sync the sealed and flushed segments to datanode (#34301) (#34318) 
issue: #33696

master pr: #34301

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-02 19:36:09 +08:00
wei liu
ad545b6fa6
enhance: refine misleading param name for bloom filter parallel factor (#34335) 
pr: #34334

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-02 19:24:09 +08:00
congqixia
6b348e4e91
enhance: [2.4] Add go-deadlock as unittest only dependency (#33063) (#34322) 
Cherry-pick from master
pr: #33063
See also #33062

This PR:

- Add lock.RWMutex & lock.Mutex alias to switch implementation based on
build flags
- When build flags has test in it, use go-deadlock to detect possible
deadlocks
- Replace all sync.RWMutex & sync.Mutex in datacoord pkg

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-02 18:48:10 +08:00
wayblink
99586066f5
feat: [cherry-pick] Major compaction (#34326) 
This PR cherry-picks the following commits:
fix: speed up segment lookup via channel name in datacoord (#33530)
needed by the next commit
  feat: Major compaction (#33620)

issue: #30633
pr: #33620

---------

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: yiwangdr <80064917+yiwangdr@users.noreply.github.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
 2024-07-02 18:29:01 +08:00
cai.zhang
f11e421839
enhance: [cherry-pick] Remove compaction plans on the datanode (#33548) (#34312) 
issue: #33546

master pr: #33548

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-02 17:39:08 +08:00
congqixia
975f3bbeab
enhance: [2.4] Refine max length exceeded error message (#34300) (#34323) 
Cherry-pick from master
pr: #34300
This PR make varchar & string array field max length exceeded error
message clearer. Also fixed a minor issue that error string format and
argument number not match.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-02 15:52:09 +08:00
congqixia
4aa8a12ce8
fix: [2.4] Check partition in current target when observing partition load status (#34282) (#34305) 
Cherry-pick from master
pr: #34282
See also #34234

`LoadPartitions` does not guarantee the current target has loading
partitions if there are some partitions already loaded before.

This PR check current target contains the partition to load when
advancing loading percentage to 100.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-02 15:48:10 +08:00
aoiasd
bc8ca593bd
enhance: [Cherry-Pick] Update access log (#34295) 
Support restful api,print with consistency level and add new write
cache.
relate: https://github.com/milvus-io/milvus/issues/31823
pr: https://github.com/milvus-io/milvus/pull/33155
https://github.com/milvus-io/milvus/pull/33503
https://github.com/milvus-io/milvus/pull/32213

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-07-02 15:30:15 +08:00
cai.zhang
7ee4ccbfb7
enhance: Don't set timeout for SyncSegments (#34291) 
issue: #32809 

master pr: #34226

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-02 14:58:08 +08:00
congqixia
8f31d2db6c
enhance: [2.4] Refine frequent log in datacoord (#33449) (#34297) 
Cherry-pick from master
pr: #33449
This PR changes:
- Frequent `ListIndexes` success log to debug level
- Aggregate collection missing log after collection dropped in
`meta.GetCollectionIndexFilesSize`

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-02 11:24:08 +08:00
congqixia
fc69b10778
enhance: [2.4] Use channel name map finding channel watcher (#34294) (#34309) 
Cherry-pick from master
pr: #34294
See also #33235

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-02 10:38:09 +08:00
congqixia
58c4d061d3
fix: [2.4] Implement singleflight for segcore ChunkCache (#34250) (#34283) 
Cherry-pick from master
pr: #34250
See also #34249

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-02 10:24:08 +08:00
yihao.dai
1b95ee7ae8
enhance: [cherry-pick] Batch pick PRs related to compaction (#34315) 
This PR cherry-picks the following commits related to compaction:

- Use a pool for CompactionExecutor.
https://github.com/milvus-io/milvus/pull/33558
- Move compaction executor to compaction pacakge.
https://github.com/milvus-io/milvus/pull/33778
- Ensure the idempotency of compaction tasks.
https://github.com/milvus-io/milvus/pull/33872
- Add comment for channel cp updater.
https://github.com/milvus-io/milvus/pull/33759

issue: https://github.com/milvus-io/milvus/issues/33182,
https://github.com/milvus-io/milvus/issues/32451

pr: https://github.com/milvus-io/milvus/pull/33558,
https://github.com/milvus-io/milvus/pull/33778,
https://github.com/milvus-io/milvus/pull/33872,
https://github.com/milvus-io/milvus/pull/33759

---------

Signed-off-by: coldWater <254244460@qq.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: coldWater <254244460@qq.com>
 2024-07-02 09:54:07 +08:00
cai.zhang
c924b0b502
enhance: [cherry-pick] Refine index code and support analyze data (#34311) 
This PR primary picks up the support analyzing functionality, including
the following commits:
- main functionality: https://github.com/milvus-io/milvus/pull/33651
- refine indexnode code: https://github.com/milvus-io/milvus/pull/33458
- related fixes:
  - https://github.com/milvus-io/milvus/pull/33832
  - https://github.com/milvus-io/milvus/pull/33161

issue: #30633 
master prs: #33651 , #33458 , #33832 , #33161

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>
Co-authored-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-07-02 09:50:39 +08:00
yihao.dai
3c5ad4999e
fix: Reduce duplicate PKs in segcore (#34267) (#34302) 
issue: issue: https://github.com/milvus-io/milvus/issues/34247

pr: https://github.com/milvus-io/milvus/pull/34267

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-01 21:08:07 +08:00
zhenshan.cao
14a11e379c
enhance: Refactor Compaction to enable persistence(#33265) (#34268) 
pr : #33265 

issue #33586

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-07-01 19:32:07 +08:00
XuanYang-cn
4ebdccd160
fix: LegacyVersionWithoutRPCWatch default value to 2.4.1 (#34185) 
See also: #31933
pr: #34184

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-07-01 18:22:08 +08:00
smellthemoon
a3fdd02586
enhance: do check when add not empty logpath(#33640) (#34212) 
pr: #33640

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-07-01 18:10:08 +08:00
smellthemoon
af442b936c
enhance:change wrong log(#33447) (#34213) 
pr: #33447

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-07-01 18:02:07 +08:00
smellthemoon
064afc9dcb
enhance: use the only MaxEtcdTxnNum(#33070) (#34215) 
pr: #33070

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-07-01 18:00:07 +08:00
wei liu
92b7eebb53
enhance: Skip update index for L0 segment (#34099) (#34280) 
pr: #34280
try to update index for l0 segment, will failed by `index not found`

This PR skip update index for l0 segment

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-01 16:32:07 +08:00
yihao.dai
3af0d91f80
fix: Fix panic due to failed to seek (#34229) (#34230) 
Converting the same msgposition's vchannel to a pchannel multiple times
would result in an invalid pchannel, leading to seek failure and panic.
This PR:
1. Make a copy of msgposition in msgdispatcher.
2. Check if channel is already a pchannel, no further channel conversion
is performed.

issue: https://github.com/milvus-io/milvus/issues/34221

pr: https://github.com/milvus-io/milvus/pull/34229

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-01 16:08:12 +08:00
wei liu
c344083f22
enhance: Optimize grow slice cost during query (#34254) 
issue: #32252
pr: #34253

This PR try to pre-allocate FieldData for Reduce operations in the Query
chain using typeutil.PrepareResultFieldData to avoid the overhead of
dynamically growing the slice during appendFieldData process.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-01 15:18:11 +08:00
xige-16
06d9d258c4
fix: [cherry-pick]Fix datacoord's properties overwriten (#34240) 
issue: https://github.com/milvus-io/milvus/issues/34144
pr: https://github.com/milvus-io/milvus/pull/34146
/kind bug

Signed-off-by: xige-16 <xige2016@gmail.com>
 2024-07-01 13:08:07 +08:00
wei liu
b18de95817
enhance: Avoid assign too much segment/channels to new querynode (#34096) (#34245) 
issue: #34095
pr: #34096

When a new query node comes online, the segment_checker,
channel_checker, and balance_checker simultaneously attempt to allocate
segments to it. If this occurs during the execution of a load task and
the distribution of the new query node hasn't been updated, the query
coordinator may mistakenly view the new query node as empty. As a
result, it assigns segments or channels to it, potentially overloading
the new query node with more segments or channels than expected.

This PR measures the workload of the executing tasks on the target query
node to prevent assigning an excessive number of segments to it.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-01 10:32:06 +08:00
jaime
b37d6fa0f9
enhance: decrease cpu overhead during filter segments on datacoord (#34231) 
issue: https://github.com/milvus-io/milvus/issues/33129
pr: #33130 
pr: #33373

---------

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-01 10:20:08 +08:00
jaime
0992f10694
enhance: improve check health (#34265) 
issue: https://github.com/milvus-io/milvus/issues/34264
pr: #33800

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-01 10:18:07 +08:00
zhagnlu
fc6bd387b8
fix: expr add skip using index while index exists (#34202) 
This PR cherry-pick part from commit:

-  enhance: add skip using array index when some situation #33947 
-  fix: [ut] regex query under unsupported index  #34087

pr: #33947, #34087

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-06-30 20:03:13 +08:00
Bingyi Sun
460815ceab
fix: fix partition loaded num metric (#33316) (#34195) 
issue: https://github.com/milvus-io/milvus/issues/32108
related pr: #33316

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-06-28 14:58:05 +08:00
cai.zhang
85aa929749
enhance: Set the timeout for SyncSegments based on the number of segments (#34228) 
issue: #32809 

master pr: #34226

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-28 14:18:06 +08:00
zhuwenxing
9cb3658d74
test: [cherry-pick]remove dup code (#34243) 
pr: https://github.com/milvus-io/milvus/pull/34242

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
 2024-06-28 10:44:05 +08:00
sammy.huang
7b4c1a0a96
enhance: [skip e2e] use disk ann when os is ubuntu, rocky or amazon (#34244) 
pr: https://github.com/milvus-io/milvus/pull/34236

Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
 2024-06-27 20:50:13 +08:00
cai.zhang
1c6e850f73
enhance: [cherry-pick] Periodically synchronize segments to datanode watcher (#33420) (#34186) 
This PR primary picks up the SyncSegments functionality, including the
following commits:
- main functionality: https://github.com/milvus-io/milvus/pull/33420
- related fixes:
  - https://github.com/milvus-io/milvus/pull/33664
  - https://github.com/milvus-io/milvus/pull/33829
  - https://github.com/milvus-io/milvus/pull/34056
  - https://github.com/milvus-io/milvus/pull/34156

issue: #32809 
master pr: #33420, #33664, #33829, #34056, #34156

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-27 11:24:05 +08:00
cai.zhang
42096a2337
enhance: [cherry-pick] Reduce the frequency of index ralted logs (#33050) (#34198) 
issue:  #33001


pr: #33050 , #33212 , #33321

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-27 11:08:05 +08:00
congqixia
88766bb714
fix: [2.4] Pass otlpSecure config when setup segcore tracing (#34193) (#34210) 
Cherry-pick from master
pr: #34193
See also #34192

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-27 10:22:06 +08:00
congqixia
f741bb7526
enhance: [2.4] Avoid merging insert data when buffering insert msgs (#34205) 
Cherry-pick from master
pr: #33526 #33817
See also #33561

This PR:
- Use zero copy when buffering insert messages
- Make `storage.InsertCodec` support serialize multiple insert data
chunk into same batch binlog files

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-27 10:14:05 +08:00
congqixia
aae94d7c40
enhance: [2.4] Unify DeleteLog parsing code (#34188) 
Cherry-pick from master
pr: #34009 
See also #33787

The parsing delete log is distributed in lots of places, which is not
recommended and hard to maintain.

This PR abstract common parsing logic into DeleteLog.Parse method to
unify implementation and make it easier to replace json parsing lib.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-27 10:12:13 +08:00
cqy123456
0bd93de627
enhance: [cherry-pick]remove CheckVecIndexWithDataTypeExist function in pkg (#34104) 
issue: https://github.com/milvus-io/milvus/issues/22837
related pr: https://github.com/milvus-io/milvus/pull/34102

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-26 21:14:26 +08:00
jaime
7f636dbdf6
enhance: refine error message for invalid limit parameter (#34179) 
pr: #34178

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-26 20:46:06 +08:00
congqixia
07a05db51b
fix: [2.4] Wrap init segcore tracing with golang timeout (#33494) (#34191) 
Cherry-pick from master
pr: #33494
See also #33483

Wrap `C.InitTrace` & `C.SetTrace` with timeout preventing otlp
initializtion hangs forever when endpoint is not set correctly

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-26 19:44:05 +08:00
wei liu
18a0efe737
enhance: Avoid search querynode return nil status in response (#34100) (#34189) 
pr: #34100

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-26 18:52:04 +08:00
sre-ci-robot
a9a0981157
[automated] Update cpu Builder image changes (#34167) 
Update cpu Builder image changes
See changes:
2ecfe1599c
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2024-06-26 17:56:05 +08:00
jaime
6423b6c718
enhance: move rocksmq from internal to pkg (#34165) 
pr:  https://github.com/milvus-io/milvus/pull/33881
issue:  https://github.com/milvus-io/milvus/issues/33956

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-26 13:36:05 +08:00
congqixia
3cf526e0cc
fix: [2.4] Deep copy ImportTask.segmentsInfo to prevent data race (#34090) (#34126) 
Cherry-pick from master
pr: #34090
See also #34089

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-26 11:50:11 +08:00
congqixia
26b2e1d43c
fix: [2.4] Make querycoord panick when rg metastore sync fail (#34106) (#34127) 
Cherry-pick from master
pr: #34106
See also #34047

When `unassignNode` sync resource group with node removed failed

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-26 10:04:03 +08:00
cqy123456
3e51256fb4
enhance: [cherry-pick]update new mmap config parmeters version (#34145) 
issue: https://github.com/milvus-io/milvus/issues/32984
related pr: https://github.com/milvus-io/milvus/pull/34143

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-25 21:24:05 +08:00
yihao.dai
b1e74dc7cb
enhance: [cherry-pick] Decouple compaction from shard (#34157) 
This PR cherry-picks the following commits:

- Implement task limit control logic in datanode.
https://github.com/milvus-io/milvus/pull/32881
- Load bf from storage instead of memory during L0 compaction.
https://github.com/milvus-io/milvus/pull/32913
- Remove dependencies on shards (e.g. SyncSegments, injection).
https://github.com/milvus-io/milvus/pull/33138
- Rename Compaction interface to CompactionV2.
https://github.com/milvus-io/milvus/pull/33858
- Remove the unused residual compaction logic.
https://github.com/milvus-io/milvus/pull/33932

issue: https://github.com/milvus-io/milvus/issues/32809

pr: https://github.com/milvus-io/milvus/pull/32881,
https://github.com/milvus-io/milvus/pull/32913,
https://github.com/milvus-io/milvus/pull/33138,
https://github.com/milvus-io/milvus/pull/33858,
https://github.com/milvus-io/milvus/pull/33932

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-25 20:22:03 +08:00