test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-11 09:46:26 +08:00
Code Issues Actions 16 Packages Projects Releases Wiki Activity
20,149 Commits 97 Branches 105 Tags 189 MiB
74adedf750
Commit Graph

20149 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
yihao.dai
a57c9e61fc
enhance: [cherry-pick] optimize datanode cpu usage and correct the update logic of ttchecker (#34383) 
This PR cherry-picks the following commits:
- Try to improve cpu usage by refactoring the ttchecker logic and
caching string. https://github.com/milvus-io/milvus/pull/33267
- Correct the update logic of timerecorder in the flowgraph to avoid
false failure: "some node(s) haven't received input".
https://github.com/milvus-io/milvus/pull/34339

issue: https://github.com/milvus-io/milvus/issues/33266,
https://github.com/milvus-io/milvus/issues/34337

pr: https://github.com/milvus-io/milvus/pull/33267,
https://github.com/milvus-io/milvus/pull/34339

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: Xiaofan <83447078+xiaofan-luan@users.noreply.github.com>
 2024-07-04 16:34:17 +08:00
Chun Han
5831908aa2
enhance: reconstruct scalar part's code for segment-pruner(#30376) (#34365) 
related: #30376
pr: https://github.com/milvus-io/milvus/pull/34346
1. support more complex expr
2. add more ut test for unrelated fields

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-07-04 16:30:10 +08:00
XuanYang-cn
0f1915ef24
fix: DataNode might OOM by estimating based on MemorySize (#34203) 
See also: #34136
pr: #34201

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-07-04 15:24:10 +08:00
shaoting-huang
dd4dfbcd8d
enhance: [cherry-pick] Batch pick PRs related to data codec (#34345) 
This PR cherry-picks the following commits related to data codec
- Fix data codec writer close. #33818
- Legacy code clean up. #33838

issue: #33813 #33839 

pr: #33818 #33838

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
 2024-07-04 15:08:11 +08:00
foxspy
b33243f236
enhance: Update Knowhere version (#34405) 
1. cherry-pick pr #34223 (update the parameter dataset from reference to
share_ptr)
2. update knowhere version from v2.3.5 to v2.3.6
(https://github.com/zilliztech/knowhere/releases/tag/v2.3.6)

/kind branch-feature

---------

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-07-04 14:46:11 +08:00
wayblink
b3aec4c8e1
fix:[cherry-pick] minor fixs for major compaction (#34402) 
This PR cherry-picks the following commits:

- fix: Avoid datarace in clustering compaction #34288
- fix: remove isFull check in compaction.enqueue #34338

issue: #30633 
pr: #34288 #34338

---------

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-04 14:44:10 +08:00
chyezh
a1a0a56f86
enhance: async search and retrieve in cgo (#34200) 
issue: #33132
pr: #33133
other pr: #33228, #34084, #33946

- implement future-based cgo utility
- async search and retrieve in cgo
- modify gc configuration document

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-07-04 13:02:09 +08:00
congqixia
3efb78e154
enhance: [2.4] Tag gotestsum version when install deps (#34308) (#34401) 
Cherry-pick from master
pr: #34308
Tagging gotestsum by ldflags to prevent reinstall gotestsum binary each
local run

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-04 10:30:09 +08:00
Chun Han
e12b701c03
enhance: add metrics for segment prune latnecy(#30376) (#34364) 
related: #30376
pr: https://github.com/milvus-io/milvus/pull/34094

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-07-04 10:14:09 +08:00
aoiasd
9087b6f42e
enhance: [Cherry-Pick] support mark error as user error (#33498) (#34396) 
relate: https://github.com/milvus-io/milvus/issues/33492
pr: https://github.com/milvus-io/milvus/pull/33498
---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-07-04 10:08:10 +08:00
Chun Han
014cb7b071
enhance: use configed max topk for iterator when input topk exceeds(#34292) (#34293) 
related: #34292
pr: https://github.com/milvus-io/milvus/pull/34290

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-07-04 09:58:09 +08:00
cai.zhang
bc1746f96c
enhance: [cherry-pick] Optimize clustering compaction (#34313) (#34398) 
issue: #30633

master pr: #34313

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-04 09:52:09 +08:00
jaime
3e0034bea2
enhance: cherry pick some improved PRs from the master branch (#34391) 
issue:
https://github.com/milvus-io/milvus/issues/33205,https://github.com/milvus-io/milvus/issues/33342
pr: https://github.com/milvus-io/milvus/pull/33530
pr: #33343
pr: #33206

---------

Signed-off-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: xiaofanluan <xiaofan.luan@zilliz.com>
Co-authored-by: Xiaofan <83447078+xiaofan-luan@users.noreply.github.com>
 2024-07-03 19:40:11 +08:00
aoiasd
07daa8f12b
enhance:[Cherry-pick] avoid maintain checkpoint info in sync manager (#33413) (#34285) 
relate: https://github.com/milvus-io/milvus/issues/32915
pr: https://github.com/milvus-io/milvus/pull/33413

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-07-03 19:02:09 +08:00
wayblink
ec91e556fe
enhance: [cherry-pick] Refine clustering_compaction_task retry mechanism (#34384) 
issue: #30633 
pr: #34194

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-03 18:20:17 +08:00
aoiasd
7000cec365
enhance: [Cherry-pick] Merge query stream result for reduce delete task (#32855) (#34281) 
relate: https://github.com/milvus-io/milvus/issues/32854
pr:  https://github.com/milvus-io/milvus/pull/32855

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-07-03 18:08:09 +08:00
cai.zhang
0c01ace0d2
fix: [cherry-pick] Only load or release Flushed segment in datanode meta (#34393) 
issue: #34376 ,  #34375,  #34379

master pr: #34390

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-03 17:44:11 +08:00
congqixia
945f0106f6
fix: [2.4] Use raw parameter value to perform CAS (#34373) 
Cherry-pick from master
pr: #34343
See also #34342

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-03 17:20:10 +08:00
cai.zhang
e944c308c7
enhance: [cherry-pick] Skip pick worker when task doesn't need to execute actually (#34382) 
issue: #34347 

master pr: #34348

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-03 15:54:08 +08:00
aoiasd
18668aaace
enhance:[Cherry-Pick] change access log write cache default config (#34352) 
pr: https://github.com/milvus-io/milvus/pull/34351

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-07-03 15:10:09 +08:00
Patrick Weizhi Xu
88550081c3
enhance: [skip e2e][2.4] update the version of MV (#34380) 
issue: #29892 
master PR: https://github.com/milvus-io/milvus/pull/34378

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit 92f3f4521a67fcc72074a4eb5a0bb644960293d9)
 2024-07-03 14:28:16 +08:00
SimFG
b3c5eb29ed
enhance: [2.4] the proxy metric in the query request (#34356) 
/kind improvement
- issue: #33306
- pr: #33307

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-07-03 10:00:09 +08:00
wayblink
c62bf8a0b0
fix: [Cherry-pick]Pick major compaction fixs and optimizations (#34360) 
This PR cherry-picks the following commits:

- fix: sync partitiion stats blocking balance task #33742
- fix: Fix meta prefix overlap bug #33830
- fix: Small fixs of major compaction #33929 
- fix: Fix memory buffer error & some renaming #33850
- fix: sync part stats task cannot be finished #34027 
- Add an option to enable/disable vector field clustering key #34097
- fix: fix error ignore in compactor #34169
- fix:load major compaction partial result #34052
- Use new stream segment reader in clustering compaction #34232

issue: #30633
pr: #33742 #33830 #33929 #33850 #34027 #34097 #34169 #34052 #34232

---------

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Chun Han <116052805+MrPresent-Han@users.noreply.github.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-07-03 09:53:37 +08:00
zhenshan.cao
760b3fafd5
enhance: Refine compaction (#33982) (#34363) 
This PR cherry-picks the following commits related to data compaction:
- enhance: Refine compaction.
[#33982](https://github.com/milvus-io/milvus/pull/33982)
- fix l0 compaction may miss some sealed segments.
[#33838](https://github.com/milvus-io/milvus/pull/33980)

issue : https://github.com/milvus-io/milvus/issues/32939
https://github.com/milvus-io/milvus/issues/33955

pr : https://github.com/milvus-io/milvus/pull/33982

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-07-03 09:52:49 +08:00
elstic
fb88267855
test: [cherry-pick] update test case (#34109) 
pr: https://github.com/milvus-io/milvus/pull/34108

Signed-off-by: elstic <hao.wang@zilliz.com>
 2024-07-02 22:04:08 +08:00
cai.zhang
6cb0f1ff74
fix: [cherry-pick] Sync the sealed and flushed segments to datanode (#34301) (#34318) 
issue: #33696

master pr: #34301

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-02 19:36:09 +08:00
wei liu
ad545b6fa6
enhance: refine misleading param name for bloom filter parallel factor (#34335) 
pr: #34334

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-02 19:24:09 +08:00
congqixia
6b348e4e91
enhance: [2.4] Add go-deadlock as unittest only dependency (#33063) (#34322) 
Cherry-pick from master
pr: #33063
See also #33062

This PR:

- Add lock.RWMutex & lock.Mutex alias to switch implementation based on
build flags
- When build flags has test in it, use go-deadlock to detect possible
deadlocks
- Replace all sync.RWMutex & sync.Mutex in datacoord pkg

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-02 18:48:10 +08:00
wayblink
99586066f5
feat: [cherry-pick] Major compaction (#34326) 
This PR cherry-picks the following commits:
fix: speed up segment lookup via channel name in datacoord (#33530)
needed by the next commit
  feat: Major compaction (#33620)

issue: #30633
pr: #33620

---------

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: yiwangdr <80064917+yiwangdr@users.noreply.github.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
 2024-07-02 18:29:01 +08:00
cai.zhang
f11e421839
enhance: [cherry-pick] Remove compaction plans on the datanode (#33548) (#34312) 
issue: #33546

master pr: #33548

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-02 17:39:08 +08:00
congqixia
975f3bbeab
enhance: [2.4] Refine max length exceeded error message (#34300) (#34323) 
Cherry-pick from master
pr: #34300
This PR make varchar & string array field max length exceeded error
message clearer. Also fixed a minor issue that error string format and
argument number not match.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-02 15:52:09 +08:00
congqixia
4aa8a12ce8
fix: [2.4] Check partition in current target when observing partition load status (#34282) (#34305) 
Cherry-pick from master
pr: #34282
See also #34234

`LoadPartitions` does not guarantee the current target has loading
partitions if there are some partitions already loaded before.

This PR check current target contains the partition to load when
advancing loading percentage to 100.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-02 15:48:10 +08:00
aoiasd
bc8ca593bd
enhance: [Cherry-Pick] Update access log (#34295) 
Support restful api,print with consistency level and add new write
cache.
relate: https://github.com/milvus-io/milvus/issues/31823
pr: https://github.com/milvus-io/milvus/pull/33155
https://github.com/milvus-io/milvus/pull/33503
https://github.com/milvus-io/milvus/pull/32213

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-07-02 15:30:15 +08:00
cai.zhang
7ee4ccbfb7
enhance: Don't set timeout for SyncSegments (#34291) 
issue: #32809 

master pr: #34226

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-02 14:58:08 +08:00
congqixia
8f31d2db6c
enhance: [2.4] Refine frequent log in datacoord (#33449) (#34297) 
Cherry-pick from master
pr: #33449
This PR changes:
- Frequent `ListIndexes` success log to debug level
- Aggregate collection missing log after collection dropped in
`meta.GetCollectionIndexFilesSize`

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-02 11:24:08 +08:00
congqixia
fc69b10778
enhance: [2.4] Use channel name map finding channel watcher (#34294) (#34309) 
Cherry-pick from master
pr: #34294
See also #33235

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-02 10:38:09 +08:00
congqixia
58c4d061d3
fix: [2.4] Implement singleflight for segcore ChunkCache (#34250) (#34283) 
Cherry-pick from master
pr: #34250
See also #34249

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-02 10:24:08 +08:00
yihao.dai
1b95ee7ae8
enhance: [cherry-pick] Batch pick PRs related to compaction (#34315) 
This PR cherry-picks the following commits related to compaction:

- Use a pool for CompactionExecutor.
https://github.com/milvus-io/milvus/pull/33558
- Move compaction executor to compaction pacakge.
https://github.com/milvus-io/milvus/pull/33778
- Ensure the idempotency of compaction tasks.
https://github.com/milvus-io/milvus/pull/33872
- Add comment for channel cp updater.
https://github.com/milvus-io/milvus/pull/33759

issue: https://github.com/milvus-io/milvus/issues/33182,
https://github.com/milvus-io/milvus/issues/32451

pr: https://github.com/milvus-io/milvus/pull/33558,
https://github.com/milvus-io/milvus/pull/33778,
https://github.com/milvus-io/milvus/pull/33872,
https://github.com/milvus-io/milvus/pull/33759

---------

Signed-off-by: coldWater <254244460@qq.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: coldWater <254244460@qq.com>
 2024-07-02 09:54:07 +08:00
cai.zhang
c924b0b502
enhance: [cherry-pick] Refine index code and support analyze data (#34311) 
This PR primary picks up the support analyzing functionality, including
the following commits:
- main functionality: https://github.com/milvus-io/milvus/pull/33651
- refine indexnode code: https://github.com/milvus-io/milvus/pull/33458
- related fixes:
  - https://github.com/milvus-io/milvus/pull/33832
  - https://github.com/milvus-io/milvus/pull/33161

issue: #30633 
master prs: #33651 , #33458 , #33832 , #33161

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>
Co-authored-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-07-02 09:50:39 +08:00
yihao.dai
3c5ad4999e
fix: Reduce duplicate PKs in segcore (#34267) (#34302) 
issue: issue: https://github.com/milvus-io/milvus/issues/34247

pr: https://github.com/milvus-io/milvus/pull/34267

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-01 21:08:07 +08:00
zhenshan.cao
14a11e379c
enhance: Refactor Compaction to enable persistence(#33265) (#34268) 
pr : #33265 

issue #33586

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-07-01 19:32:07 +08:00
XuanYang-cn
4ebdccd160
fix: LegacyVersionWithoutRPCWatch default value to 2.4.1 (#34185) 
See also: #31933
pr: #34184

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-07-01 18:22:08 +08:00
smellthemoon
a3fdd02586
enhance: do check when add not empty logpath(#33640) (#34212) 
pr: #33640

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-07-01 18:10:08 +08:00
smellthemoon
af442b936c
enhance:change wrong log(#33447) (#34213) 
pr: #33447

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-07-01 18:02:07 +08:00
smellthemoon
064afc9dcb
enhance: use the only MaxEtcdTxnNum(#33070) (#34215) 
pr: #33070

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-07-01 18:00:07 +08:00
wei liu
92b7eebb53
enhance: Skip update index for L0 segment (#34099) (#34280) 
pr: #34280
try to update index for l0 segment, will failed by `index not found`

This PR skip update index for l0 segment

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-01 16:32:07 +08:00
yihao.dai
3af0d91f80
fix: Fix panic due to failed to seek (#34229) (#34230) 
Converting the same msgposition's vchannel to a pchannel multiple times
would result in an invalid pchannel, leading to seek failure and panic.
This PR:
1. Make a copy of msgposition in msgdispatcher.
2. Check if channel is already a pchannel, no further channel conversion
is performed.

issue: https://github.com/milvus-io/milvus/issues/34221

pr: https://github.com/milvus-io/milvus/pull/34229

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-01 16:08:12 +08:00
wei liu
c344083f22
enhance: Optimize grow slice cost during query (#34254) 
issue: #32252
pr: #34253

This PR try to pre-allocate FieldData for Reduce operations in the Query
chain using typeutil.PrepareResultFieldData to avoid the overhead of
dynamically growing the slice during appendFieldData process.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-01 15:18:11 +08:00
xige-16
06d9d258c4
fix: [cherry-pick]Fix datacoord's properties overwriten (#34240) 
issue: https://github.com/milvus-io/milvus/issues/34144
pr: https://github.com/milvus-io/milvus/pull/34146
/kind bug

Signed-off-by: xige-16 <xige2016@gmail.com>
 2024-07-01 13:08:07 +08:00
wei liu
b18de95817
enhance: Avoid assign too much segment/channels to new querynode (#34096) (#34245) 
issue: #34095
pr: #34096

When a new query node comes online, the segment_checker,
channel_checker, and balance_checker simultaneously attempt to allocate
segments to it. If this occurs during the execution of a load task and
the distribution of the new query node hasn't been updated, the query
coordinator may mistakenly view the new query node as empty. As a
result, it assigns segments or channels to it, potentially overloading
the new query node with more segments or channels than expected.

This PR measures the workload of the executing tasks on the target query
node to prevent assigning an excessive number of segments to it.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-01 10:32:06 +08:00