test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-04 12:59:23 +08:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity
20,167 Commits 96 Branches 105 Tags 185 MiB
d16320705e
Commit Graph

22 Commits

Author SHA1 Message Date
congqixia
d16320705e
enhance: [2.4] Add Segment Level in milvus segment info APIs (#34763) (#35023) 
Cherry-pick from master
pr: #34763
See also #34746

This PR add segment level field in response of
`GetPersistentSegmentInfo` and `GetQuerySegmentInfo`

---------

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-29 10:11:52 +08:00
cai.zhang
9cd6dbcbc9
fix: [cherry-pick] Fix bug for block clustering compaction (#35021) 
issue: #34703 

master pr: #35019

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-26 11:33:40 +08:00
cai.zhang
74adedf750
enhance: Optimized the GC logic to ensure that memory is released in time (#34950) 
issue: #34703 

master pr: #34949

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-24 14:07:43 +08:00
wei liu
c13c48d99a
fix: Failed to unmarshal field stats's bloom filter (#34922) 
pr #34377 introduce this issue, which miss some new changes during the
cherry-pick

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-23 16:45:47 +08:00
cai.zhang
6986dfdd5b
enhance:[cherry-pick]Send flush signal when the water level reaches the high watermark (#34908) 
issue: #30633 

master pr: #34907

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-23 10:27:42 +08:00
cai.zhang
4ed62e9dbb
enhance: [cherry-pick] Add integration test for clustering compaction (#34860) 
issue: #34792 

master pr: #34881

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2024-07-22 17:49:42 +08:00
wayblink
33bbc614df
enhance: [cherry-pick] add ut for clustering_compactor (#34817) 
issue: #34792
pr: #34852

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-21 19:25:41 +08:00
cai.zhang
323dee2fbc
fix: [cherry-pick] Fix the issue of concurrent packing of the same segment (#34838) 
issue: #34703 

master pr: #34840

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-19 18:27:40 +08:00
wayblink
a26e965e6a
enhance:[cherry-pick] Add compaction task slot usage logic (#34625) 
issue: #34544
pr: #34581

---------

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-18 09:55:43 +08:00
cai.zhang
d74a5bdc1d
fix: [cherry-pick] Fix bug where binlogs already flushed with new segment during pack (#34760) 
issue: #34703

master pr: #34762

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-18 09:15:47 +08:00
cai.zhang
19d3606e0d
fix: [cherry-pick] Fix the bug that caused small segment flush frequently (#34727) 
issue: #34703 

master pr: #34725

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-17 09:47:43 +08:00
XuanYang-cn
5909a62ca6
fix: Fix accidentlly exit MixCompaction task loop (#34689) 
See also: #33431, #34460
pr: #34688

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-07-16 16:21:38 +08:00
cai.zhang
0c5aafd2d1
fix:[cherry-pick] Reset flushed row num after pack segment for clustering compaction (#34704) 
issue: #34703 
master pr: #34702

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-16 15:57:43 +08:00
SimFG
737bd7c734
enhance: [2.4] release the record in delete codec and add some log for compaction (#34506) 
/kind improvement
- pr: #34454

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-07-09 15:40:17 +08:00
XuanYang-cn
0f1915ef24
fix: DataNode might OOM by estimating based on MemorySize (#34203) 
See also: #34136
pr: #34201

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-07-04 15:24:10 +08:00
cai.zhang
bc1746f96c
enhance: [cherry-pick] Optimize clustering compaction (#34313) (#34398) 
issue: #30633

master pr: #34313

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-04 09:52:09 +08:00
wayblink
c62bf8a0b0
fix: [Cherry-pick]Pick major compaction fixs and optimizations (#34360) 
This PR cherry-picks the following commits:

- fix: sync partitiion stats blocking balance task #33742
- fix: Fix meta prefix overlap bug #33830
- fix: Small fixs of major compaction #33929 
- fix: Fix memory buffer error & some renaming #33850
- fix: sync part stats task cannot be finished #34027 
- Add an option to enable/disable vector field clustering key #34097
- fix: fix error ignore in compactor #34169
- fix:load major compaction partial result #34052
- Use new stream segment reader in clustering compaction #34232

issue: #30633
pr: #33742 #33830 #33929 #33850 #34027 #34097 #34169 #34052 #34232

---------

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: Chun Han <116052805+MrPresent-Han@users.noreply.github.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-07-03 09:53:37 +08:00
wayblink
99586066f5
feat: [cherry-pick] Major compaction (#34326) 
This PR cherry-picks the following commits:
fix: speed up segment lookup via channel name in datacoord (#33530)
needed by the next commit
  feat: Major compaction (#33620)

issue: #30633
pr: #33620

---------

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: yiwangdr <80064917+yiwangdr@users.noreply.github.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
 2024-07-02 18:29:01 +08:00
yihao.dai
1b95ee7ae8
enhance: [cherry-pick] Batch pick PRs related to compaction (#34315) 
This PR cherry-picks the following commits related to compaction:

- Use a pool for CompactionExecutor.
https://github.com/milvus-io/milvus/pull/33558
- Move compaction executor to compaction pacakge.
https://github.com/milvus-io/milvus/pull/33778
- Ensure the idempotency of compaction tasks.
https://github.com/milvus-io/milvus/pull/33872
- Add comment for channel cp updater.
https://github.com/milvus-io/milvus/pull/33759

issue: https://github.com/milvus-io/milvus/issues/33182,
https://github.com/milvus-io/milvus/issues/32451

pr: https://github.com/milvus-io/milvus/pull/33558,
https://github.com/milvus-io/milvus/pull/33778,
https://github.com/milvus-io/milvus/pull/33872,
https://github.com/milvus-io/milvus/pull/33759

---------

Signed-off-by: coldWater <254244460@qq.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: coldWater <254244460@qq.com>
 2024-07-02 09:54:07 +08:00
cai.zhang
1c6e850f73
enhance: [cherry-pick] Periodically synchronize segments to datanode watcher (#33420) (#34186) 
This PR primary picks up the SyncSegments functionality, including the
following commits:
- main functionality: https://github.com/milvus-io/milvus/pull/33420
- related fixes:
  - https://github.com/milvus-io/milvus/pull/33664
  - https://github.com/milvus-io/milvus/pull/33829
  - https://github.com/milvus-io/milvus/pull/34056
  - https://github.com/milvus-io/milvus/pull/34156

issue: #32809 
master pr: #33420, #33664, #33829, #34056, #34156

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-27 11:24:05 +08:00
yihao.dai
b1e74dc7cb
enhance: [cherry-pick] Decouple compaction from shard (#34157) 
This PR cherry-picks the following commits:

- Implement task limit control logic in datanode.
https://github.com/milvus-io/milvus/pull/32881
- Load bf from storage instead of memory during L0 compaction.
https://github.com/milvus-io/milvus/pull/32913
- Remove dependencies on shards (e.g. SyncSegments, injection).
https://github.com/milvus-io/milvus/pull/33138
- Rename Compaction interface to CompactionV2.
https://github.com/milvus-io/milvus/pull/33858
- Remove the unused residual compaction logic.
https://github.com/milvus-io/milvus/pull/33932

issue: https://github.com/milvus-io/milvus/issues/32809

pr: https://github.com/milvus-io/milvus/pull/32881,
https://github.com/milvus-io/milvus/pull/32913,
https://github.com/milvus-io/milvus/pull/33138,
https://github.com/milvus-io/milvus/pull/33858,
https://github.com/milvus-io/milvus/pull/33932

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-25 20:22:03 +08:00
XuanYang-cn
a33b68678d
enhance: [cherry-pick] Move compactor into sub package (#34098) 
This PR consists of the following commits:

- enhance: Tidy compactor and remove dup codes (#32198)
- fix: Fix l0 compactor may cause DN from OOM (#33554)
- enhance: Add deltaRowCount in l0 compaction (#33997)
- enhance: enable stream writer in compactions (#32612)
- fix: turn on compression on stream writers (#34067)
- fix: adding blob memory size in binlog serde (#33324)

See also: #32451, #33547, #33998, #31679
pr: #32198, #33554, #33997, #32612

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
 2024-06-25 11:16:02 +08:00