XuanYang-cn
a45a288a25
fix: Separate L0 and Mix trigger interval ( #37190 )
...
See also: #37108
- Add MixCompactionTriggerInterval, default 60s
- Add L0CompactionTriggerInterval, default 10s
- Export Single related compaction configs
- Raise SingleCompactionDeltaLogMaxSize from 2MB to 16MB
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-11-12 10:56:37 +08:00
yihao.dai
d230b91bd1
enhance: Add PreallocatedSegmentIDs for the compaction task ( #36734 )
...
Add `PreallocatedSegmentIDs` field to the compaction task, allowing the
`ResultSegments` in the compaction task to represent the final segments
produced by the compaction.
issue: https://github.com/milvus-io/milvus/issues/36733
also related: https://github.com/milvus-io/milvus/issues/36686
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-13 17:59:21 +08:00
wayblink
00a5025949
enhance: support clustering compaction on null value ( #36372 )
...
issue: #36055
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-09-30 14:33:17 +08:00
XuanYang-cn
323400c190
enhance: Enable to write multiple segments in mix compactor ( #35705 )
...
Prevent segments to be written larger than maxSize * expansionRate
See also: #35584
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-30 11:29:01 +08:00
congqixia
c992a61a23
enhance: Separate allocator pkg in datacoord ( #35622 )
...
Related to #28861
Move allocator interface and implementation into separate package. Also
update some unittest logic.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-22 10:06:56 +08:00
wayblink
bb15ecdc13
enhance: improve ut for compaction_policy_clustering ( #35205 )
...
#34792
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-07 14:42:17 +08:00
wayblink
95462668ca
enhance: unify time in clustering compaction task to unix ( #35167 )
...
#34495
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-02 10:30:19 +08:00
cai.zhang
196a7986b3
enhance: Change the fixed value to a ratio for clustering segment size ( #35076 )
...
issue: #34495
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-01 22:04:14 +08:00
wayblink
5bbb1c201c
enhance:support l2 single compaction ( #34935 )
...
#34928
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-01 14:36:13 +08:00
cai.zhang
4c45bc412f
enhance: Add integration test for clustering compaction ( #34881 )
...
issue: #34792
---------
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-23 10:13:43 +08:00
wayblink
52fa668f5d
fix: fix collection is compacting logic ( #34855 )
...
fix : #34861
also related to: #34792
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-22 11:35:41 +08:00
yihao.dai
ca758c36cc
enhance: Pre-allocate ids for compaction ( #34187 )
...
This PR removes the dependency of compaction on the ID allocator by
pre-allocating the logID and segmentID.
issue: https://github.com/milvus-io/milvus/issues/33957
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-17 13:23:42 +08:00
XuanYang-cn
d7966f46ad
fix: Compaction task l0 state transfer wrong ( #34597 )
...
This bug caused failed L0 compaction tasks never end.
See also: #34460
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-11 21:39:36 +08:00
XuanYang-cn
d7a3697fb5
enhance: Add back compactionTaskNum metrics ( #34583 )
...
Fix L0 compaction task recover unable to set segment not isCompacting
See also: #34460
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-11 17:45:37 +08:00
XuanYang-cn
e0b39d8bf4
fix: Milvus panic when compaction disabled and dropping a collection ( #34103 )
...
See also: #31059
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-11 14:44:52 +08:00
wayblink
0c4d82c6d3
fix: fix compaction task startTime is 0 ( #34496 )
...
fix bug: #34497 #34498
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-09 10:22:13 +08:00
XuanYang-cn
467113deab
fix: Accidently exit the check loop ( #34481 )
...
See also: #34460
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-08 16:46:13 +08:00
wayblink
efdaed4ac6
fix: fix clustering compaction can't enqueue when compaction queue is full ( #34445 )
...
#30633
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-08 10:02:13 +08:00
cai.zhang
23e2a0e78b
fix: Remove unused code to fix compiling ( #34406 )
...
issue: #34407
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-04 08:48:08 +08:00
wayblink
a1232fafda
feat: Major compaction ( #33620 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-10 21:34:08 +08:00
zhenshan.cao
ac4f3997ce
enhance: Reconstructing Compaction to possess persistence capability ( #33265 )
...
issue #33586
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-06-05 10:17:50 +08:00
yihao.dai
7730b910b9
enhance: Decouple compaction from shard ( #33138 )
...
Decouple compaction from shard, remove dependencies on shards (e.g.
SyncSegments, injection).
issue: https://github.com/milvus-io/milvus/issues/32809
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-24 09:07:41 +08:00
XuanYang-cn
a1386bae7f
fix: Skip to submit l0 tasks when scheduler full ( #31270 )
...
See also: #31242
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-03-15 10:21:12 +08:00
XuanYang-cn
def72947c7
fix: Trigger l0 compaction when l0 views don't change ( #30729 )
...
Trigger l0 compaction when l0 views don't change
So that leftover l0 segments would be compacted in the end.
1. Refresh LevelZero plans in comactionPlanHandler, remove the meta
dependency
of compaction trigger v2
2. Add ForceTrigger method for CompactionView interface
3. rename mu to taskGuard
4. Add a new TriggerTypeLevelZeroViewIDLE
5. Add an idleTicker for compaction view manager
See also: #30098 , #30556
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-03-05 16:37:00 +08:00
XuanYang-cn
0b6beb7e0f
fix: Fill in info in CompactionSegmentBinlogs ( #30279 )
...
After #28873 , PartitionID and CollectionID should be filled in
CompactionSegmentBinlog so that DataNode can compose
the correct logPath. However There're some places left forgotten to fill
in the information, causing Datanode downloading `xxx/0/0/xxxx/xxxx`
binlogs during compaction
See also: #30213
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-26 10:51:02 +08:00
smellthemoon
e52ce370b6
enhance:don't store logPath in meta to reduce memory ( #28873 )
...
don't store logPath in meta to reduce memory, when service get
segmentinfo, generate logpath from logid.
#28885
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-18 22:06:31 +08:00
XuanYang-cn
2c62bcbbf6
fix: Fix some left comments ( #29027 )
...
See also: #28814
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-13 18:48:39 +08:00
XuanYang-cn
5bac7f7897
fix: Fix L0 compaction in datacoord ( #28814 )
...
See also: #27606
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-12-05 18:44:37 +08:00
XuanYang-cn
9b371067d2
feat: Add Compaction views and triggers ( #27906 )
...
- Add Compaction l0 views
- Add Compaction scheduler
- Add Compaction triggerv2
- Add Compaction view manager
See also: #27606
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-23 17:30:25 +08:00
