Commit Graph

28 Commits

Author SHA1 Message Date
zhenshan.cao
ac4f3997ce
enhance: Reconstructing Compaction to possess persistence capability (#33265)
issue #33586

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-06-05 10:17:50 +08:00
yihao.dai
7730b910b9
enhance: Decouple compaction from shard (#33138)
Decouple compaction from shard, remove dependencies on shards (e.g.
SyncSegments, injection).

issue: https://github.com/milvus-io/milvus/issues/32809

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-24 09:07:41 +08:00
XuanYang-cn
22bddde5ff
enhance: Tidy compactor and remove dup codes (#32198)
See also: #32451

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-05-23 09:53:40 +08:00
XuanYang-cn
b3bcc107bb
fix: Remove L0 compactor in completedCompactor (#33169)
See also: #33168

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-05-21 11:35:38 +08:00
XuanYang-cn
aad3ed3835
fix: [cherry-pick]Skip changing meta if nodeID not match with channel (#31672)
See also: #31648
pr: #31665, #31694

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-04-10 15:09:18 +08:00
XuanYang-cn
2867f50fcc
fix: Clear DN unkown compaction tasks (#30850)
If DC restarted,  those unkonwn compaction tasks
will never get call back in DN, so that the segments in the compaction
task will be locked, unable to sync and compaction again, blocking cp
advance and compaction executing.

See also: #30137

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-03-01 11:31:00 +08:00
XuanYang-cn
fb5e09d94d
fix: call injectDone after compaction failed (#30277)
syncMgr.Block() will lock the segment when executing compaction.

Previous implementation was unable to Unblock thoese segments when
compaction failed. If next compaction of the same segments arrives,
it'll stuck forever and block all later compation tasks.

This PR makes sure compaction executor would Unblock these segments
after a failure compaction.

Apart form that, this PR also refines some logs and clean some codes of
compaction, compactor:

1. Log segment count instead of segmentIDs to avoid logging too many
segments
2. Flush RPC returns L1 segments only, skip L0 and L2
3. CompactionType is checked in `Compaction`, no need to check again
inside compactor
4. Use ligter method to replace `getSegmentMeta`
5. Log information for L0 compaction when encounters an error

See also: #30213

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-02-01 14:25:04 +08:00
XuanYang-cn
86f48861c1
fix: Add more throughput in related metrics (#30038)
This PR also fixes bugs in l0 compactor where
l0 results would never be removed from datanode

See also: #30099

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-19 11:34:54 +08:00
XuanYang-cn
321c5c32e3
fix: Separate schedule and check results loop (#28692)
This PR:

- Separates compaction scheduler and check results loop So that slow in
check-loop doesn't influence execution.

- Cleans compaction tasks when drop a vchannel so dropped-channel's
compaction tasks won't be checked over and over again.

  - Skips meta change when meta's already changed, avoid panic
  - Remove not inuse injectDone(bool) parameter

See also: #28628, #28209

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-29 10:50:29 +08:00
XuanYang-cn
40d5c902b6
Enable getting multiple segments in plan result (#28350)
Compaction plan result contained one segment for one plan. For l0
compaction would write to multiple segments, this PR expand the segments
number in plan results and refactor some names for readibility.

- Name refactory: - CompactionStateResult -> CompactionPlanResult -
CompactionResult -> CompactionSegment

See also: #27606

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-14 15:56:19 +08:00
congqixia
3c503afe7c
Use typeutil.ConcurrentMap instead of sync.Map (#25846)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-24 10:23:01 +08:00
jaime
b75dcf90c0
Fix Flush hang after SyncSegments timeout (#24953)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-06-20 10:20:41 +08:00
Xiaofan
f57fe6d70b
Add compaction log (#24976)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-06-19 14:18:41 +08:00
jaime
c9d0c157ec
Move some modules from internal to public package (#22572)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
SimFG
4a90490a67
Fix the segment not found error (#22772)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-03-17 17:27:56 +08:00
wei liu
0ce70b4f10
fix compact task data race (#22299)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-02-21 16:10:26 +08:00
aoiasd
1894766235
Update datacoord compaction plan after datanode update plan to ensure consistency (#22143)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-02-15 16:00:33 +08:00
XuanYang-cn
e081eb287d
Refine flowgraph nodes for readability (#19864)
- delete compaction tasks from executor while dropping

See also: #19780

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-10-20 16:39:29 +08:00
Enwei Jiao
d9c7519107
refactor compaction concurency logic (#18660)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-08-23 15:50:52 +08:00
bigsheeper
13177a90af
Add prometheus metrics for DataNode (#15650)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Co-authored-by: Cai Yudong <yudong.cai@zilliz.com>
2022-02-28 19:11:55 +08:00
XuanYang-cn
8ab9d769de
Fix logs of compaction_executor (#15365)
- Remove not useful log in `stopExecutingtaskByVChannelName`
- Add planID info in `mergeFlushedSegments`

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-01-25 13:29:40 +08:00
XuanYang-cn
87394e29b6
Fix compaction ut datarace (#15267)
Fixes: #15241

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-01-18 17:49:39 +08:00
godchen
fefeaa2e1d
[skip e2e]Fix compaction executor typo (#13646)
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2021-12-17 19:20:46 +08:00
Bingyi Sun
8bbaae04ff
Decrease compaction parallel (#12707)
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2021-12-03 18:57:33 +08:00
congqixia
8f9e62fa18
Apply DropVirtualChannel and FlushManager drop mode (#12563)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-02 16:39:33 +08:00
XuanYang-cn
8a8ebed23d
Fix DN GC bug (#12258)
Fixes: #12249

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-25 09:43:15 +08:00
XuanYang-cn
c4bb73dc74
Complete drop collection procedure in DN (#11552)
drop, cancel compaction, flush all buffers and release fg

See also: #11426

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-11 20:56:49 +08:00
XuanYang-cn
3cd0584a04
Add compaction in datanode (#11353)
See also: #9530

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-08 19:49:07 +08:00