cai.zhang
2c9bb4dfa3
feat: Support stats task to sort segment by PK ( #35054 )
...
issue: #33744
This PR includes the following changes:
1. Added a new task type to the task scheduler in datacoord: stats task,
which sorts segments by primary key.
2. Implemented segment sorting in indexnode.
3. Added a new field `FieldStatsLog` to SegmentInfo to store token index
information.
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-02 14:19:03 +08:00
yihao.dai
1413ffe9b1
enhance: Rename preAllocatedSegments ( #35871 )
...
Rename `preAllocatedSegments` to `preAllocatedSegmentIDs` to avoid
confusion.
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-01 17:09:01 +08:00
XuanYang-cn
323400c190
enhance: Enable to write multiple segments in mix compactor ( #35705 )
...
Prevent segments to be written larger than maxSize * expansionRate
See also: #35584
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-30 11:29:01 +08:00
cai.zhang
aaab827a16
fix: Fix the issue of missing stats log after clustering compaction ( #35266 )
...
issue: #35265
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-08 14:24:17 +08:00
yihao.dai
a4439cc911
enhance: Implement flusher in streamingNode ( #34942 )
...
- Implement flusher to:
- Manage the pipelines (creation, deletion, etc.)
- Manage the segment write buffer
- Manage sync operation (including receive flushMsg and execute flush)
- Add a new `GetChannelRecoveryInfo` RPC in DataCoord.
- Reorganize packages: `flushcommon` and `datanode`.
issue: https://github.com/milvus-io/milvus/issues/33285
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-02 18:30:23 +08:00
wayblink
c79d1af390
enhance: Add compaction task slot usage logic ( #34581 )
...
#34544
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-18 10:27:41 +08:00
yihao.dai
ca758c36cc
enhance: Pre-allocate ids for compaction ( #34187 )
...
This PR removes the dependency of compaction on the ID allocator by
pre-allocating the logID and segmentID.
issue: https://github.com/milvus-io/milvus/issues/33957
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-17 13:23:42 +08:00
SimFG
5016038781
enhance: release the record in delete codec and add some log for compaction ( #34454 )
...
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-09 15:40:17 +08:00
wayblink
fbe3231b1f
fix: fix error ignore in compactor ( #34169 )
...
#34170
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-26 10:24:03 +08:00
wayblink
380d3f4469
fix: Fix memory buffer error & some renaming ( #33850 )
...
#30633
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-21 17:30:01 +08:00
wayblink
a1232fafda
feat: Major compaction ( #33620 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-10 21:34:08 +08:00
yihao.dai
7730b910b9
enhance: Decouple compaction from shard ( #33138 )
...
Decouple compaction from shard, remove dependencies on shards (e.g.
SyncSegments, injection).
issue: https://github.com/milvus-io/milvus/issues/32809
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-24 09:07:41 +08:00
XuanYang-cn
22bddde5ff
enhance: Tidy compactor and remove dup codes ( #32198 )
...
See also: #32451
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-05-23 09:53:40 +08:00
