cai.zhang
feb13cdf07
enhance: Skip pick worker when task doesn't need to execute actually ( #34348 )
...
issue: #34347
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-03 15:52:09 +08:00
cai.zhang
8165961223
enhance: Optimeize clustering compaction ( #34313 )
...
issue: #30633
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-03 15:14:08 +08:00
wayblink
fb61344dc9
fix: Revert a optimize in clustering compaction ( #34299 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-03 10:32:08 +08:00
cai.zhang
a5be322ab2
enhance: Don't set the timeout for SyncSegments ( #34226 )
...
issue: #32809
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-02 14:28:07 +08:00
cai.zhang
4cf1a358ba
fix: Sync the sealed and flushed segments to datanode ( #34301 )
...
issue: #33696
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-01 22:42:08 +08:00
congqixia
e083d99dd9
enhance: Use channel name map finding channel watcher ( #34294 )
...
See also #33235
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-01 18:46:07 +08:00
jaime
d6afb31b94
enhance: make subfunctions of datanode component modular ( #33992 )
...
issue: #33994
also remove deprecated channel manager based on the etcd implementation
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-01 14:46:07 +08:00
jaime
0426390f06
enhance: improve check health ( #33800 )
...
issue: #34264
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-01 10:16:06 +08:00
wayblink
73ffc1b424
Refine clustering_compaction_task retry mechanism ( #34194 )
...
#32939
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-30 20:22:09 +08:00
Chun Han
96dcee5dff
fix:load major compaction partial result( #34051 ) ( #34052 )
...
related: #34051
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-06-28 14:04:05 +08:00
jaime
9630974fbb
enhance: move rocksmq from internal to pkg module ( #33881 )
...
issue: #33956
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-25 21:18:15 +08:00
cai.zhang
c65f41dc60
fix: Only sync flushed segments to datanode ( #34156 )
...
issue: #33540
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-06-25 20:46:07 +08:00
yihao.dai
940a0ac0a3
fix: Prevent import from generating orphaned files ( #34070 )
...
When import failed, mark the import segment as dropped instead of drop
it directly to prevent generating orphaned files.
issue: https://github.com/milvus-io/milvus/issues/34068
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-25 16:58:02 +08:00
zhenshan.cao
d18c49013b
enhance: Refine compaction ( #33982 )
...
issue : https://github.com/milvus-io/milvus/issues/32939
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-06-25 10:08:03 +08:00
Patrick Weizhi Xu
b961767005
enhance: support integral type for MV and skip MV if there is only one category ( #33161 )
...
issue: #29892
---------
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-06-24 10:20:01 +08:00
Bingyi Sun
622be36ad7
fix: fix l0 compaction may miss some sealed segments ( #33980 )
...
issue: https://github.com/milvus-io/milvus/issues/33955
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-06-21 16:52:15 +08:00
XuanYang-cn
04edb07d82
enhance: Add deltaRowCount in l0 compaction ( #33997 )
...
See also: #33998
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-06-20 17:46:01 +08:00
congqixia
6a15b6e2be
fix: Remove loopclosure issue in ChannelManagerImplV2 ( #33989 )
...
See also #33987
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-19 21:25:59 +08:00
wayblink
5fac2fa1d2
fix: Panic if ProcessActiveStandBy returns error ( #33369 )
...
#33368
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-19 11:16:00 +08:00
wayblink
5cb0760187
fix: Small fixs of major compaction ( #33929 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-18 16:53:58 +08:00
cai.zhang
95148866ed
fix: Don't remove growing L0 segment in datanode metacache ( #33829 )
...
issue: #33540
1. gorwing L0 segments is invisible to datacoord.
2. flushed L0 segments need to clean by datacoord.
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-06-17 10:09:57 +08:00
yihao.dai
8537f3daeb
enhance: Rename Compaction to CompactionV2 ( #33858 )
...
Due to the removal of injection and syncSegments from the compaction, we
need to ensure that no compaction is successfully executed during the
rolling upgrade. This PR renames Compaction to CompactionV2, with the
following effects:
- New datacoord + old datanode: Utilizes the CompactionV2 interface,
resulting in the datanode error "CompactionV2 not implemented," causing
compaction to fail;
- Old datacoord + new datanode: Utilizes the CompactionV1 interface,
resulting in the datanode error "CompactionV1 not implemented," causing
compaction to fail.
issue: https://github.com/milvus-io/milvus/issues/32809
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-16 22:07:57 +08:00
wei liu
4987067375
enhance: Execute bloom filter apply in parallel to speed up segment predict ( #33792 )
...
issue: #33610
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-14 11:37:56 +08:00
chyezh
b3ee1a4bd5
fix: lost DroppedAt field when DropCollection ( #33766 )
...
issue: #33097
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-12 20:21:56 +08:00
Chun Han
f7af323d1e
fix: sync partitiion stats blocking balance task( #33741 ) ( #33742 )
...
related: #33741
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-11 14:21:56 +08:00
yihao.dai
b1d46eb34b
fix: Fix multiple vector fields import ( #33723 )
...
1. Fix dim mismatch with multi-vector fields and JSON import
2. Enhance: do not display file ID in GetImportResponse.
issue: https://github.com/milvus-io/milvus/issues/33681 ,
https://github.com/milvus-io/milvus/issues/33682
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-10 21:57:54 +08:00
yihao.dai
eb5d4de390
fix: Check if the import job exists ( #33672 )
...
issue: https://github.com/milvus-io/milvus/issues/33671
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-10 21:51:55 +08:00
wayblink
a1232fafda
feat: Major compaction ( #33620 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-10 21:34:08 +08:00
yihao.dai
3540eee977
enhance: Support L0 import ( #33514 )
...
issue: https://github.com/milvus-io/milvus/issues/33157
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-07 14:17:20 +08:00
smellthemoon
c61fb1eff5
enhance: do check when add not empty logpath ( #33640 )
...
meta only store logid
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-06-07 10:19:51 +08:00
cai.zhang
27cc9f2630
enhance: Support analyze data ( #33651 )
...
issue: #30633
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>
2024-06-06 17:37:51 +08:00
cai.zhang
cfea3f43cf
fix: Don't sync L0 segments to channel watcher ( #33664 )
...
issue: #33540
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-06-06 15:59:50 +08:00
cai.zhang
feeb869ff9
enhance: Remove compaction plans on the datanode ( #33548 )
...
issue: #33546
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-06-05 20:27:51 +08:00
yihao.dai
01ce32caa1
enhance: Print more disk quota info ( #33596 )
...
/kind enhancement
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-05 16:51:56 +08:00
yihao.dai
35532a3e7d
fix: Fill stats log id and check validity ( #33477 )
...
1. Fill log ID of stats log from import
2. Add a check to validate the log ID before writing to meta
issue: https://github.com/milvus-io/milvus/issues/33476
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-05 11:17:56 +08:00
zhenshan.cao
ac4f3997ce
enhance: Reconstructing Compaction to possess persistence capability ( #33265 )
...
issue #33586
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-06-05 10:17:50 +08:00
yiwangdr
180d754158
fix: speed up segment lookup via channel name in datacoord ( #33530 )
...
issue: #33342
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-06-03 14:47:47 +08:00
yihao.dai
7a2127b09f
enhance: Avoid redundant meta operations of import ( #33518 )
...
issue: https://github.com/milvus-io/milvus/issues/33513
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-03 14:15:53 +08:00
cai.zhang
77637180fa
enhance: Periodically synchronize segments to datanode watcher ( #33420 )
...
issue: #32809
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-05-30 13:37:44 +08:00
congqixia
54797b4286
enhance: Refine frequent log in datacoord ( #33449 )
...
This PR changes:
- Frequent `ListIndexes` success log to debug level
- Aggregate collection missing log after collection dropped in
`meta.GetCollectionIndexFilesSize`
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-29 19:15:43 +08:00
jaime
3d29907b6e
enhance: decrease cpu overhead during filter segments on datacoord ( #33130 )
...
issue: #33129
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-05-28 19:17:43 +08:00
XuanYang-cn
5e39aa9272
enhance: Make channel meta able to writer 200k plus segments ( #33279 )
...
See also: #33125
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-05-28 12:33:42 +08:00
SimFG
2453181218
fix: not found database name in the datacoord meta object ( #33411 )
...
- issue: #33410
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-05-28 10:09:48 +08:00
yiwangdr
e895cfed84
fix: reduce redundant map operations in datacoord ( #33343 )
...
More refactories will be added.
issue: #33342
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-05-24 12:47:40 +08:00
yihao.dai
7730b910b9
enhance: Decouple compaction from shard ( #33138 )
...
Decouple compaction from shard, remove dependencies on shards (e.g.
SyncSegments, injection).
issue: https://github.com/milvus-io/milvus/issues/32809
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-24 09:07:41 +08:00
cai.zhang
be77ceba84
enhance: Use proto for passing info in cgo ( #33184 )
...
issue: #33183
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-05-23 10:31:40 +08:00
XuanYang-cn
22bddde5ff
enhance: Tidy compactor and remove dup codes ( #32198 )
...
See also: #32451
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-05-23 09:53:40 +08:00
XuanYang-cn
819a624753
fix: Return error when startup Delete/AddNode fail ( #33193 )
...
See also: #33151 , #33149
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-05-22 11:17:38 +08:00
cai.zhang
ed39a38953
enhance: Reduce the frequency of logs describing indexing failures ( #33212 )
...
issue: #33001 #33102
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-05-21 19:27:39 +08:00
yihao.dai
017fd7bc25
enhance: Select L2 segments in L0Compaction as well ( #32991 )
...
/kind improvement
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-05-21 16:13:39 +08:00
