wei liu
8123bea1ae
enhance: Avoid assign too much segment/channels to new querynode ( #34096 )
...
issue: #34095
When a new query node comes online, the segment_checker,
channel_checker, and balance_checker simultaneously attempt to allocate
segments to it. If this occurs during the execution of a load task and
the distribution of the new query node hasn't been updated, the query
coordinator may mistakenly view the new query node as empty. As a
result, it assigns segments or channels to it, potentially overloading
the new query node with more segments or channels than expected.
This PR measures the workload of the executing tasks on the target query
node to prevent assigning an excessive number of segments to it.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-27 19:06:05 +08:00
Cai Yudong
ad90360162
enhance: Update knowhere commit ( #34223 )
...
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-06-27 18:20:06 +08:00
chyezh
d2bc4a53be
enhance: implement rmq and pulsar as wal ( #34046 )
...
issue: #33285
- use reader but not consumer for pulsar
- advanced test framework
- move some streaming related package into pkg
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-27 15:11:05 +08:00
jaime
9c2eeff427
enhance: refine error message for invalid limit parameter ( #34178 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-26 22:14:04 +08:00
cqy123456
51ebe95be5
enhance: remove CheckVecIndexWithDataTypeExist function in pkg and remove some cgo call ( #34102 )
...
issue: https://github.com/milvus-io/milvus/issues/22837
related pr: https://github.com/milvus-io/milvus/pull/34104
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-26 21:13:44 +08:00
congqixia
948bfba93a
fix: Pass otlpSecure config when setup segcore tracing ( #34193 )
...
See also #34192
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-26 19:18:04 +08:00
congqixia
e04f1f9748
enhance: Add unittest for storage.DeleteLog ( #34190 )
...
See also #33787
Backport unit test part in #34188
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-26 17:14:04 +08:00
xige-16
f8bf6909c6
fix: Fix datacoord's properties overwriten ( #34146 )
...
issue: #34144
/kind bug
Signed-off-by: xige-16 <xige2016@gmail.com>
2024-06-26 11:56:04 +08:00
wei liu
45203425fd
enhance: Avoid search querynode return nil status in response ( #34100 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-26 11:50:11 +08:00
Gao
a933f6731b
fix: centroids file not removed when data skew in major compaction ( #34050 )
...
issue: https://github.com/milvus-io/milvus/issues/30633
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-06-26 10:48:04 +08:00
wayblink
fbe3231b1f
fix: fix error ignore in compactor ( #34169 )
...
#34170
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-26 10:24:03 +08:00
sre-ci-robot
cc54ab2e47
[automated] Update Knowhere Commit ( #34172 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-06-26 02:20:03 +08:00
jaime
9630974fbb
enhance: move rocksmq from internal to pkg module ( #33881 )
...
issue: #33956
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-25 21:18:15 +08:00
cai.zhang
c65f41dc60
fix: Only sync flushed segments to datanode ( #34156 )
...
issue: #33540
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-06-25 20:46:07 +08:00
wayblink
f9a0f7bb25
Add an option to enable/disable vector field clustering key ( #34097 )
...
#30633
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-25 18:52:04 +08:00
yihao.dai
940a0ac0a3
fix: Prevent import from generating orphaned files ( #34070 )
...
When import failed, mark the import segment as dropped instead of drop
it directly to prevent generating orphaned files.
issue: https://github.com/milvus-io/milvus/issues/34068
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-25 16:58:02 +08:00
congqixia
962a5446f8
enhance: Add ctx in SyncTask.Run to be cancellable ( #34042 )
...
Related to #33716
This PR add context param in SyncTask.Run execution functions to make it
cancellable from the caller.
This make it possible to cancel task when datanode/data sync service is
beeing shut down.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-25 14:22:04 +08:00
zhenshan.cao
d18c49013b
enhance: Refine compaction ( #33982 )
...
issue : https://github.com/milvus-io/milvus/issues/32939
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-06-25 10:08:03 +08:00
congqixia
506a915272
fix: Deep copy ImportTask.segmentsInfo to prevent data race ( #34090 )
...
See also #34089
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-25 10:06:02 +08:00
sre-ci-robot
254b555d53
[automated] Update Knowhere Commit ( #34119 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-06-25 01:22:02 +08:00
congqixia
07c25a19d9
fix: Make querycoord panick when rg metastore sync fail ( #34106 )
...
See also #34047
When `unassignNode` sync resource group with node removed failed
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-24 21:38:02 +08:00
congqixia
fd922d921a
enhance: Add nilness linter and fix some small issues ( #34049 )
...
Add `nilness` for govet linter and fixed some detected issues
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-24 14:52:03 +08:00
Jiquan Long
aa36f9feed
fix: [ut] regex query under unsupported index ( #34087 )
...
/kind improvement
issue: https://github.com/milvus-io/milvus/issues/29988
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-06-24 14:08:03 +08:00
jaime
d08cb885ca
enhance: enable flush rate limiter of collection level ( #33837 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-24 10:52:03 +08:00
chyezh
b9237280c2
enhance: wal interface definition ( #33745 )
...
issue: #33285
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-24 10:34:12 +08:00
Patrick Weizhi Xu
b961767005
enhance: support integral type for MV and skip MV if there is only one category ( #33161 )
...
issue: #29892
---------
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-06-24 10:20:01 +08:00
congqixia
b5c9a7364b
fix: Prevent remove new growing L1 segment when SyncSegments ( #34056 )
...
Related to #34018
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-24 10:18:01 +08:00
Chun Han
ca7ef26e4b
fix: sync part stats task cannot be finished( #30376 ) ( #34027 )
...
related: #30376
also: refine log output for query_coord task by rephrasing action string
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-06-24 10:16:02 +08:00
Ted Xu
78885a44c4
fix: turn on compression on stream writers ( #34067 )
...
See #31679
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-06-24 10:08:02 +08:00
zhagnlu
03a3f50892
enhance: add skip using array index when some situation ( #33947 )
...
#32900
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-23 21:26:02 +08:00
zhagnlu
0d7ea8ec42
enhance: Enhance and correct exception module ( #33705 )
...
#33704
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-23 21:22:01 +08:00
yihao.dai
6c1d815894
enhance: Remove the unused compaction logic from shard ( #33932 )
...
1. Remove the `compactTo` field in `SegmentInfo`.
2. Remove the target segment not match and its retry logic in
`SyncManager`.
issue: https://github.com/milvus-io/milvus/issues/32809
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-23 21:12:01 +08:00
yihao.dai
fb870d2426
fix: Do compressBinlog to fix logID 0 ( #34060 )
...
issue: https://github.com/milvus-io/milvus/issues/34059
Do compressBinlog to ensure that reloadFromKV will fill binlogs' logID
after datacoord restarts.
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-22 16:14:01 +08:00
chyezh
259a682673
enhance: async search and retrieve in cgo ( #33228 )
...
issue: #30926 , #33132
related pr: #33133
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-22 09:38:02 +08:00
wayblink
380d3f4469
fix: Fix memory buffer error & some renaming ( #33850 )
...
#30633
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-21 17:30:01 +08:00
congqixia
2f691f1e67
enhance: Unify DeleteLog parsing code ( #34009 )
...
See also #33787
The parsing delete log is distributed in lots of places, which is not
recommended and hard to maintain.
This PR abstract common parsing logic into `DeleteLog.Parse` method to
unify implementation and make it easier to replace json parsing lib.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-21 16:54:01 +08:00
Bingyi Sun
622be36ad7
fix: fix l0 compaction may miss some sealed segments ( #33980 )
...
issue: https://github.com/milvus-io/milvus/issues/33955
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-06-21 16:52:15 +08:00
jaime
e653ad27e2
fix: metrics database_num is 0 after restarting rootcoord ( #34011 )
...
issue: #34041
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-06-21 11:36:00 +08:00
sre-ci-robot
d9f986439d
[automated] Update Knowhere Commit ( #34040 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-06-21 01:50:01 +08:00
presburger
dac20d41c0
fix: fix milvus gpu compile error ( #33995 )
...
NVIDIA RAFT uses customized versions of spdlog and fmt, which conflict
with the versions provided by Conan. Therefore, it is necessary to
isolate the build environment for knowhere to avoid these conflicts.
issue: https://github.com/milvus-io/milvus/issues/33991
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
2024-06-20 20:30:01 +08:00
XuanYang-cn
04edb07d82
enhance: Add deltaRowCount in l0 compaction ( #33997 )
...
See also: #33998
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-06-20 17:46:01 +08:00
cqy123456
dc4437ff82
enhance: use segment id and type to register in MmapChunkManager and opt malloc in variableChunk ( #33993 )
...
issue: https://github.com/milvus-io/milvus/issues/32984
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-20 17:42:02 +08:00
wei liu
935bc1fb71
fix: Fix GetReplicas API return nil status ( #33715 )
...
issue: #33702
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-20 14:40:15 +08:00
shaoting-huang
5f02e52561
enhance: Refactor data codec deserialize ( #33923 )
...
#33922
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-06-20 11:17:59 +08:00
wei liu
31ef0a1fe8
enhance: Add trace for bf cost in l0 compactor ( #33860 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-20 10:10:05 +08:00
congqixia
6a15b6e2be
fix: Remove loopclosure issue in ChannelManagerImplV2 ( #33989 )
...
See also #33987
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-19 21:25:59 +08:00
smellthemoon
2a1356985d
enhance: support null in go payload ( #32296 )
...
#31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-06-19 17:08:00 +08:00
Ted Xu
6d5747cb3e
feat: adding deltalog stream reader and writer ( #33844 )
...
See #31679
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-06-19 14:42:01 +08:00
Jiquan Long
7b9462c0d3
enhance: fix copying hits of inverted index twice ( #33968 )
...
issue: https://github.com/milvus-io/milvus/issues/29793
The custom `VecCollector` have already transformed the results into
vector of offsets, no need to copy them twice.
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-06-19 12:40:01 +08:00
wayblink
5fac2fa1d2
fix: Panic if ProcessActiveStandBy returns error ( #33369 )
...
#33368
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-06-19 11:16:00 +08:00
