congqixia
3fe0f82923
enhance: Add balance report log for qc balancer ( #36747 )
...
Related to #36746
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-10-11 10:25:24 +08:00
aoiasd
db34572c56
feat: support load and query with bm25 metric ( #36071 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-10-11 10:23:20 +08:00
sre-ci-robot
90285830de
[automated] Update Pytest image changes ( #36755 )
...
Update Pytest image changes
See changes:
f7b811450dsre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-10-11 09:29:19 +08:00
zhagnlu
b1e678dcba
fix: fix json in [] expr bug ( #36721 )
...
#36718
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-10-11 01:11:20 +08:00
Buqian Zheng
f7b811450d
feat: add enable_tokenizer params to VarChar field ( #36480 )
...
issue: #35922
add an enable_tokenizer param to varchar field: must be set to true so
that a varchar field can enable_match or used as input of BM25 function
---------
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-10-10 20:33:21 +08:00
XuanYang-cn
290ceb4e84
enhance: Add more info in logs ( #36731 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-10 17:51:25 +08:00
aoiasd
a5f98e4cb3
enhance: UpdateSegmentsInfo should update remaining segment info even if some one not exist ( #36726 )
...
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-10-10 15:17:19 +08:00
Ted Xu
62679efa0c
enhance: remove unnecessary clone in meta cache ( #36628 )
...
See #36627
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-10-10 15:05:20 +08:00
yihao.dai
3685edb264
enhance: Use common gc config ( #36668 )
...
Use the GC config from `common` and remove the GC config from
`queryNode`.
issue: https://github.com/milvus-io/milvus/issues/36667
related pr: https://github.com/milvus-io/milvus/pull/34949
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-09 19:47:19 +08:00
Ted Xu
5fc731795b
enhance: Datacoord to support prioritization of compaction tasks ( #36547 )
...
See #36550
This PR made 2 changes:
1. Introducing a prioritization mechanism, if
`dataCoord.compaction.taskPrioritizer` is set to `level`, compaction
tasks are always executed as the priority of L0>Mix>Clustering
2. `dataCoord.compaction.maxParallelTaskNum` now controls the
parallelism of executing tasks, not the task number of queue +
executing.
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-10-09 19:11:20 +08:00
nico
e1b511a219
enhance: open mmap on standalone-kafka in ci-e2e ( #36688 )
...
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-10-09 19:03:19 +08:00
Zhen Ye
efb10d3b86
fix: streaming metrics do not clear ( #36701 )
...
issue: #33285
Signed-off-by: chyezh <chyezh@outlook.com>
2024-10-09 17:51:20 +08:00
SimFG
130a923dec
enhance: the estimate method when loading the collection ( #36307 )
...
- issue: #36530
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
Co-authored-by: xianliang.li <xianliang.li@zilliz.com>
2024-10-09 17:35:19 +08:00
XuanYang-cn
c84bdfa766
fix: raise l0 compaction memory ratio to 0.5 ( #36690 )
...
5 percent of free memory is too less for l0 compaction. This pr will
raise it to 50 percent.
See also: #36614
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-09 17:19:24 +08:00
congqixia
c3d910756b
enhance: Update knowhere commit to fix mac compilation ( #36706 )
...
Related to zilliztech/knowhere#879
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-10-09 16:05:20 +08:00
yihao.dai
0fc2a4aa53
enhance: Optimize import scheduling and add time cost metric ( #36601 )
...
1. Optimize import scheduling strategic:
a. Revise slot weights, calculating them based on the number of files
and segments for both import and pre-import tasks.
b. Ensure that the DN executes tasks in ascending order of task ID.
2. Add time cost metric and log.
issue: https://github.com/milvus-io/milvus/issues/36600 ,
https://github.com/milvus-io/milvus/issues/36518
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-09 14:41:20 +08:00
aoiasd
2c3a8b7dea
enhance: fix skip msg log with empty position info ( #36626 )
...
All skiped message position info was nil, should print BeginTs or
MessageID instead.
relate: https://github.com/milvus-io/milvus/issues/36569
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-10-09 12:01:20 +08:00
Zhen Ye
addced27f9
fix: rocksmq consume too slow when the channel is full ( #36617 )
...
issue: #36569
Signed-off-by: chyezh <chyezh@outlook.com>
2024-10-09 11:59:31 +08:00
yihao.dai
4f50b7dbab
enhance: Swap the position of import job states ( #36699 )
...
issue: https://github.com/milvus-io/milvus/issues/36698
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-09 11:29:19 +08:00
Patrick Weizhi Xu
90a0e90732
enhance: remove warnings when there is no partition key set ( #36674 )
...
issue: #36673
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-10-09 10:27:20 +08:00
cai.zhang
fc8b5ab791
enhance: Convert unincode to ascii to improving expression parsing efficiency ( #36675 )
...
issue: #36672
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-10-09 09:23:24 +08:00
Zhen Ye
2ec6e602d6
enhance: add streaming client metrics ( #36523 )
...
issue: #33285
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-10-08 21:25:19 +08:00
sre-ci-robot
8b867aae63
Update all contributors
...
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2024-10-08 12:01:01 +00:00
jaime
ef1832ff9c
enhance: enable manual compaction for collections without indexes ( #36577 )
...
issue: #36576
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-08 19:57:18 +08:00
nico
9fdf0505a8
test: update test case for support dynamic load replica ( #36589 )
...
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-10-08 17:31:20 +08:00
binbin
0d57ff01a6
test: add null and default test cases ( #36612 )
...
issue: #36129
Signed-off-by: binbin lv <binbin.lv@zilliz.com>
2024-10-08 17:25:24 +08:00
jaime
1fded42277
enhance: decrease CPU overhead when calculating index file size ( #36579 )
...
issue: #36578
---------
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-08 14:29:25 +08:00
XuanYang-cn
4e0ea39235
fix: Remove neighbors if compactTo is unindexed ( #36503 )
...
See also: #36360
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-08 14:15:19 +08:00
wayblink
c7a81d5de3
enhance:use proto.clone in compaction tasks to prevent omissions ( #36624 )
...
issue: #36623
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-10-08 11:23:23 +08:00
yihao.dai
1f47d5510b
fix: Fix import segments leak in segment manager ( #36602 )
...
Directly add import segments from the meta, eliminating the dependency
on the segment manager.
issue: https://github.com/milvus-io/milvus/issues/34648
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-08 10:11:22 +08:00
sre-ci-robot
ca9842e4d5
Update all contributors
...
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2024-10-08 00:01:23 +00:00
sre-ci-robot
3936d12661
[automated] Update Knowhere Commit ( #36634 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-10-01 01:05:15 +08:00
sre-ci-robot
3543d0c81d
Update all contributors
...
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2024-09-30 12:00:40 +00:00
congqixia
ddc3e76803
fix: Add defer Unpin when error happens ( #36620 )
...
Resolves : #36619
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-30 19:49:17 +08:00
smellthemoon
2055df81aa
enhance: upgrade pulsar-client-go to 0.12.1 ( #36615 )
...
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-30 17:13:16 +08:00
wei liu
470bb0cc3f
enhance: Enable balance on querynode with different mem capacity ( #36466 )
...
issue: #36464
This PR enable balance on querynode with different mem capacity, for
query node which has more mem capactity will be assigned more records,
and query node with the largest difference between assignedScore and
currentScore will have a higher priority to carry the new segment.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-30 16:15:17 +08:00
wayblink
00a5025949
enhance: support clustering compaction on null value ( #36372 )
...
issue: #36055
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-09-30 14:33:17 +08:00
congqixia
e4292569c7
enhance: Unify InboundLabel case using metrics.Label ( #36613 )
...
Previous label case broken by #36107 , this PR make all inbound label
using label constants from metrics package.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-30 14:15:17 +08:00
Rijin-N
a05a37a583
enhance: GCS native support (GCS implemented using Google Cloud Storage libraries) ( #36214 )
...
Native support for Google cloud storage using the Google Cloud Storage
libraries. Authentication is performed using GCS service account
credentials JSON.
Currently, Milvus supports Google Cloud Storage using S3-compatible APIs
via the AWS SDK. This approach has the following limitations:
1. Overhead: Translating requests between S3-compatible APIs and GCS can
introduce additional overhead.
2. Compatibility Limitations: Some features of the original S3 API may
not fully translate or work as expected with GCS.
To address these limitations, This enhancement is needed.
Related Issue: #36212
2024-09-30 13:23:32 +08:00
jaime
5686a9a024
fix: unhandle error in upsert task ( #36604 )
...
issue: #36611
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-30 12:59:16 +08:00
Chun Han
a25dc98794
enhance: support group_size and hybridsearch+groupby on httpv2 side( #36386 ) ( #36461 )
...
related: #36386
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-30 11:09:16 +08:00
Buqian Zheng
94005b7198
fix: Sparse float vector incorrectly ExpandData at mmap mode ( #36603 )
...
issue: #36561
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-09-30 10:39:16 +08:00
wt
798ef223f1
test: add test cases for param checking ( #36595 )
...
Signed-off-by: wangting0128 <ting.wang@zilliz.com>
2024-09-30 10:33:16 +08:00
Chun Han
a54bffd6cd
fix: refine test case for search_group_by( #36401 ) ( #36511 )
...
related: #36401
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-30 10:13:17 +08:00
cai.zhang
ecb2b242e2
enhance: Add sorted for segment info ( #36469 )
...
issue: #33744
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-30 10:01:16 +08:00
smellthemoon
a78a6b33ab
fix: not check json data when set nullable or default_value ( #36575 )
...
#36354
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-29 20:39:15 +08:00
SimFG
9c1772f659
enhance: avoid to create many timer object in the target ( #36570 )
...
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-29 19:23:16 +08:00
Zhen Ye
a47abb2f2b
fix: stats log lost after disable stats log loading on flush ( #36592 )
...
issue: #36555
Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-29 16:53:15 +08:00
Zhen Ye
a6545b2e29
fix: refactor milvus config and change default txn timeout ( #36522 )
...
issue: #36498
Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-29 11:01:15 +08:00
nico
c43d94319f
enhance: [skip e2e]open querynode mmap for all pipeline ( #36586 )
...
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-09-29 09:41:15 +08:00
