test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-11-30 02:48:45 +08:00
Code Issues Actions 11 Packages Projects Releases Wiki Activity
20,978 Commits 94 Branches 105 Tags 184 MiB
5a1f752272
Commit Graph

20978 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
XuanYang-cn
290ceb4e84
enhance: Add more info in logs (#36731) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-10-10 17:51:25 +08:00
aoiasd
a5f98e4cb3
enhance: UpdateSegmentsInfo should update remaining segment info even if some one not exist (#36726) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-10-10 15:17:19 +08:00
Ted Xu
62679efa0c
enhance: remove unnecessary clone in meta cache (#36628) 
See #36627

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-10-10 15:05:20 +08:00
yihao.dai
3685edb264
enhance: Use common gc config (#36668) 
Use the GC config from `common` and remove the GC config from
`queryNode`.

issue: https://github.com/milvus-io/milvus/issues/36667

related pr: https://github.com/milvus-io/milvus/pull/34949

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-09 19:47:19 +08:00
Ted Xu
5fc731795b
enhance: Datacoord to support prioritization of compaction tasks (#36547) 
See #36550

This PR made 2 changes:

1. Introducing a prioritization mechanism, if
`dataCoord.compaction.taskPrioritizer` is set to `level`, compaction
tasks are always executed as the priority of L0>Mix>Clustering
2. `dataCoord.compaction.maxParallelTaskNum` now controls the
parallelism of executing tasks, not the task number of queue +
executing.

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-10-09 19:11:20 +08:00
nico
e1b511a219
enhance: open mmap on standalone-kafka in ci-e2e (#36688) 
Signed-off-by: nico <cheng.yuan@zilliz.com>
 2024-10-09 19:03:19 +08:00
Zhen Ye
efb10d3b86
fix: streaming metrics do not clear (#36701) 
issue: #33285

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-10-09 17:51:20 +08:00
SimFG
130a923dec
enhance: the estimate method when loading the collection (#36307) 
- issue: #36530

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
Co-authored-by: xianliang.li <xianliang.li@zilliz.com>
 2024-10-09 17:35:19 +08:00
XuanYang-cn
c84bdfa766
fix: raise l0 compaction memory ratio to 0.5 (#36690) 
5 percent of free memory is too less for l0 compaction. This pr will
raise it to 50 percent.

See also: #36614

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-10-09 17:19:24 +08:00
congqixia
c3d910756b
enhance: Update knowhere commit to fix mac compilation (#36706) 
Related to zilliztech/knowhere#879

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-10-09 16:05:20 +08:00
yihao.dai
0fc2a4aa53
enhance: Optimize import scheduling and add time cost metric (#36601) 
1. Optimize import scheduling strategic:
a. Revise slot weights, calculating them based on the number of files
and segments for both import and pre-import tasks.
b. Ensure that the DN executes tasks in ascending order of task ID.
2. Add time cost metric and log.

issue: https://github.com/milvus-io/milvus/issues/36600,
https://github.com/milvus-io/milvus/issues/36518

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-09 14:41:20 +08:00
aoiasd
2c3a8b7dea
enhance: fix skip msg log with empty position info (#36626) 
All skiped message position info was nil, should print BeginTs or
MessageID instead.
relate: https://github.com/milvus-io/milvus/issues/36569

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-10-09 12:01:20 +08:00
Zhen Ye
addced27f9
fix: rocksmq consume too slow when the channel is full (#36617) 
issue: #36569

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-10-09 11:59:31 +08:00
yihao.dai
4f50b7dbab
enhance: Swap the position of import job states (#36699) 
issue: https://github.com/milvus-io/milvus/issues/36698

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-09 11:29:19 +08:00
Patrick Weizhi Xu
90a0e90732
enhance: remove warnings when there is no partition key set (#36674) 
issue: #36673

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-10-09 10:27:20 +08:00
cai.zhang
fc8b5ab791
enhance: Convert unincode to ascii to improving expression parsing efficiency (#36675) 
issue: #36672

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-10-09 09:23:24 +08:00
Zhen Ye
2ec6e602d6
enhance: add streaming client metrics (#36523) 
issue: #33285

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-10-08 21:25:19 +08:00
sre-ci-robot
8b867aae63 Update all contributors 
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
 2024-10-08 12:01:01 +00:00
jaime
ef1832ff9c
enhance: enable manual compaction for collections without indexes (#36577) 
issue: #36576

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-10-08 19:57:18 +08:00
nico
9fdf0505a8
test: update test case for support dynamic load replica (#36589) 
Signed-off-by: nico <cheng.yuan@zilliz.com>
 2024-10-08 17:31:20 +08:00
binbin
0d57ff01a6
test: add null and default test cases (#36612) 
issue: #36129

Signed-off-by: binbin lv <binbin.lv@zilliz.com>
 2024-10-08 17:25:24 +08:00
jaime
1fded42277
enhance: decrease CPU overhead when calculating index file size (#36579) 
issue: #36578

---------

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-10-08 14:29:25 +08:00
XuanYang-cn
4e0ea39235
fix: Remove neighbors if compactTo is unindexed (#36503) 
See also: #36360

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-10-08 14:15:19 +08:00
wayblink
c7a81d5de3
enhance:use proto.clone in compaction tasks to prevent omissions (#36624) 
issue: #36623

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-10-08 11:23:23 +08:00
yihao.dai
1f47d5510b
fix: Fix import segments leak in segment manager (#36602) 
Directly add import segments from the meta, eliminating the dependency
on the segment manager.

issue: https://github.com/milvus-io/milvus/issues/34648

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-10-08 10:11:22 +08:00
sre-ci-robot
ca9842e4d5 Update all contributors 
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
 2024-10-08 00:01:23 +00:00
sre-ci-robot
3936d12661
[automated] Update Knowhere Commit (#36634) 
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
 2024-10-01 01:05:15 +08:00
sre-ci-robot
3543d0c81d Update all contributors 
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
 2024-09-30 12:00:40 +00:00
congqixia
ddc3e76803
fix: Add defer Unpin when error happens (#36620) 
Resolves: #36619

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-30 19:49:17 +08:00
smellthemoon
2055df81aa
enhance: upgrade pulsar-client-go to 0.12.1 (#36615) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-09-30 17:13:16 +08:00
wei liu
470bb0cc3f
enhance: Enable balance on querynode with different mem capacity (#36466) 
issue: #36464
This PR enable balance on querynode with different mem capacity, for
query node which has more mem capactity will be assigned more records,
and query node with the largest difference between assignedScore and
currentScore will have a higher priority to carry the new segment.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-09-30 16:15:17 +08:00
wayblink
00a5025949
enhance: support clustering compaction on null value (#36372) 
issue: #36055

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-09-30 14:33:17 +08:00
congqixia
e4292569c7
enhance: Unify InboundLabel case using metrics.Label (#36613) 
Previous label case broken by #36107, this PR make all inbound label
using label constants from metrics package.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-30 14:15:17 +08:00
Rijin-N
a05a37a583
enhance: GCS native support (GCS implemented using Google Cloud Storage libraries) (#36214) 
Native support for Google cloud storage using the Google Cloud Storage
libraries. Authentication is performed using GCS service account
credentials JSON.

Currently, Milvus supports Google Cloud Storage using S3-compatible APIs
via the AWS SDK. This approach has the following limitations:

1. Overhead: Translating requests between S3-compatible APIs and GCS can
introduce additional overhead.
2. Compatibility Limitations: Some features of the original S3 API may
not fully translate or work as expected with GCS.

To address these limitations, This enhancement is needed.

Related Issue: #36212
 2024-09-30 13:23:32 +08:00
jaime
5686a9a024
fix: unhandle error in upsert task (#36604) 
issue: #36611

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-09-30 12:59:16 +08:00
Chun Han
a25dc98794
enhance: support group_size and hybridsearch+groupby on httpv2 side(#36386) (#36461) 
related: #36386

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-09-30 11:09:16 +08:00
Buqian Zheng
94005b7198
fix: Sparse float vector incorrectly ExpandData at mmap mode (#36603) 
issue: #36561

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-09-30 10:39:16 +08:00
wt
798ef223f1
test: add test cases for param checking (#36595) 
Signed-off-by: wangting0128 <ting.wang@zilliz.com>
 2024-09-30 10:33:16 +08:00
Chun Han
a54bffd6cd
fix: refine test case for search_group_by(#36401) (#36511) 
related: #36401

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-09-30 10:13:17 +08:00
cai.zhang
ecb2b242e2
enhance: Add sorted for segment info (#36469) 
issue: #33744

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-30 10:01:16 +08:00
smellthemoon
a78a6b33ab
fix: not check json data when set nullable or default_value (#36575) 
#36354

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-09-29 20:39:15 +08:00
SimFG
9c1772f659
enhance: avoid to create many timer object in the target (#36570) 
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-09-29 19:23:16 +08:00
Zhen Ye
a47abb2f2b
fix: stats log lost after disable stats log loading on flush (#36592) 
issue: #36555

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-09-29 16:53:15 +08:00
Zhen Ye
a6545b2e29
fix: refactor milvus config and change default txn timeout (#36522) 
issue: #36498

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-09-29 11:01:15 +08:00
nico
c43d94319f
enhance: [skip e2e]open querynode mmap for all pipeline (#36586) 
Signed-off-by: nico <cheng.yuan@zilliz.com>
 2024-09-29 09:41:15 +08:00
binbin
d1d5a50014
test: add null and default test cases (#36539) 
issue: #36129

Signed-off-by: binbin lv <binbin.lv@zilliz.com>
 2024-09-29 09:39:22 +08:00
yihao.dai
8ed34dce84
enhance: Reopen chunk cache cpp ut (#33622) 
issue: https://github.com/milvus-io/milvus/issues/33210

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-09-28 18:19:15 +08:00
zhagnlu
9e3efa06be
fix:fix empty search result bug (#36582) 
#36450

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-09-28 17:45:16 +08:00
yihao.dai
80f25d497f
enhance: Add metrics to monitor import throughput and imported rows (#36519) 
issue: https://github.com/milvus-io/milvus/issues/36518

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-09-28 17:31:15 +08:00
cai.zhang
acc9b5ada5
fix: Regenerate partition stats after stats task for clustering compaction (#36408) 
issue: #33744

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-28 17:29:21 +08:00