Commit Graph

20847 Commits

Author SHA1 Message Date
wayblink
00a5025949
enhance: support clustering compaction on null value (#36372)
issue: #36055

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-09-30 14:33:17 +08:00
congqixia
e4292569c7
enhance: Unify InboundLabel case using metrics.Label (#36613)
Previous label case broken by #36107, this PR make all inbound label
using label constants from metrics package.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-30 14:15:17 +08:00
Rijin-N
a05a37a583
enhance: GCS native support (GCS implemented using Google Cloud Storage libraries) (#36214)
Native support for Google cloud storage using the Google Cloud Storage
libraries. Authentication is performed using GCS service account
credentials JSON.

Currently, Milvus supports Google Cloud Storage using S3-compatible APIs
via the AWS SDK. This approach has the following limitations:

1. Overhead: Translating requests between S3-compatible APIs and GCS can
introduce additional overhead.
2. Compatibility Limitations: Some features of the original S3 API may
not fully translate or work as expected with GCS.

To address these limitations, This enhancement is needed.

Related Issue: #36212
2024-09-30 13:23:32 +08:00
jaime
5686a9a024
fix: unhandle error in upsert task (#36604)
issue: #36611

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-30 12:59:16 +08:00
Chun Han
a25dc98794
enhance: support group_size and hybridsearch+groupby on httpv2 side(#36386) (#36461)
related: #36386

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-30 11:09:16 +08:00
Buqian Zheng
94005b7198
fix: Sparse float vector incorrectly ExpandData at mmap mode (#36603)
issue: #36561

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-09-30 10:39:16 +08:00
wt
798ef223f1
test: add test cases for param checking (#36595)
Signed-off-by: wangting0128 <ting.wang@zilliz.com>
2024-09-30 10:33:16 +08:00
Chun Han
a54bffd6cd
fix: refine test case for search_group_by(#36401) (#36511)
related: #36401

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-30 10:13:17 +08:00
cai.zhang
ecb2b242e2
enhance: Add sorted for segment info (#36469)
issue: #33744

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-30 10:01:16 +08:00
smellthemoon
a78a6b33ab
fix: not check json data when set nullable or default_value (#36575)
#36354

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-29 20:39:15 +08:00
SimFG
9c1772f659
enhance: avoid to create many timer object in the target (#36570)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-29 19:23:16 +08:00
Zhen Ye
a47abb2f2b
fix: stats log lost after disable stats log loading on flush (#36592)
issue: #36555

Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-29 16:53:15 +08:00
Zhen Ye
a6545b2e29
fix: refactor milvus config and change default txn timeout (#36522)
issue: #36498

Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-29 11:01:15 +08:00
nico
c43d94319f
enhance: [skip e2e]open querynode mmap for all pipeline (#36586)
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-09-29 09:41:15 +08:00
binbin
d1d5a50014
test: add null and default test cases (#36539)
issue: #36129

Signed-off-by: binbin lv <binbin.lv@zilliz.com>
2024-09-29 09:39:22 +08:00
yihao.dai
8ed34dce84
enhance: Reopen chunk cache cpp ut (#33622)
issue: https://github.com/milvus-io/milvus/issues/33210

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-28 18:19:15 +08:00
zhagnlu
9e3efa06be
fix:fix empty search result bug (#36582)
#36450

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-28 17:45:16 +08:00
yihao.dai
80f25d497f
enhance: Add metrics to monitor import throughput and imported rows (#36519)
issue: https://github.com/milvus-io/milvus/issues/36518

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-28 17:31:15 +08:00
cai.zhang
acc9b5ada5
fix: Regenerate partition stats after stats task for clustering compaction (#36408)
issue: #33744

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-28 17:29:21 +08:00
cai.zhang
7bf40694fd
fix: Record the nodeID before assigning tasks (#36371)
issue: #33744

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-28 17:21:15 +08:00
cai.zhang
2adca8b754
fix: Fix data race for cluerting compaction (#36440)
issue: #36438

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-28 17:19:21 +08:00
zhuwenxing
31353ae406
test: add restful cases for text match feature (#36405)
/kind improvement

Since creating a collection with text match is not yet implemented on
the RESTful interface, we will temporarily use pymilvus to create a
collection for now. This PR includes a case to test using text match
filters in search queries through the RESTful interface.

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-09-28 16:51:15 +08:00
Zhen Ye
bd7910632a
fix: etcd session discoverer cannot recover from ErrCompacted (#36568)
issue: #36554

Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-28 10:51:14 +08:00
aoiasd
ffc12fb5c4
fix: split delete task msg to MaxMessageSize to avoid mq message too large error (#36197)
relate: https://github.com/milvus-io/milvus/issues/36089

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-27 18:15:19 +08:00
smellthemoon
b1ac3f0df0
enhance: do not log out the full req (#36546)
![img_v3_02f3_61ac166c-df6f-4b5e-b06a-756e5f40a65g](https://github.com/user-attachments/assets/df668339-d0d8-4295-a2b6-52d89ad98766)

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-27 16:39:15 +08:00
wt
1f271e39c6
test: add partition key & bitmap test case (#36564)
Signed-off-by: wangting0128 <ting.wang@zilliz.com>
2024-09-27 14:17:14 +08:00
yanliang567
e6c7fd6605
test: Add group search test on sparse vectors (#36562)
related issue: #36295
add one test for alter alias

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2024-09-27 14:15:14 +08:00
zhuwenxing
7c2cb8c5d4
test: add bulk insert case for text match feature (#36398)
/kind improvement

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-09-27 10:01:14 +08:00
wt
ff4c62e44f
test: add more scalar filter expressions (#36545)
Signed-off-by: wangting0128 <ting.wang@zilliz.com>
2024-09-27 09:59:14 +08:00
zhuwenxing
9444329da1
test: relax the checks on range search (#36542)
/kind improvement

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-09-27 09:57:13 +08:00
cai.zhang
50905e0b45
fix: Check string array max length after type matching (#36449)
issue: #36029

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-27 09:45:14 +08:00
Zhen Ye
d29e01e284
fix: port listen racing in mix or standalone mode (#36442)
issue: #36441

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-26 21:23:16 +08:00
wayblink
7ff41697f9
enhance: make slotUsage a field of CompactionTask (#36510)
#36509

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-09-26 20:19:14 +08:00
wei liu
c056620899
fix: Skip unnecessary query node health check in proxy (#36491)
issue: #36490
After the query node changes from a delegator to a worker, proxy should
skip this querynode's health check.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-26 18:13:15 +08:00
wei liu
55be814a58
enhance: make TransferChannel/TransferSegment idempotent (#36489)
issue: #36488
when call TransferChannel/TransferSegment, querycoord will generate and
submit balance task to scheduler, if segment/channel's task already
exist in scheduler, submit task will failed.

to make TransferChannel/TransferSegment idempotent, we should skip to
submit if task already exist in scheduler.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-26 18:11:23 +08:00
wei liu
5dfa1c3397
fix: Segment unbalance after many times load/release (#36537)
issue: #36536
query coord use `segmentTaskDeleta/channelTaskDelta` to measure the
executing workload for querynode in scheduler, and we maintains the
`segmentTaskDeleta/channelTaskDelta` by `scheulder.Add(task)` and
`scheduler.remove(task)`, but `scheduler.remove(task)` has been called
in unexpected way, which cause a wrong
`segmentTaskDeleta/channelTaskDelta` value and affect the segment assign
logic, causes segment unbalance.

This PR moves to compute the `segmentTaskDeleta/channelTaskDelta` when
access, to avoid the wrong value affect.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-26 15:13:15 +08:00
zhagnlu
0799d927c6
fix:fix term expr overflow bug (#36525)
#36520

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-26 15:01:14 +08:00
smellthemoon
b60164b882
enhance: support null in bulk insert of binlog to help backup null (#36526)
https://github.com/milvus-io/milvus/issues/36341

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-26 14:35:14 +08:00
SimFG
c94b69c2f6
enhance: update the expr version and format the expr http response (#36406)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-26 14:27:20 +08:00
sthuang
4493aa2142
fix: querycoord collection num metric (#36471)
related to: #36456

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-09-26 14:23:13 +08:00
zhuwenxing
954d8a5f68
test: add search test cases with text match (#36399)
/kind improvement

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-09-26 10:15:14 +08:00
congqixia
4fd9b0a8e3
enhance: Return segment id hint in QueryStream response (#36487)
Related to #36482

This PR reuses `SealedSegmentIDsRetrieved` field in `RetrieveResults`
struct to store segment id hint.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-26 10:13:14 +08:00
yihao.dai
9e8cafcbe2
enhance: Skip loading bf in datanode (#36367)
Skip loading bf in datanode:
1. When watching vchannels, skip loading bloom filters for segments.
2. Bypass bloom filter checks for delete messages, directly writing to
L0 segments.
3. Remove flushed segments proactively after flush.

issue: https://github.com/milvus-io/milvus/issues/34585

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-26 10:11:15 +08:00
smellthemoon
a8c80abe36
enhance: upgrade pulsar-client-go to 0.11.1 (#36435)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-26 10:09:13 +08:00
sre-ci-robot
1970ab4c34
[automated] Update Pytest image changes (#36496)
Update Pytest image changes
See changes:
58baeee8f1
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-09-26 10:05:13 +08:00
nico
cfd636ed5b
test: add different language tests and modify some cases (#36465)
fix: #36396

Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-09-26 09:21:13 +08:00
sre-ci-robot
447e326629
[automated] Update Knowhere Commit (#36527)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-09-26 01:15:13 +08:00
wt
14be89df29
test: add case to enable offset cache and mmap at the same time (#36505)
Signed-off-by: wangting0128 <ting.wang@zilliz.com>
2024-09-25 19:59:13 +08:00
Buqian Zheng
8495bc6bbc
fix: fix broken Sparse Float Vector raw data mmap (#36183)
issue: https://github.com/milvus-io/milvus/issues/36182

* improved `Column.h` to make the code much more readable and
maintainable, and added detailed comments.
* fixed an issue where `ArrayColumn::NumRows()` always returns 0 when
the mmap backing storage is a file.
* removed unused `ColumnBase` constructors and unnecessary members so we
don't get confused.
* Updated `test_chunk_cache.cpp` to make the tests parameterized: to
test both mmap enabled and disabled. Added sparse field in the test to
add coverage.
* re-enabled test `Sealed::GetSparseVectorFromChunkCache`. 
* But 2 other disabled tests `Sealed::WarmupChunkCache` and
`Sealed::GetVectorFromChunkCache` remain disabled, there seems to be
errors. @bigsheeper PTAL.

---------

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-09-25 18:59:13 +08:00
zhuwenxing
58baeee8f1
test: add query with text match filter (#36381)
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Co-authored-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
2024-09-25 15:17:13 +08:00