Commit Graph

20987 Commits

Author SHA1 Message Date
SimFG
9c1772f659
enhance: avoid to create many timer object in the target (#36570)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-29 19:23:16 +08:00
Zhen Ye
a47abb2f2b
fix: stats log lost after disable stats log loading on flush (#36592)
issue: #36555

Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-29 16:53:15 +08:00
Zhen Ye
a6545b2e29
fix: refactor milvus config and change default txn timeout (#36522)
issue: #36498

Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-29 11:01:15 +08:00
nico
c43d94319f
enhance: [skip e2e]open querynode mmap for all pipeline (#36586)
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-09-29 09:41:15 +08:00
binbin
d1d5a50014
test: add null and default test cases (#36539)
issue: #36129

Signed-off-by: binbin lv <binbin.lv@zilliz.com>
2024-09-29 09:39:22 +08:00
yihao.dai
8ed34dce84
enhance: Reopen chunk cache cpp ut (#33622)
issue: https://github.com/milvus-io/milvus/issues/33210

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-28 18:19:15 +08:00
zhagnlu
9e3efa06be
fix:fix empty search result bug (#36582)
#36450

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-28 17:45:16 +08:00
yihao.dai
80f25d497f
enhance: Add metrics to monitor import throughput and imported rows (#36519)
issue: https://github.com/milvus-io/milvus/issues/36518

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-28 17:31:15 +08:00
cai.zhang
acc9b5ada5
fix: Regenerate partition stats after stats task for clustering compaction (#36408)
issue: #33744

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-28 17:29:21 +08:00
cai.zhang
7bf40694fd
fix: Record the nodeID before assigning tasks (#36371)
issue: #33744

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-28 17:21:15 +08:00
cai.zhang
2adca8b754
fix: Fix data race for cluerting compaction (#36440)
issue: #36438

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-28 17:19:21 +08:00
zhuwenxing
31353ae406
test: add restful cases for text match feature (#36405)
/kind improvement

Since creating a collection with text match is not yet implemented on
the RESTful interface, we will temporarily use pymilvus to create a
collection for now. This PR includes a case to test using text match
filters in search queries through the RESTful interface.

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-09-28 16:51:15 +08:00
Zhen Ye
bd7910632a
fix: etcd session discoverer cannot recover from ErrCompacted (#36568)
issue: #36554

Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-28 10:51:14 +08:00
aoiasd
ffc12fb5c4
fix: split delete task msg to MaxMessageSize to avoid mq message too large error (#36197)
relate: https://github.com/milvus-io/milvus/issues/36089

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-27 18:15:19 +08:00
smellthemoon
b1ac3f0df0
enhance: do not log out the full req (#36546)
![img_v3_02f3_61ac166c-df6f-4b5e-b06a-756e5f40a65g](https://github.com/user-attachments/assets/df668339-d0d8-4295-a2b6-52d89ad98766)

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-27 16:39:15 +08:00
wt
1f271e39c6
test: add partition key & bitmap test case (#36564)
Signed-off-by: wangting0128 <ting.wang@zilliz.com>
2024-09-27 14:17:14 +08:00
yanliang567
e6c7fd6605
test: Add group search test on sparse vectors (#36562)
related issue: #36295
add one test for alter alias

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2024-09-27 14:15:14 +08:00
zhuwenxing
7c2cb8c5d4
test: add bulk insert case for text match feature (#36398)
/kind improvement

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-09-27 10:01:14 +08:00
wt
ff4c62e44f
test: add more scalar filter expressions (#36545)
Signed-off-by: wangting0128 <ting.wang@zilliz.com>
2024-09-27 09:59:14 +08:00
zhuwenxing
9444329da1
test: relax the checks on range search (#36542)
/kind improvement

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-09-27 09:57:13 +08:00
cai.zhang
50905e0b45
fix: Check string array max length after type matching (#36449)
issue: #36029

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-27 09:45:14 +08:00
Zhen Ye
d29e01e284
fix: port listen racing in mix or standalone mode (#36442)
issue: #36441

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-26 21:23:16 +08:00
wayblink
7ff41697f9
enhance: make slotUsage a field of CompactionTask (#36510)
#36509

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-09-26 20:19:14 +08:00
wei liu
c056620899
fix: Skip unnecessary query node health check in proxy (#36491)
issue: #36490
After the query node changes from a delegator to a worker, proxy should
skip this querynode's health check.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-26 18:13:15 +08:00
wei liu
55be814a58
enhance: make TransferChannel/TransferSegment idempotent (#36489)
issue: #36488
when call TransferChannel/TransferSegment, querycoord will generate and
submit balance task to scheduler, if segment/channel's task already
exist in scheduler, submit task will failed.

to make TransferChannel/TransferSegment idempotent, we should skip to
submit if task already exist in scheduler.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-26 18:11:23 +08:00
wei liu
5dfa1c3397
fix: Segment unbalance after many times load/release (#36537)
issue: #36536
query coord use `segmentTaskDeleta/channelTaskDelta` to measure the
executing workload for querynode in scheduler, and we maintains the
`segmentTaskDeleta/channelTaskDelta` by `scheulder.Add(task)` and
`scheduler.remove(task)`, but `scheduler.remove(task)` has been called
in unexpected way, which cause a wrong
`segmentTaskDeleta/channelTaskDelta` value and affect the segment assign
logic, causes segment unbalance.

This PR moves to compute the `segmentTaskDeleta/channelTaskDelta` when
access, to avoid the wrong value affect.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-26 15:13:15 +08:00
zhagnlu
0799d927c6
fix:fix term expr overflow bug (#36525)
#36520

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-09-26 15:01:14 +08:00
smellthemoon
b60164b882
enhance: support null in bulk insert of binlog to help backup null (#36526)
https://github.com/milvus-io/milvus/issues/36341

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-26 14:35:14 +08:00
SimFG
c94b69c2f6
enhance: update the expr version and format the expr http response (#36406)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-26 14:27:20 +08:00
sthuang
4493aa2142
fix: querycoord collection num metric (#36471)
related to: #36456

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-09-26 14:23:13 +08:00
zhuwenxing
954d8a5f68
test: add search test cases with text match (#36399)
/kind improvement

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-09-26 10:15:14 +08:00
congqixia
4fd9b0a8e3
enhance: Return segment id hint in QueryStream response (#36487)
Related to #36482

This PR reuses `SealedSegmentIDsRetrieved` field in `RetrieveResults`
struct to store segment id hint.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-26 10:13:14 +08:00
yihao.dai
9e8cafcbe2
enhance: Skip loading bf in datanode (#36367)
Skip loading bf in datanode:
1. When watching vchannels, skip loading bloom filters for segments.
2. Bypass bloom filter checks for delete messages, directly writing to
L0 segments.
3. Remove flushed segments proactively after flush.

issue: https://github.com/milvus-io/milvus/issues/34585

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-26 10:11:15 +08:00
smellthemoon
a8c80abe36
enhance: upgrade pulsar-client-go to 0.11.1 (#36435)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-26 10:09:13 +08:00
sre-ci-robot
1970ab4c34
[automated] Update Pytest image changes (#36496)
Update Pytest image changes
See changes:
58baeee8f1
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-09-26 10:05:13 +08:00
nico
cfd636ed5b
test: add different language tests and modify some cases (#36465)
fix: #36396

Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-09-26 09:21:13 +08:00
sre-ci-robot
447e326629
[automated] Update Knowhere Commit (#36527)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-09-26 01:15:13 +08:00
wt
14be89df29
test: add case to enable offset cache and mmap at the same time (#36505)
Signed-off-by: wangting0128 <ting.wang@zilliz.com>
2024-09-25 19:59:13 +08:00
Buqian Zheng
8495bc6bbc
fix: fix broken Sparse Float Vector raw data mmap (#36183)
issue: https://github.com/milvus-io/milvus/issues/36182

* improved `Column.h` to make the code much more readable and
maintainable, and added detailed comments.
* fixed an issue where `ArrayColumn::NumRows()` always returns 0 when
the mmap backing storage is a file.
* removed unused `ColumnBase` constructors and unnecessary members so we
don't get confused.
* Updated `test_chunk_cache.cpp` to make the tests parameterized: to
test both mmap enabled and disabled. Added sparse field in the test to
add coverage.
* re-enabled test `Sealed::GetSparseVectorFromChunkCache`. 
* But 2 other disabled tests `Sealed::WarmupChunkCache` and
`Sealed::GetVectorFromChunkCache` remain disabled, there seems to be
errors. @bigsheeper PTAL.

---------

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-09-25 18:59:13 +08:00
zhuwenxing
58baeee8f1
test: add query with text match filter (#36381)
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Co-authored-by: yanliang567 <82361606+yanliang567@users.noreply.github.com>
2024-09-25 15:17:13 +08:00
zhuwenxing
aee046e52c
test: add some restful negative case to ci (#36479)
/kind improvement

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-09-25 10:19:13 +08:00
wei liu
3cd0b26285
enhance: Enable dynamic update loaded collection's replica (#35822)
issue: #35821
After collection loaded, if we need to increase/decrease collection's
replica, we need to release and load it again.

milvus offers 4 solution to update loaded collection's replica, this PR
aims to dynamic change the replica number without release, and after
replica number changed, milvus will execute load replica or release
replica in async, and the replica loaded status can be checked by
getReplicas API.

Notice that if set too much replicas than querynode can afford,the new
replica won't be loaded successfully until enough querynode joins.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-25 10:13:18 +08:00
sre-ci-robot
da2d3ce763
[automated] Update Pytest image changes (#36463)
Update Pytest image changes
See changes:
d55d9d6e1d
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-09-24 19:49:12 +08:00
congqixia
ed95568a05
enhance: Fix PR conflict in reduce unit test (#36470)
Related to #36433 #36180

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-24 18:01:13 +08:00
wei liu
3bd7ec8751
fix: Fix cornor case that segment can't be move out from stopping node (#36431)
issue: #36426
the old constriant requires only segment on current target can be
balanced, which is wrong, and caused that segment can't be move out from
stopping node, if it's only exist in next target.

by design, stopping balance need to move out all segment on it by
balance task, thus the unfair old constriant should be removed.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-09-24 17:01:14 +08:00
jaime
52cce4de58
fix: iaccurate size estimation for encoded array data (#36373)
issue: #36029

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-09-24 14:51:14 +08:00
Zhen Ye
350dde666d
fix: streaming node dead lock (#36403)
issue: #36388

- fix dead lock.
- fix barrier timetick failure.

Signed-off-by: chyezh <chyezh@outlook.com>
2024-09-24 14:33:13 +08:00
Chun Han
d55d9d6e1d
fix: change pymilvus version for hybridsearch-groupby(#36407) (#36451)
related: #36407

---------

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-24 14:29:13 +08:00
congqixia
98a917c5d4
enhance: [skip e2e] Add unittest for reducing duplicated pk from multi segments (#36433)
Related to #35505 #36362

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-24 14:11:13 +08:00
yihao.dai
8cda48a96a
enhance: Use mmap.scalarIndex config for text index (#36400)
issue: https://github.com/milvus-io/milvus/issues/35273

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-24 12:21:13 +08:00