Commit Graph

21327 Commits

Author SHA1 Message Date
Zhen Ye
3c225e5c94
fix: data race when using fields_ (#37612)
issue: #37609

Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-13 04:06:30 +08:00
XuanYang-cn
5ab3c562da
enhance: tidy compaction logs (#37595)
Some checks are pending
Code Checker / Code Checker AMD64 Ubuntu 22.04 (push) Waiting to run
Code Checker / Code Checker Amazonlinux 2023 (push) Waiting to run
Code Checker / Code Checker rockylinux8 (push) Waiting to run
Mac Code Checker / Code Checker MacOS 12 (push) Waiting to run
Build and test / Build and test AMD64 Ubuntu 22.04 (push) Waiting to run
Build and test / UT for Cpp (push) Blocked by required conditions
Build and test / UT for Go (push) Blocked by required conditions
Build and test / Integration Test (push) Blocked by required conditions
Build and test / Upload Code Coverage (push) Blocked by required conditions
Remove some annoying logs and lower a log level from warn to info

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-11-12 21:18:29 +08:00
cai.zhang
14e007d6fb
fix: Fix the bug that retrieved from wrong field for L0 segments (#37598)
issue: #37574

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-11-12 20:58:29 +08:00
yellow-shine
cef5b7fd60
enhance: [cpp-unit-test] abort previous build if it's a PR, otherwise… (#37571)
[cpp-unit-test] abort previous build if it's a PR, otherwise queue the
build

Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-11-12 19:14:29 +08:00
nico
2630717836
test: update sdk version and cases (#37507)
Some checks failed
Code Checker / Code Checker AMD64 Ubuntu 22.04 (push) Waiting to run
Code Checker / Code Checker Amazonlinux 2023 (push) Waiting to run
Code Checker / Code Checker rockylinux8 (push) Waiting to run
Mac Code Checker / Code Checker MacOS 12 (push) Waiting to run
Build and test / Build and test AMD64 Ubuntu 22.04 (push) Waiting to run
Build and test / UT for Cpp (push) Blocked by required conditions
Build and test / UT for Go (push) Blocked by required conditions
Build and test / Integration Test (push) Blocked by required conditions
Build and test / Upload Code Coverage (push) Blocked by required conditions
Publish Test Images / PyTest (push) Has been cancelled
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-11-12 17:48:28 +08:00
jaime
1e8ea4a7e7
feat: add segment/channel/task/slow query render (#37561)
issue: #36621

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-12 17:44:29 +08:00
Bingyi Sun
2742e9573f
enhance: Print results if a test fails (#37606)
test_hybrid_search_different_limit_round_decimal fails occasionally. 
Print more details for debug.

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-12 17:12:28 +08:00
wei liu
266f8ef1f5
fix: Search may return less result after qn recover (#36549)
issue: #36293 #36242
after qn recover, delegator may be loaded in new node, after all segment
has been loaded, delegator becomes serviceable. but delegator's target
version hasn't been synced, and if search/query comes, delegator will
use wrong target version to filter out a empty segment list, which
caused empty search result.

This pr will block delegator's serviceable status until target version
is synced

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-12 16:34:28 +08:00
Zhen Ye
b5b003551e
enhance: use localhost for it and ut (#37529)
issue: #37528

Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-12 11:36:27 +08:00
aoiasd
4682bba586
enhance: Support gc with bm25 stats logs (#37408)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-12 11:30:28 +08:00
congqixia
24c6a4bb29
enhance: [GoSDK] support Doc-in-doc-out APIs (#37590)
Related to #35853

This PR contains following changes:

- Add function and related proto and helper functions
- Remove the insert column missing check and leave it to server
- Add text as search input data
- Add some unit tests for logic above

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-12 10:58:28 +08:00
XuanYang-cn
a45a288a25
fix: Separate L0 and Mix trigger interval (#37190)
See also: #37108

- Add MixCompactionTriggerInterval, default 60s
- Add L0CompactionTriggerInterval, default 10s
- Export Single related compaction configs
- Raise SingleCompactionDeltaLogMaxSize from 2MB to 16MB

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-11-12 10:56:37 +08:00
wei liu
2a4c00de9d
enhance: Decouple shard client manager from shard cache (#37371)
issue: #37115
the old implementation update shard cache and shard client manager at
same time, which causes lots of conor case due to concurrent issue
without lock.

This PR decouple shard client manager from shard cache, so only shard
cache will be updated if delegator changes. and make sure shard client
manager will always return the right client, and create a new client if
not exist. in case of client leak, shard client manager will purge
client in async for every 10 minutes.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-12 10:30:28 +08:00
congqixia
f5b06a3c9f
enhance: Invalidate collection cache when release collection (#37577)
Related to #37395

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-12 10:16:29 +08:00
Chun Han
2d29dcd30c
enhance:refine group_strict_size parameter(#37482) (#37483)
Some checks are pending
Code Checker / Code Checker AMD64 Ubuntu 22.04 (push) Waiting to run
Code Checker / Code Checker Amazonlinux 2023 (push) Waiting to run
Code Checker / Code Checker rockylinux8 (push) Waiting to run
Mac Code Checker / Code Checker MacOS 12 (push) Waiting to run
Build and test / Build and test AMD64 Ubuntu 22.04 (push) Waiting to run
Build and test / UT for Cpp (push) Blocked by required conditions
Build and test / UT for Go (push) Blocked by required conditions
Build and test / Integration Test (push) Blocked by required conditions
Build and test / Upload Code Coverage (push) Blocked by required conditions
Publish Test Images / PyTest (push) Waiting to run
related: #37482

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-11-12 09:56:28 +08:00
binbin
21b68029a0
test: Add nullable test cases for bulk writer (#37572)
issue: #36129

Signed-off-by: binbin lv <binbin.lv@zilliz.com>
2024-11-12 09:46:28 +08:00
Bingyi Sun
c1eccce2fa
enhance: enable multiple chunked segment by default (#37570)
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-12 09:20:28 +08:00
foxspy
81141bd18d
fix: use yaml.v2 as yaml parser (#37423)
Some checks are pending
Code Checker / Code Checker AMD64 Ubuntu 22.04 (push) Waiting to run
Code Checker / Code Checker Amazonlinux 2023 (push) Waiting to run
Code Checker / Code Checker rockylinux8 (push) Waiting to run
Mac Code Checker / Code Checker MacOS 12 (push) Waiting to run
Build and test / Build and test AMD64 Ubuntu 22.04 (push) Waiting to run
Build and test / UT for Cpp (push) Blocked by required conditions
Build and test / UT for Go (push) Blocked by required conditions
Build and test / Integration Test (push) Blocked by required conditions
Build and test / Upload Code Coverage (push) Blocked by required conditions
issue: #34298 
Viper uses yaml.v2 as the parser. This PR will adopt the parsing logic
from Viper to handle YAML files, ensuring maximum consistency in
parsing.

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-11-11 21:26:27 +08:00
Zhen Ye
1b6edd0b4b
enhance: refactor the consumer grpc proto for reusing grpc stream for multi-consumer (#37564)
Some checks failed
Code Checker / Code Checker AMD64 Ubuntu 22.04 (push) Waiting to run
Code Checker / Code Checker Amazonlinux 2023 (push) Waiting to run
Code Checker / Code Checker rockylinux8 (push) Waiting to run
Mac Code Checker / Code Checker MacOS 12 (push) Waiting to run
Build and test / Build and test AMD64 Ubuntu 22.04 (push) Waiting to run
Build and test / UT for Cpp (push) Blocked by required conditions
Build and test / UT for Go (push) Blocked by required conditions
Build and test / Integration Test (push) Blocked by required conditions
Build and test / Upload Code Coverage (push) Blocked by required conditions
Weekly Release / Run Weekly Release (push) Has been cancelled
issue: #33285

- Modify the proto of consumer of streaming service.
- Make VChannel as a required option for streaming

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-11 17:24:29 +08:00
XuanYang-cn
5e6c3df253
fix: l0RowCount metrics value always empty (#37306)
See also: #36953

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-11-11 17:00:27 +08:00
wei liu
61a5b15ada
fix: Lost loading collection's updateTs after qc restart (#37538)
issue: #37537

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-11 14:34:28 +08:00
congqixia
83d2729f94
enhance: Generate milvusclient mockery with latest proto (#37567)
Related to #37558

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-11 14:28:30 +08:00
Ted Xu
31d0c84f67
fix: panic calclulating data size on writing binlogs (#37575)
See: #37568

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-11-11 14:08:27 +08:00
Zhen Ye
fca946dee1
enhance: move the search utilities from querynode into new module (#37531)
issue: #33285

- The search utilities will be shared between query node and streaming
node.

Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-11 11:10:27 +08:00
congqixia
5e90f348fc
enhance: Handle legacy proxy load fields request (#37565)
Related to #35415

In rolling upgrade, legacy proxy may dispatch load request wit empty
load field list. The upgraded querycoord may report error by mistake
that load field list is changed.

This PR:

- Auto field empty load field list with all user field ids
- Refine the error messag when load field list updates
- Refine load job unit test with service cases

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-11 10:14:26 +08:00
sre-ci-robot
a88fff130f Update all contributors
Some checks failed
Code Checker / Code Checker AMD64 Ubuntu 22.04 (push) Waiting to run
Code Checker / Code Checker Amazonlinux 2023 (push) Waiting to run
Code Checker / Code Checker rockylinux8 (push) Waiting to run
Mac Code Checker / Code Checker MacOS 12 (push) Waiting to run
Build and test / Build and test AMD64 Ubuntu 22.04 (push) Waiting to run
Build and test / UT for Cpp (push) Blocked by required conditions
Build and test / UT for Go (push) Blocked by required conditions
Build and test / Integration Test (push) Blocked by required conditions
Build and test / Upload Code Coverage (push) Blocked by required conditions
Publish Test Images / PyTest (push) Has been cancelled
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2024-11-11 00:01:25 +00:00
ThreadDao
f42869c0ce
test: remove case skip label (#37539)
[test] remove case skip label
issue: #33952

Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2024-11-10 16:14:27 +08:00
aoiasd
12951f0abb
enhance: rename tokenizer to analyzer and check analyzer params (#37478)
relate: https://github.com/milvus-io/milvus/issues/35853

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-10 16:12:26 +08:00
sthuang
ff00a12805
enhance: RBAC custom privilege group ut coverage (#37558)
Some checks failed
Code Checker / Code Checker AMD64 Ubuntu 22.04 (push) Waiting to run
Code Checker / Code Checker Amazonlinux 2023 (push) Waiting to run
Code Checker / Code Checker rockylinux8 (push) Waiting to run
Mac Code Checker / Code Checker MacOS 12 (push) Waiting to run
Build and test / Build and test AMD64 Ubuntu 22.04 (push) Waiting to run
Build and test / UT for Cpp (push) Blocked by required conditions
Build and test / UT for Go (push) Blocked by required conditions
Build and test / Integration Test (push) Blocked by required conditions
Build and test / Upload Code Coverage (push) Blocked by required conditions
SIMD Compatibility Test / test-simd-compatibility (avx) (push) Has been cancelled
SIMD Compatibility Test / test-simd-compatibility (avx2) (push) Has been cancelled
SIMD Compatibility Test / test-simd-compatibility (avx512) (push) Has been cancelled
SIMD Compatibility Test / test-simd-compatibility (sse4_2) (push) Has been cancelled
issue: https://github.com/milvus-io/milvus/issues/37031

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-09 20:40:25 +08:00
zhuwenxing
ffdde391e3
test: fix unstable testcase caused by server refactor (#37545)
Some checks are pending
Code Checker / Code Checker AMD64 Ubuntu 22.04 (push) Waiting to run
Code Checker / Code Checker Amazonlinux 2023 (push) Waiting to run
Code Checker / Code Checker rockylinux8 (push) Waiting to run
Mac Code Checker / Code Checker MacOS 12 (push) Waiting to run
Build and test / Build and test AMD64 Ubuntu 22.04 (push) Waiting to run
Build and test / UT for Cpp (push) Blocked by required conditions
Build and test / UT for Go (push) Blocked by required conditions
Build and test / Integration Test (push) Blocked by required conditions
Build and test / Upload Code Coverage (push) Blocked by required conditions
issue: https://github.com/milvus-io/milvus/issues/37533

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-11-09 09:58:26 +08:00
sthuang
70605cf5b3
enhance: Support custom privilege group for RBAC (#37087)
issue: #37031

---------

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-09 08:44:28 +08:00
yihao.dai
ff9bdf7029
fix: Fix load slowly (#37454)
When there're a lot of loaded collections, they would occupy the target
observer scheduler’s pool. This prevents loading collections from
updating the current target in time, slowing down the load process.
This PR adds a separate target dispatcher for loading collections.

issue: https://github.com/milvus-io/milvus/issues/37166

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-11-09 07:48:26 +08:00
XuanYang-cn
ce2fa3de01
fix: Set correct memory size (#37551)
Some checks are pending
Code Checker / Code Checker AMD64 Ubuntu 22.04 (push) Waiting to run
Code Checker / Code Checker Amazonlinux 2023 (push) Waiting to run
Code Checker / Code Checker rockylinux8 (push) Waiting to run
Mac Code Checker / Code Checker MacOS 12 (push) Waiting to run
Build and test / Build and test AMD64 Ubuntu 22.04 (push) Waiting to run
Build and test / UT for Cpp (push) Blocked by required conditions
Build and test / UT for Go (push) Blocked by required conditions
Build and test / Integration Test (push) Blocked by required conditions
Build and test / Upload Code Coverage (push) Blocked by required conditions
See also: #37549

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-11-09 01:42:26 +08:00
yihao.dai
ebc3c82761
fix: Fix subscription leak (#37382)
Some checks are pending
Code Checker / Code Checker AMD64 Ubuntu 22.04 (push) Waiting to run
Code Checker / Code Checker Amazonlinux 2023 (push) Waiting to run
Code Checker / Code Checker rockylinux8 (push) Waiting to run
Mac Code Checker / Code Checker MacOS 12 (push) Waiting to run
Build and test / Build and test AMD64 Ubuntu 22.04 (push) Waiting to run
Build and test / UT for Cpp (push) Blocked by required conditions
Build and test / UT for Go (push) Blocked by required conditions
Build and test / Integration Test (push) Blocked by required conditions
Build and test / Upload Code Coverage (push) Blocked by required conditions
Close (unsubscribe) the msg stream after completing the
`PreCreatedTopic` check to prevent backlog issue.

issue: https://github.com/milvus-io/milvus/issues/36021

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-11-08 16:16:26 +08:00
cai.zhang
ae227e3934
enhance: Add integration test for stats task (#37506)
issue: #33744

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-11-08 15:48:26 +08:00
congqixia
0f514708ff
enhance: Accelerate Build stage by skipping cpp ut compilation (#37530)
Since master branch run cpp unittest in jenken, the cpp unit test
building could be skipped since the output is not needed in github
runner.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-08 14:06:26 +08:00
congqixia
dcc1b506dc
enhance: Add context trace for querycoord queryable check (#37524)
When check health logic failed to collection not-queryable, the related
reason is hard to find in log.

This PR add context for log with trace id and print unqueryable
collection info log.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-08 14:00:26 +08:00
wei liu
a03157838b
enhance: Enable node assign policy on resource group (#36968)
issue: #36977
with node_label_filter on resource group, user can add label on
querynode with env `MILVUS_COMPONENT_LABEL`, then resource group will
prefer to accept node which match it's node_label_filter.

then querynode's can't be group by labels, and put querynodes with same
label to same resource groups.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-08 11:18:27 +08:00
SimFG
bcb6420540
enhance: enable the lock scheduler for the rootcoord (#37513)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-11-08 10:20:26 +08:00
Xiaofan
e073906a19
enhance: optimize describe collection and index (#37490)
fix #37489
combine multiple describe collection and list index into one call

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-11-08 10:18:34 +08:00
Zhen Ye
49657c4690
enhance: add create segment message, enable empty segment flush (#37407)
issue: #37172

- add redo interceptor to implement append context refresh. (make new
timetick)
- add create segment handler for flusher.
- make empty segment flushable and directly change it into dropped.
- add create segment message into wal when creating new growing segment.
- make the insert operation into following seq: createSegment -> insert
-> insert -> flushSegment.
- make manual flush into following seq: flushTs -> flushsegment ->
flushsegment -> manualflush.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-08 10:16:34 +08:00
yihao.dai
81879425e1
enhance: Optimize the performance of stats task (#37374)
1. Increase the writer's `batchSize` to avoid multiple serialization
operations.
2. Perform asynchronous upload of binlog files to prevent blocking the
data processing flow.
3. Reduce multiple calls to `writer.Flush()`.

issue: https://github.com/milvus-io/milvus/issues/37373

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-11-08 10:08:27 +08:00
Ted Xu
bc9562feb1
enhance: avoid memory copy and serde in mix compaction (#37479)
Some checks are pending
Code Checker / Code Checker AMD64 Ubuntu 22.04 (push) Waiting to run
Code Checker / Code Checker Amazonlinux 2023 (push) Waiting to run
Code Checker / Code Checker rockylinux8 (push) Waiting to run
Mac Code Checker / Code Checker MacOS 12 (push) Waiting to run
Build and test / Build and test AMD64 Ubuntu 22.04 (push) Waiting to run
Build and test / UT for Cpp (push) Blocked by required conditions
Build and test / UT for Go (push) Blocked by required conditions
Build and test / Integration Test (push) Blocked by required conditions
Build and test / Upload Code Coverage (push) Blocked by required conditions
See: #37234

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-11-07 16:30:57 -08:00
sre-ci-robot
1f7ce9e9c1 Update all contributors
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2024-11-08 00:02:00 +00:00
congqixia
11f4fe0177
enhance: [GoSDK] move client pkg go files to sub one (#37492)
Related to #31293

Client source files under client pkg cannot be evaluate correctly by
codecov. This PR moves them to `milvusclient` sub-package to fix this
issue and follow go major version best practice.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-08 07:32:26 +08:00
Bingyi Sun
40ba5a3414
fix: fix chunked segment term filter expression and add ut (#37392)
issue: https://github.com/milvus-io/milvus/issues/37143

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-07 11:04:19 -08:00
congqixia
5310d3469f
fix: Escape brace of dumped JSON for index err message (#37504)
Some checks are pending
Code Checker / Code Checker AMD64 Ubuntu 22.04 (push) Waiting to run
Code Checker / Code Checker Amazonlinux 2023 (push) Waiting to run
Code Checker / Code Checker rockylinux8 (push) Waiting to run
Mac Code Checker / Code Checker MacOS 12 (push) Waiting to run
Build and test / Build and test AMD64 Ubuntu 22.04 (push) Waiting to run
Build and test / UT for Cpp (push) Blocked by required conditions
Build and test / UT for Go (push) Blocked by required conditions
Build and test / Integration Test (push) Blocked by required conditions
Build and test / Upload Code Coverage (push) Blocked by required conditions
Related to #37503

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-07 18:58:25 +08:00
Bingyi Sun
7ec449fee5
enhance: regenerate datacoord mockery (#37501)
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-07 17:38:24 +08:00
yihao.dai
36bb87d9b6
fix: Fix large growing segment (#37388)
Consider the `sealProportion` factor during segment allocation.

issue: https://github.com/milvus-io/milvus/issues/37387

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-11-07 17:12:25 +08:00
yihao.dai
994f52fab8
fix: Revert "enhance: Support db for bulkinsert (#37012)" (#37420)
This reverts commit 6e90f9e8d9.

issue: https://github.com/milvus-io/milvus/issues/31273

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-11-07 17:02:25 +08:00