Commit Graph

20677 Commits

Author SHA1 Message Date
wei liu
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 (#34394)
issue: #34252

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-29 11:31:51 +08:00
Chun Han
e2e38e98df
fix: nil part stats without l2 compaction(#34923) (#34992)
related: #34923

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-29 11:07:48 +08:00
XuanYang-cn
9a07790215
enhance: Add ut for l0CompactionTask processExecuting (#34800)
See also: #34796

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-29 10:53:48 +08:00
yihao.dai
4a3d98d88c
enhance: Fast close msgdispatcher target (#34803)
/kind improvement

issue: https://github.com/milvus-io/milvus/issues/34075

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-29 00:25:49 +08:00
zhagnlu
f77f5364b2
fix: disable use_index when some array expr (#34894)
#34797

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-29 00:17:46 +08:00
PowderLi
5584ae2e14
enhance: add metrics for restful apis (#34969)
issue: #34968

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-07-29 00:13:47 +08:00
cai.zhang
497afcb897
enhance: Refine code for GetRecoveryInfo (#34973)
issue: #34495

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-29 00:01:46 +08:00
cai.zhang
2372452fac
enhance: Optimized the GC logic to ensure that memory is released in time (#34949)
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-28 23:53:47 +08:00
jaime
08fa51d4f4
fix: memory leak while parsing query plan (#34931)
issue: #34930

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-28 21:50:20 +08:00
Aldrin
9463eeef2b
fix: Avoided dereferencing NULL pointer (#34836)
issue : https://github.com/milvus-io/milvus/issues/34835

Signed-off-by: Ald392 <imagesai32@gmail.com>
2024-07-27 17:27:52 +08:00
zhagnlu
804ec24c02
fix:fix retrieve raw data from bitmap array index (#34848)
#34795

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-27 01:53:47 +08:00
sre-ci-robot
e93687cdd3
[automated] Update Knowhere Commit (#35039)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-07-27 00:59:46 +08:00
congqixia
b86d795d6d
enhance: [GoSDK]Support specifying field schema to be cluster key (#35027)
See also #32939 milvus-io/milvus-sdk-go#794

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-26 20:01:45 +08:00
congqixia
783f9d9c33
fix: Unify hook singleton implementation in proxy (#34887)
Related to #34885

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-26 18:07:53 +08:00
wei liu
6e9fbd1630
enhance: Solve channel unbalance on datanode (#34984)
issue: #33583
the old policy permit datanode has at most 2 more channels than other
datanode. so if milvus has 2 datanode and 2 channels, both 2 channels
will be assign to 1 datanode, left another datanode empty.

This PR refine the balance policy to solve channel unbalance on datanode

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-26 17:35:47 +08:00
congqixia
94d8f48b85
enhance: Use auto detection proto interface to fetch grpc size (#35024)
Related to #34763

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-26 14:23:47 +08:00
cai.zhang
ea13b97b6a
fix: Fix bug for block clustering compaction (#35019)
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-26 11:33:58 +08:00
chyezh
1cff55381d
enhance: add manual alloc segment rpc for datacoord (#35002)
issue: #33285

- segment allocation will move to streamingnode, so a manual alloc
segment rpc is required

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-26 10:15:46 +08:00
congqixia
4ee6c69217
enhance: Add Segment Level in milvus segment info APIs (#34763)
See also #34746

This PR add segment level field in response of
`GetPersistentSegmentInfo` and `GetQuerySegmentInfo`

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-26 10:01:46 +08:00
congqixia
e8038a76e1
fix: Remove timeout in datanode watch ctx (#35011)
See also #35008

Use tickle timeout logic instead of hardcode context timeout

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-25 22:17:45 +08:00
Ted Xu
63f0154dfb
fix: enable milvus.yaml check (#34567)
See #32168

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-07-25 18:05:46 +08:00
edward.zeng
95c4600958
enhance: Refine milvus dashboard for milvus 2.3 and 2.4 (#34994)
/kind improvement
cc @yanliang567 
issue: https://github.com/milvus-io/milvus/issues/34476

Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
2024-07-25 15:41:46 +08:00
Chun Han
c46c401112
fix: refine handling type for segment pruner(#34923) (#34925)
related: #34923

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-25 13:57:45 +08:00
chyezh
d93c51d5f6
enhance: change GetVChannels rpc into GetPChannelInfo (#34940)
issue: #33285

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-25 13:29:45 +08:00
chyezh
bef06e5acf
enhance: add streaming coord and node grpc service register (#34655)
issue: #33285

- register streaming coord service into datacoord.
- add new streaming node role.
- add global static switch to enable streaming service or not.

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-25 12:19:44 +08:00
chyezh
4f6cbfd520
enhance: specialized immutable and mutable message (#34951)
issue: #33285

- add specialized mutable and immutable message, make type safe.
- add version based constructor and type.

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-25 11:57:45 +08:00
Jiquan Long
b843c91bad
feat: record the number of times milvus turns into force-deny-writing states (#34960)
/kind improvement
fix: #34961

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-07-25 11:39:46 +08:00
congqixia
e2f40fc2a8
fix: Check legacy guarantee ts when skipping alloc ts (#34981)
See also #34980

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-25 10:17:45 +08:00
smellthemoon
7ec9d856f3
fix: access address was not malloc (#34971)
issue: #34972  
fix string type data use memcpy to fill cause segv for not malloc enough
memory in advance.

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-07-25 02:31:44 +08:00
wei liu
166fc902b0
enhance: Limit collection's normal balance speed (#34810)
issue: #34798

after we remove the task priority on query coord, to avoid load/release
segment blocked by too much balance task, we limit the balance task size
in each round. at same time, we reduce the balance interval to trigger
balance more frequently.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-24 19:11:44 +08:00
AnthonyTsu1984
c5da4fa094
doc: Update the description about hybrid search in README.md (#34941)
Signed-off-by: AnthonyTsu1984 <115786031+AnthonyTsu1984@users.noreply.github.com>
2024-07-24 10:17:44 +08:00
sre-ci-robot
7464ae0f2b Update all contributors
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2024-07-23 12:00:54 +00:00
congqixia
482373b3d3
enhance: Sync go e2e package proto dependency (#34913)
Go e2e package point to a lower verion of milvus-proto and may cause
compile error locally.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-23 16:09:43 +08:00
smellthemoon
5616b7e8d2
enhance: support null in c data_datacodec and load null value (#32183)
1. support read and write null in segcore
    will store valid_data(use uint8_t type to save memory) in fieldData.
2. support load null
binlog reader read and write data into column(sealed segment),
insertRecord(growing segment). In sealed segment, store valid_data
directly. In growing segment, considering prior implementation and easy
code reading, it covert uint8_t to fbvector<bool>, which may optimize in
future.
3.  retrieve valid_data.
    parse valid_data in search/query.
#31728

---------

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-07-23 16:07:51 +08:00
wei liu
92de49e38c
fix: Segment may bounce between delegator and worker (#34830)
issue: #34595

pr#34596 to we add an overloaded factor to segment in delegator, which
cause same segment got different score in delegator and worker. which
may cause segment bounce between delegator and worker.

This PR use average score to compute the delegator overloaded factor, to
avoid segment bounce between delegator and worker.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-23 15:57:49 +08:00
David Pichler
a4d7da4973
feat: add ipv6 support (#32960)
related with #34917
This PR updates the func utils get IP address function to support
getting a IPv6 address if it is available and return it. I've tested
that this works locally when running inside a docker container that has
an IPv6 network address.

This PR should address
https://github.com/milvus-io/milvus/discussions/20217

Thank you for review!

Signed-off-by: David Pichler <david.pichler@grainger.com>
Co-authored-by: David Pichler <david.pichler@grainger.com>
2024-07-23 15:39:43 +08:00
chyezh
f6c6e98ec5
enhance: recover stack info when non-cgo thread crash (#34865)
issue: #34864

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-23 14:25:44 +08:00
Jiquan Long
a2ac84bd64
feat: record the duration waiting in the proxy queue (#34744)
fix: https://github.com/milvus-io/milvus/issues/34743

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-07-23 14:23:52 +08:00
Min Tian
a4aed9b0b5
enhance: new knowhere param for range_search (#34686)
issue: #34685 
knowhere needs a new json param `range_search_k` for RangeSearch to
early terminate the iterator.

Signed-off-by: min.tian <min.tian.cn@gmail.com>
2024-07-23 11:45:43 +08:00
yellow-shine
1f55061717
feat: [skip e2e] effective way to use cache (#34774)
issue: https://github.com/milvus-io/milvus/issues/34876
1. try best to download cache
2. upload cache if it is not pull request

Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-07-23 10:49:51 +08:00
cai.zhang
260a6e2ba9
enhance: Send flush signal when the water level reaches the high watermark (#34907)
issue: #30633

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-23 10:29:42 +08:00
cai.zhang
575ce91039
fix: Get current index version from knowhere before building index (#34901)
issue: #34900

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-23 10:23:42 +08:00
cai.zhang
4c45bc412f
enhance: Add integration test for clustering compaction (#34881)
issue: #34792

---------

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-23 10:13:43 +08:00
cai.zhang
643b9d521c
fix: Fix the issue of concurrent packing of the same segment (#34840)
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-23 10:11:49 +08:00
sre-ci-robot
12b49859c0
[automated] Update Knowhere Commit (#34909)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-07-23 00:53:43 +08:00
jaime
3cd24f7548
fix: collection meta is not removed after gc in DataCoord (#34883)
issue: #34847

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-22 21:11:48 +08:00
chyezh
39c7e06bc5
enhance: add message and msgstream msgpack adaptor (#34874)
issue: #33285

- make message builder and message conversion type safe
- add adaptor type and function to adapt old msgstream msgpack and
message interface

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-22 20:59:42 +08:00
Chun Han
6c19f9baf8
enhance: optimize search reduce perf(#32507) (#34607)
related: #32507

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-22 20:57:43 +08:00
wei liu
40e39ef7c9
fix: Avoid segment lack caused by deduplicate segment task (#34782)
issue: #34781

when balance segment hasn't finished yet, query coord may found 2 loaded
copy of segment, then it will generate task to deduplicate, which may
cancel the balance task. then the old copy has been released, and the
new copy hasn't be ready yet but canceled, then search failed by segment
lack.

this PR set deduplicate segment task's proirity to low, to avoid balance
segment task canceled by deduplicate task.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-22 16:35:43 +08:00
wayblink
52fa668f5d
fix: fix collection is compacting logic (#34855)
fix: #34861
also related to: #34792

---------

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-22 11:35:41 +08:00