Commit Graph

20507 Commits

Author SHA1 Message Date
Ted Xu
63f0154dfb
fix: enable milvus.yaml check (#34567)
See #32168

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-07-25 18:05:46 +08:00
edward.zeng
95c4600958
enhance: Refine milvus dashboard for milvus 2.3 and 2.4 (#34994)
/kind improvement
cc @yanliang567 
issue: https://github.com/milvus-io/milvus/issues/34476

Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
2024-07-25 15:41:46 +08:00
Chun Han
c46c401112
fix: refine handling type for segment pruner(#34923) (#34925)
related: #34923

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-25 13:57:45 +08:00
chyezh
d93c51d5f6
enhance: change GetVChannels rpc into GetPChannelInfo (#34940)
issue: #33285

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-25 13:29:45 +08:00
chyezh
bef06e5acf
enhance: add streaming coord and node grpc service register (#34655)
issue: #33285

- register streaming coord service into datacoord.
- add new streaming node role.
- add global static switch to enable streaming service or not.

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-25 12:19:44 +08:00
chyezh
4f6cbfd520
enhance: specialized immutable and mutable message (#34951)
issue: #33285

- add specialized mutable and immutable message, make type safe.
- add version based constructor and type.

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-25 11:57:45 +08:00
Jiquan Long
b843c91bad
feat: record the number of times milvus turns into force-deny-writing states (#34960)
/kind improvement
fix: #34961

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-07-25 11:39:46 +08:00
congqixia
e2f40fc2a8
fix: Check legacy guarantee ts when skipping alloc ts (#34981)
See also #34980

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-25 10:17:45 +08:00
smellthemoon
7ec9d856f3
fix: access address was not malloc (#34971)
issue: #34972  
fix string type data use memcpy to fill cause segv for not malloc enough
memory in advance.

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-07-25 02:31:44 +08:00
wei liu
166fc902b0
enhance: Limit collection's normal balance speed (#34810)
issue: #34798

after we remove the task priority on query coord, to avoid load/release
segment blocked by too much balance task, we limit the balance task size
in each round. at same time, we reduce the balance interval to trigger
balance more frequently.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-24 19:11:44 +08:00
AnthonyTsu1984
c5da4fa094
doc: Update the description about hybrid search in README.md (#34941)
Signed-off-by: AnthonyTsu1984 <115786031+AnthonyTsu1984@users.noreply.github.com>
2024-07-24 10:17:44 +08:00
sre-ci-robot
7464ae0f2b Update all contributors
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2024-07-23 12:00:54 +00:00
congqixia
482373b3d3
enhance: Sync go e2e package proto dependency (#34913)
Go e2e package point to a lower verion of milvus-proto and may cause
compile error locally.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-23 16:09:43 +08:00
smellthemoon
5616b7e8d2
enhance: support null in c data_datacodec and load null value (#32183)
1. support read and write null in segcore
    will store valid_data(use uint8_t type to save memory) in fieldData.
2. support load null
binlog reader read and write data into column(sealed segment),
insertRecord(growing segment). In sealed segment, store valid_data
directly. In growing segment, considering prior implementation and easy
code reading, it covert uint8_t to fbvector<bool>, which may optimize in
future.
3.  retrieve valid_data.
    parse valid_data in search/query.
#31728

---------

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-07-23 16:07:51 +08:00
wei liu
92de49e38c
fix: Segment may bounce between delegator and worker (#34830)
issue: #34595

pr#34596 to we add an overloaded factor to segment in delegator, which
cause same segment got different score in delegator and worker. which
may cause segment bounce between delegator and worker.

This PR use average score to compute the delegator overloaded factor, to
avoid segment bounce between delegator and worker.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-23 15:57:49 +08:00
David Pichler
a4d7da4973
feat: add ipv6 support (#32960)
related with #34917
This PR updates the func utils get IP address function to support
getting a IPv6 address if it is available and return it. I've tested
that this works locally when running inside a docker container that has
an IPv6 network address.

This PR should address
https://github.com/milvus-io/milvus/discussions/20217

Thank you for review!

Signed-off-by: David Pichler <david.pichler@grainger.com>
Co-authored-by: David Pichler <david.pichler@grainger.com>
2024-07-23 15:39:43 +08:00
chyezh
f6c6e98ec5
enhance: recover stack info when non-cgo thread crash (#34865)
issue: #34864

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-23 14:25:44 +08:00
Jiquan Long
a2ac84bd64
feat: record the duration waiting in the proxy queue (#34744)
fix: https://github.com/milvus-io/milvus/issues/34743

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-07-23 14:23:52 +08:00
Min Tian
a4aed9b0b5
enhance: new knowhere param for range_search (#34686)
issue: #34685 
knowhere needs a new json param `range_search_k` for RangeSearch to
early terminate the iterator.

Signed-off-by: min.tian <min.tian.cn@gmail.com>
2024-07-23 11:45:43 +08:00
yellow-shine
1f55061717
feat: [skip e2e] effective way to use cache (#34774)
issue: https://github.com/milvus-io/milvus/issues/34876
1. try best to download cache
2. upload cache if it is not pull request

Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-07-23 10:49:51 +08:00
cai.zhang
260a6e2ba9
enhance: Send flush signal when the water level reaches the high watermark (#34907)
issue: #30633

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-23 10:29:42 +08:00
cai.zhang
575ce91039
fix: Get current index version from knowhere before building index (#34901)
issue: #34900

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-23 10:23:42 +08:00
cai.zhang
4c45bc412f
enhance: Add integration test for clustering compaction (#34881)
issue: #34792

---------

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-23 10:13:43 +08:00
cai.zhang
643b9d521c
fix: Fix the issue of concurrent packing of the same segment (#34840)
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-23 10:11:49 +08:00
sre-ci-robot
12b49859c0
[automated] Update Knowhere Commit (#34909)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-07-23 00:53:43 +08:00
jaime
3cd24f7548
fix: collection meta is not removed after gc in DataCoord (#34883)
issue: #34847

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-22 21:11:48 +08:00
chyezh
39c7e06bc5
enhance: add message and msgstream msgpack adaptor (#34874)
issue: #33285

- make message builder and message conversion type safe
- add adaptor type and function to adapt old msgstream msgpack and
message interface

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-22 20:59:42 +08:00
Chun Han
6c19f9baf8
enhance: optimize search reduce perf(#32507) (#34607)
related: #32507

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-22 20:57:43 +08:00
wei liu
40e39ef7c9
fix: Avoid segment lack caused by deduplicate segment task (#34782)
issue: #34781

when balance segment hasn't finished yet, query coord may found 2 loaded
copy of segment, then it will generate task to deduplicate, which may
cancel the balance task. then the old copy has been released, and the
new copy hasn't be ready yet but canceled, then search failed by segment
lack.

this PR set deduplicate segment task's proirity to low, to avoid balance
segment task canceled by deduplicate task.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-22 16:35:43 +08:00
wayblink
52fa668f5d
fix: fix collection is compacting logic (#34855)
fix: #34861
also related to: #34792

---------

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-22 11:35:41 +08:00
yihao.dai
8aab6cbfac
enhance: Organize the common modules of streamingNode and dataNode (#34773)
1. Move the common modules of streamingNode and dataNode to flushcommon
2. Add new GetVChannels interface for rootcoord

issue: https://github.com/milvus-io/milvus/issues/33285

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-22 11:33:51 +08:00
chyezh
f4de99e129
enhance: implement streaming coord client (#34654)
issue: #33285

- add streaming coord channel assignment watch client

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-22 11:32:04 +08:00
Patrick Weizhi Xu
80c0ae3519
feat: [restful v2] add partition key isolation prop (#34710)
issue: https://github.com/milvus-io/milvus/issues/34332

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-07-22 11:29:59 +08:00
wayblink
c339df26fc
enhance: refine clustering compaction basic it (#34793)
#34792

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-22 11:27:51 +08:00
wayblink
d294fdd060
enhance: add ut for clustering_compactor (#34852)
issue: #34792

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-21 19:29:42 +08:00
wayblink
c2b8b5fe84
enhance: refine clustering compaction configs and logs (#34784)
#30633

---------

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-21 19:23:40 +08:00
SimFG
88c38a3756
fix: execute method should return an error when result is a failure (#34813)
- issue: #34812

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-20 23:13:40 +08:00
SimFG
16c002b51b
enhance: unified channel name generation and conversion (#34805)
/kind improvement
- issue: #34804

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-20 23:11:39 +08:00
yihao.dai
b22e549844
enhance: Rename config of sealing by growing segmetns size (#34787)
/kind improvement

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-19 20:27:41 +08:00
Xiaofan
7eaef09ba5
enhance: enable profile and block profiling (#34642)
fix #34641

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-07-19 18:43:41 +08:00
wayblink
a11714913b
fix: fix unstable ut of compaction pickSlot (#34828)
#34827

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-19 17:53:47 +08:00
zhagnlu
fbad77f22a
fix: fix ut test fail (#34832)
#34831

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-19 17:51:40 +08:00
chyezh
86eff6e589
enhance: streaming node client implementation (#34653)
issue: #33285

- add streaming node grpc client wrapper
- add unittest for streaming node grpc client side
- fix binary unsafe bug for message

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-19 17:37:40 +08:00
Chun Han
ed057e6fce
fix: non-init seg_offset for growing raw-data when doing groupby (#34748)
related:  #34713

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-19 17:01:40 +08:00
congqixia
e4e18cb8c3
enhance: Add dynamic cgo pool for proxy CGO call (#34768)
Related to #34705

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-19 15:11:41 +08:00
congqixia
2ac7164c39
enhance: Remove useless ops when there is no write (#34767)
Related to #33235

THe querynode pipeline will make map & call ProcessInsert when there is
no write messages. So querynodes will have high CPU usage even when
there is no workload.

This PR check msg length before composing data struct and calling method

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-19 14:31:42 +08:00
foxspy
8e64bf929c
enhance: add scalar filtering and vector search latency metrics (#34785)
add scalar filtering and vector search latency metrics to distinguish
the cost of scalar filtering.
To add metrics in query chain, add a monitor module and move the metric
files from original storage module.
issue: #34780

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-07-19 14:01:39 +08:00
sre-ci-robot
c61592dcea
[automated] Update Knowhere Commit (#34819)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-07-19 02:09:42 +08:00
yihao.dai
c8bf6c8a46
enhance: Optimize ConsistencyLevel and GuaranteeTimestamp log (#34738)
/kind improvement

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-18 15:17:41 +08:00
zhagnlu
f1b2f7b640
enhance: refactor bitmap index and internal hybrid index (#34450)
#32900

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-18 10:39:42 +08:00