Commit Graph

20023 Commits

Author SHA1 Message Date
congqixia
b90999b741
fix: Compare config value then swap when caching param value (#33785)
See also #33784

This PR change the behavior of `SetCacheValue` of config manager:

- Use mutex and map instead of concurrent map for `configCache`
- Compare config raw value before set cache value

With this implementation, concurrent caching & eviction shall always
have current output:

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |get     |        |old value|null     |
|t1  |CAS OK  |        |old value|old value|
|t2  |        |update  |new value|old value|
|t3  |        |eviction|new value|null     |

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |get     |        |old value|null     |
|t1  |        |update  |new value|null     |
|t2  |CAS fail|        |old value|null     |
|t3  |        |eviction|new value|null     |

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |        |update  |new value|null     |
|t1  |get     |        |new value|null     |
|t2  |CAS OK  |        |new value|new value|
|t3  |        |eviction|new value|null     |

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |        |update  |new value|null     |
|t1  |get     |        |new value|null     |
|t2  |        |eviction|new value|null     |
|t3  |CAS OK  |        |new value|new value|

|time|caching |eviction|config   |cached   |
|----|--------|------- |---------|---------|
|t0  |        |update  |new value|null     |
|t1  |        |eviction|new value|null     |
|t2  |get     |        |new value|null     |
|t3  |CAS OK  |        |new value|new value|

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-12 18:51:56 +08:00
Cai Yudong
9d4535ce0b
enhance: Handle Float16Vector/BFloat16Vector numpy bulk insert as same as BinaryVector (#33760)
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-06-12 17:17:55 +08:00
Buqian Zheng
47b04ea167
enhance: support sparse cardinal hnsw index (#33656)
issue: #29419

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-06-12 16:57:55 +08:00
zhuwenxing
1697706ac0
test: [skip e2e] optimize get_image_tag_by_short_name function (#33780)
Refactor the function to improve performance and readability. Instead of
making API requests to Docker Hub, the function now retrieves tags from
the Harbor registry. It also filters the tags based on the provided
architecture and selects the latest tag that matches the prefix. This
change enhances the efficiency of retrieving image tags by short name.

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-06-12 14:27:56 +08:00
Buqian Zheng
8cb350598c
enhance: Improve GetVectorById of Sparse Float Vector (#33209)
issue: #29419

* sparse float vector to support raw data mmap

For get vector from chunk cache, I added a unit test but marking it as
skipped due to a known issue. I have tested it locally.

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-06-12 10:09:55 +08:00
congqixia
323f8982ec
fix: [GoSDK] Use varchar when row field type is string (#33749)
See also #33457

Also add `max_length` tag for specify varchar field max length attribute
from tagging

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-11 23:39:55 +08:00
sre-ci-robot
64f673cb4e Update all contributors
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2024-06-11 12:01:03 +00:00
edward.zeng
17120afec6
test: [skip e2e] fix image tag (#33764)
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
2024-06-11 19:45:55 +08:00
edward.zeng
9a141b52f9
test: [skip e2e] fix the groovy for building docker images (#33763)
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
2024-06-11 19:15:54 +08:00
edward.zeng
4ed3b5e5c1
test: [skip e2e] fix the push order for milvus image (#33762)
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
2024-06-11 18:55:54 +08:00
coldWater
6b9901c59f
enhance: add a semaphore for CompactionExecutor (#33558)
#33182

---------

Signed-off-by: coldWater <254244460@qq.com>
2024-06-11 17:25:55 +08:00
Chun Han
f7af323d1e
fix: sync partitiion stats blocking balance task(#33741) (#33742)
related: #33741

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-11 14:21:56 +08:00
Jiquan Long
ecf2bcee42
enhance: speed up array-equal operator via inverted index (#33633)
fix: #33632

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-06-11 14:13:54 +08:00
zhuwenxing
fa26953168
test: fix unstable import case (#33726)
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-06-11 14:03:54 +08:00
chyezh
2b7ee1968f
enhance: new messsage interface for log service (#33286)
issue: #33285

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-11 10:38:01 +08:00
chyezh
8ca5ced821
fix: async warmup will be blocked by state lock (#33686)
issue: #33685

Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-10 21:59:53 +08:00
yihao.dai
b1d46eb34b
fix: Fix multiple vector fields import (#33723)
1. Fix dim mismatch with multi-vector fields and JSON import
2. Enhance: do not display file ID in GetImportResponse.

issue: https://github.com/milvus-io/milvus/issues/33681,
https://github.com/milvus-io/milvus/issues/33682

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-10 21:57:54 +08:00
yihao.dai
eb5d4de390
fix: Check if the import job exists (#33672)
issue: https://github.com/milvus-io/milvus/issues/33671

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-10 21:51:55 +08:00
wayblink
a1232fafda
feat: Major compaction (#33620)
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
2024-06-10 21:34:08 +08:00
Aldrin
8a4ef1b836
fix: Masked kafka credentials getting printed in logs (#33730)
issue: https://github.com/milvus-io/milvus/issues/33727

Signed-off-by: Aldrin <imagesai32@gmail.com>
2024-06-10 00:05:54 +08:00
chyezh
f53ab54c5d
enhance: async cgo utility (#33133)
issue: #30926, #33132

- implement future-based cgo utility.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-06-09 22:55:53 +08:00
congqixia
80a2cd192a
fix: [GOSDK] Pass outputFields param in search request (#33710)
See also #33627

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-07 18:27:53 +08:00
ThreadDao
3729d2c8e7
feat: add test case for go client insert (#33603)
issue: #33419

Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2024-06-07 14:47:52 +08:00
XuanYang-cn
1629833060
enhance: Add consts of MsgDispatcher to configs (#33679)
See also: #33676

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-06-07 14:21:59 +08:00
yihao.dai
3540eee977
enhance: Support L0 import (#33514)
issue: https://github.com/milvus-io/milvus/issues/33157

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-07 14:17:20 +08:00
congqixia
60baaed832
fix: Make fp16&bf16 column use correct byte length (#33684)
See also milvus-io/milvus-sdk-go#756 #31293

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-07 11:31:53 +08:00
zhagnlu
6ce9df913f
fix: clean vector memory (#33692)
#33533

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-07 10:37:54 +08:00
wei liu
0a3d456688
enhance: add restful api to trigger component stop (#32076)
issue: #32698
This PR add two rest api for component stop and status check:
1. `/management/stop?role=querynode` can stop the specified component
2. `/management/check/ready?role=rootcoord` can check whether the target
component is serviceable

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-07 10:35:54 +08:00
zhuwenxing
e3d50a192d
test: wait all tasks finished in import test (#33674)
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-06-07 10:35:12 +08:00
sammy.huang
19936d9994
fix:[skip e2e] make more retries on codecov uploader (#33683)
fix #32235

inspired by
https://github.com/Kong/kubernetes-testing-framework/blob/main/.github/workflows/tests.yaml#L53-L69

### What is changing:
Introducing the
[wretry.action](https://github.com/Wandalen/wretry.action) to
automatically retry the
[codecov-action](https://github.com/codecov/codecov-action) in our CI/CD
pipeline. This change is intended to enhance the robustness of our
workflow by mitigating transient failures in the codecov-action.

### Follow-up changes needed:
This retry mechanism will be removed once the issue
[codecov/codecov-action#926](https://github.com/codecov/codecov-action/issues/926)
is resolved or if GitHub Actions provides a built-in retry feature for
actions.

Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
2024-06-07 10:33:52 +08:00
smellthemoon
c61fb1eff5
enhance: do check when add not empty logpath (#33640)
meta only store logid

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-06-07 10:19:51 +08:00
SimFG
ecee7d90d4
enhance: try to speed up the loading of small collections (#33570)
- issue: #33569

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-06-07 08:25:53 +08:00
zhuwenxing
9c2e3259d1
test: add coo format sparse vector in restful test (#33677)
* add coo format sparse vector
* search data and insert data in the same sparse format or a different
format

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-06-06 18:05:51 +08:00
cai.zhang
27cc9f2630
enhance: Support analyze data (#33651)
issue: #30633

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>
2024-06-06 17:37:51 +08:00
cai.zhang
cfea3f43cf
fix: Don't sync L0 segments to channel watcher (#33664)
issue: #33540

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-06-06 15:59:50 +08:00
zhuwenxing
86274f70bd
test: improve concurrency and reduce import test execution time (#33356)
improve concurrency and reduce import test execution time

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-06-06 15:57:58 +08:00
XuanYang-cn
4dd0c54ca0
fix: Fix l0 compactor may cause DN from OOM (#33554)
See also: #33547

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-06-06 14:33:52 +08:00
congqixia
f6e251514f
fix: Write back dbid modification for nonDB id collection (#33641)
See also #33608

Make `fixDefaultDBIDConsistency` also write back collection dbid
modification when nonDB id collection is found.

This fix shall prevent dropped collections of this kind show up again
after dropping and restart.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-06 14:29:53 +08:00
wei liu
b69740c8f3
enhance: Remove unnecessary log info during load segment (#33663)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-06-06 14:13:50 +08:00
XuanYang-cn
68c9e7db8c
fix: Sync dropped segment for dropped partition (#33331)
See also: #33330

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-06-06 10:25:52 +08:00
sre-ci-robot
fd191dd7db
[automated] Update Knowhere Commit (#33655)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-06-06 01:53:50 +08:00
cai.zhang
feeb869ff9
enhance: Remove compaction plans on the datanode (#33548)
issue: #33546

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-06-05 20:27:51 +08:00
cqy123456
703fc73f71
enhance: disk index support binary vector (#33631)
issue:https://github.com/milvus-io/milvus/issues/22837
related https://github.com/milvus-io/milvus/pull/33575

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-05 19:37:57 +08:00
cai.zhang
412ccfbb20
enhance: Refine IndexNode code and ensure compatibility (#33458)
issue: #33432 , #33183

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-06-05 19:35:52 +08:00
zhagnlu
8ad26093ba
fix: fix load failure (#33599)
issue: #33533

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-06-05 19:19:51 +08:00
aoiasd
b9bafc76b4
enhance: access log support print with consistency level (#33503)
relate: https://github.com/milvus-io/milvus/issues/33563

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-06-05 17:13:50 +08:00
yihao.dai
01ce32caa1
enhance: Print more disk quota info (#33596)
/kind enhancement

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-05 16:51:56 +08:00
yihao.dai
bbdf99a45e
fix: Fix import segment size is uneven (#33605)
The data coordinator computed the appropriate number of import segments,
thus when importing in the data node, one can randomly select a segment.

issue: https://github.com/milvus-io/milvus/issues/33604

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-06-05 15:41:51 +08:00
Gao
545d4725fb
fix: correct get vector data size for bf16/fp16/binary vector (#33377)
related #22837

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-06-05 14:31:57 +08:00
congqixia
597f4c5e03
enhance: Make hasMoreResult accurate when hit number larger than limit (#33609)
See also milvus-io/milvus-sdk-go#756

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-06-05 11:51:51 +08:00