Commit Graph

3064 Commits

Author SHA1 Message Date
nico
cf66fbee8a
test: update test cases (#37215)
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-11-06 17:50:25 +08:00
aoiasd
d67853fa89
feat: Tokenizer support build with params and clone for concurrency (#37048)
relate: https://github.com/milvus-io/milvus/issues/35853
https://github.com/milvus-io/milvus/issues/36751

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-06 17:48:24 +08:00
congqixia
c83b93946e
enhance: [GoSDK] Add alter collection API & expose options (#37365)
Related to #31293

This PR:

- Add `AlterCollection` API for collection property modification
- Expose hidden or missing option methods

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-06 14:36:25 +08:00
zhuwenxing
0fc6c634b0
test: fix tokenizer and monkey patch faker function (#37119)
/kind improvement

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
Signed-off-by: zhuwenxing <wxzhuyeah@gmail.com>
2024-11-05 08:42:23 +08:00
cai.zhang
ba9f36ba33
enhance: Increase the concurrency of index tasks in CI (#37335)
#37096

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-11-04 12:04:23 +08:00
foxspy
eaf86f7649
fix: optimize invalid datatype error msg (#37376)
issue: #37151

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-11-04 11:46:28 +08:00
cai.zhang
50de122dc7
enhance: Rename textmatch to text_match (#37290)
issue: #36672

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-11-03 18:40:27 +08:00
congqixia
be71b98146
enhance: Fix case error msg after knowhere upgrade (#37339)
Previous PR: #37315

Error message updated after knowhere behavior change, this PR update
corresponding test error message.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-01 14:00:22 +08:00
wei liu
f190e5d802
fix: [skip e2e] TestNodeDownOnSingleReplica has unstable result (#37288)
issue: #37289
those test case use search to verify replica's status, but if the search
gap is 1s, the node down's effect may be fixed up by balance.

This PR remove the 1 second gap between search operation.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-01 10:50:21 +08:00
zhuwenxing
247f75180f
test: add restful cases for full text search and some minor fix (#37148)
/kind improvement

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-10-31 21:18:23 +08:00
zhuwenxing
d24970c090
test: fix potential unstable cases of full-text search (#37328)
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-10-31 21:12:21 +08:00
zhenshan.cao
63843dce33
fix: Fix conan gdal building problem (#37338)
issue:https://github.com/milvus-io/milvus/issues/27576

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-10-31 21:04:16 +08:00
Hao Tan
67c4340565
feat: Geospatial Data Type and GIS Function Support for milvus server (#35990)
issue:https://github.com/milvus-io/milvus/issues/27576

# Main Goals
1. Create and describe collections with geospatial fields, enabling both
client and server to recognize and process geo fields.
2. Insert geospatial data as payload values in the insert binlog, and
print the values for verification.
3. Load segments containing geospatial data into memory.
4. Ensure query outputs can display geospatial data.
5. Support filtering on GIS functions for geospatial columns.

# Solution
1. **Add Type**: Modify the Milvus core by adding a Geospatial type in
both the C++ and Go code layers, defining the Geospatial data structure
and the corresponding interfaces.
2. **Dependency Libraries**: Introduce necessary geospatial data
processing libraries. In the C++ source code, use Conan package
management to include the GDAL library. In the Go source code, add the
go-geom library to the go.mod file.
3. **Protocol Interface**: Revise the Milvus protocol to provide
mechanisms for Geospatial message serialization and deserialization.
4. **Data Pipeline**: Facilitate interaction between the client and
proxy using the WKT format for geospatial data. The proxy will convert
all data into WKB format for downstream processing, providing column
data interfaces, segment encapsulation, segment loading, payload
writing, and cache block management.
5. **Query Operators**: Implement simple display and support for filter
queries. Initially, focus on filtering based on spatial relationships
for a single column of geospatial literal values, providing parsing and
execution for query expressions.
6. **Client Modification**: Enable the client to handle user input for
geospatial data and facilitate end-to-end testing.Check the modification
in pymilvus.

---------

Signed-off-by: tasty-gumi <1021989072@qq.com>
2024-10-31 20:58:20 +08:00
cai.zhang
2ef6cbbf59
feat: The expression supports filling elements through templates (#37033)
issue: #36672

The expression supports filling elements through templates, which helps
to reduce the overhead of parsing the elements.

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-10-31 14:20:22 +08:00
zhuwenxing
6e37372619
test: update checker (#37275)
Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-10-31 09:50:20 +08:00
yanliang567
3a3404658e
test: Add a test for iterator enhancement (#37296)
related issue: https://github.com/milvus-io/milvus/issues/37084

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2024-10-30 16:54:20 +08:00
Zhen Ye
889434691c
enhance: enable asan for e2e (#37149)
issue: #35854

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-10-29 14:14:24 +08:00
zhuwenxing
4c108b1564
test: update jieba tokenizer in test (#37199)
/kind improvement

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-10-28 19:22:22 +08:00
zhuwenxing
cdee149191
test: fix testcases for verification after chaos (#37153)
/kind improvement

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-10-28 10:33:29 +08:00
zhuwenxing
c8dd665bf6
test: supplementing case for text match (#36693)
/kind improvement

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-10-28 10:31:40 +08:00
foxspy
d7b2ffe5aa
enhance: add an unify vector index config checker (#36844)
issue: #34298

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-10-28 10:11:37 +08:00
yihao.dai
ed37c27bda
fix: Fix collection leak in querynode (#37061)
Unref the removed L0 segment count.

issue: https://github.com/milvus-io/milvus/issues/36918

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-25 19:59:29 +08:00
Yinzuo Jiang
3628593d20
feat: Implement custom function module in milvus expr (#36560)
OSPP 2024 project:
https://summer-ospp.ac.cn/org/prodetail/247410235?list=org&navpage=org

Solutions:

- parser (planparserv2)
    - add CallExpr in planparserv2/Plan.g4
    - update parser_visitor and show_visitor
- grpc protobuf
    - add CallExpr in plan.proto
- execution (`core/src/exec`)
- add `CallExpr` `ValueExpr` and `ColumnExpr` (both logical and
physical) for function call and function parameters
- function factory (`core/src/exec/expression/function`)
    - create a global hashmap when starting milvus (see server.go)
- the global hashmap stores function signatures and their function
pointers, the CallExpr in execution engine can get the function pointer
by function signature.
- custom functions
    - empty(string)
    - starts_with(string, string)
- add cpp/go unittests and E2E tests

closes: #36559

Signed-off-by: Yinzuo Jiang <jiangyinzuo@foxmail.com>
2024-10-25 15:25:30 +08:00
Buqian Zheng
088d5d7d76
fix: optimize BM25 err message (#37074)
issue: https://github.com/milvus-io/milvus/issues/37022

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-10-25 14:35:45 +08:00
zhuwenxing
ac2858d418
test: add full text search checker in test (#37122)
/kind improvement

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-10-25 14:09:29 +08:00
qixuan
80aa9ab4d6
test: Add insert and upsert related cases for null and default value support (#37088)
issue: #36129

---------

Signed-off-by: qixuan <673771573@qq.com>
2024-10-25 11:03:29 +08:00
binbin
c285853de8
test: Add test cases about expr for null and default support (#37121)
issue: #36129

Signed-off-by: binbin lv <binbin.lv@zilliz.com>
2024-10-25 11:01:30 +08:00
smellthemoon
2b3f5bec07
fix: panic when create index on all none data (#37046)
#37045

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-10-24 17:09:28 +08:00
sre-ci-robot
53836f320a
[automated] Update Pytest image changes (#37067)
Update Pytest image changes
See changes:
3b024f9b36
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-10-24 16:53:28 +08:00
yanliang567
0d376f1a8f
test: Add query iterator test and seperate a new test file (#37085)
related issue: #37084

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2024-10-24 11:47:29 +08:00
jaime
4746f47282
feat: management WebUI homepage (#36822)
issue: #36784
1. Implement an embedded web server for WebUI access.  
2. Complete the homepage development.

Home page demo:
<img width="2177" alt="iShot_2024-10-10_17 57 34"
src="https://github.com/user-attachments/assets/38539917-ce09-4e54-a5b5-7f4f7eaac353">

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-23 11:29:28 +08:00
zhuwenxing
3b024f9b36
test: Add full-text search test cases (#36998)
/kind improvement

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-10-23 09:51:27 +08:00
zhuwenxing
80d48f1e53
test: add text match checker in test (#37052)
/kind improvement

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-10-23 09:49:27 +08:00
qixuan
36147b10f5
test: Add insert and upsert related cases for null and default value support (#36932)
issue: #36129

Signed-off-by: qixuan <673771573@qq.com>
2024-10-22 10:13:27 +08:00
zhuwenxing
ef923958bb
test: add restful param check testcases (#36931)
/kind improvement

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-10-19 19:13:25 +08:00
yanliang567
d17135d3bc
test: Add alias test with rename collection (#36978)
related issue: #36963 
1. add alias tests with rename collection for issue #36963
2. update some test levels
3. fix some xfail tests

---------

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2024-10-18 16:27:24 +08:00
smellthemoon
eb3e4583ec
enhance: all op(Null) is false in expr (#35527)
#31728

---------

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-10-17 21:14:30 +08:00
wt
97ff012c67
test: check query response data (#36952)
Signed-off-by: wangting0128 <ting.wang@zilliz.com>
2024-10-17 19:13:25 +08:00
congqixia
e5948bd039
enhance: [GoSDK] Use variadic params for options (#36912)
Use variadic parameter function for options make client options easier
to use.

Related to #31293

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-10-16 19:33:23 +08:00
smellthemoon
44d80c1355
fix: not return err if consistencyLevel is not set to a valid value (#36714)
https://github.com/milvus-io/milvus/issues/36444

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-10-16 13:23:22 +08:00
nico
2291d0c9f5
test: update test cases (#36878)
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-10-15 19:13:24 +08:00
yanliang567
0ec92513a5
test: Update Diskann supporting group by (#36853)
related issue: #36813 

milvus diskann does not support group_search before, it supports now.

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2024-10-15 10:35:23 +08:00
ThreadDao
d566b0ceff
test: add stats task feature cases and DefaultVectorSearchParams (#36768)
issue: #36767

---------

Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2024-10-14 19:15:23 +08:00
nico
937ebec2ce
test: update test cases (#36841)
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-10-14 19:13:23 +08:00
sre-ci-robot
af2da8d8d6
[automated] Update Pytest image changes (#36845)
Update Pytest image changes
See changes:
c96bbe19ba
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-10-14 19:11:23 +08:00
yanliang567
c96bbe19ba
test: Add upsert in rows tests (#36820)
related issue: #36710

---------

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2024-10-14 16:41:22 +08:00
yanliang567
621dbc9107
test: Add json key in [] test to verify issue 36718 (#36738)
related issue: #36718

---------

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2024-10-11 14:29:21 +08:00
aoiasd
db34572c56
feat: support load and query with bm25 metric (#36071)
relate: https://github.com/milvus-io/milvus/issues/35853

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-10-11 10:23:20 +08:00
sre-ci-robot
90285830de
[automated] Update Pytest image changes (#36755)
Update Pytest image changes
See changes:
f7b811450d
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-10-11 09:29:19 +08:00
Buqian Zheng
f7b811450d
feat: add enable_tokenizer params to VarChar field (#36480)
issue: #35922

add an enable_tokenizer param to varchar field: must be set to true so
that a varchar field can enable_match or used as input of BM25 function

---------

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-10-10 20:33:21 +08:00