Commit Graph

30 Commits

Author SHA1 Message Date
Chun Han
c3264ca3e3
feat: support segment pruner (#31003)
related: #30376
2024-03-22 13:57:06 +08:00
Patrick Weizhi Xu
982dd2834b
enhance: add materialized view search info (#30888)
issue: #29892 

This PR
1. Pass Materialized View (MV) search information obtained from the
expression parsing planning procedure to Knowhere. It only performs when
MV is enabled and the partition key is involved in the expression. The
search information includes:
1. Touched field_id and the count of related categories in the
expression. E.g., `color == red && color == blue` yields `field_id ->
2`.
2. Whether the expression only includes AND (&&) logical operator,
default `true`.
    3. Whether the expression has NOT (!) operator, default `false`.
4. Store if turning on MV on the proxy to eliminate reading from
paramtable for every search request.
5. Renames to MV.

## Rebuttals
1. Did not write in `ExtractInfoPlanNodeVisitor` since the new scalar
framework was introduced and this part might be removed in the future.
2. Currently only interested in `==` and `in` expression, `string` data
type, anything else is a bonus.
3. Leave handling expressions like `F == A || F == A` for future works
of the optimizer.

## Detailed MV Info

![image](https://github.com/milvus-io/milvus/assets/6563846/b27c08a0-9fd3-4474-8897-30a3d6d6b36f)

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-03-21 11:19:07 +08:00
Buqian Zheng
96cfae55a5
feat: [Sparse Float Vector] segcore to support sparse vector search and get raw vector by id (#30629)
This PR adds the ability to search/get sparse float vectors in segcore,
and added unit tests by modifying lots of existing tests into
parameterized ones.

https://github.com/milvus-io/milvus/issues/29419

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-03-12 09:16:30 -07:00
Xu Tong
e429965f32
Add float16 approve for multi-type part (#28427)
issue:https://github.com/milvus-io/milvus/issues/22837

Add bfloat16 vector, add the index part of float16 vector.

Signed-off-by: Writer-X <1256866856@qq.com>
2024-01-11 15:48:51 +08:00
MrPresent-Han
9e2e7157e9
feat: support search_group_by for milvus(#25324) (#28983)
related: #25324

Search GroupBy function, used to aggregate result entities based on a
specific scalar column.
several points to mention:

1. Temporarliy, the whole groupby is implemented separated from
iterative expr framework **for the first period**
2. In the long term, the groupBy operation will be incorporated into the
iterative expr framework:https://github.com/milvus-io/milvus/pull/28166
3. This pr includes some unrelated mocked interface regarding alterIndex
due to some unworth-to-mention reasons. All these un-associated content
will be removed before the final pr is merged. This version of pr is
only for review
4. All other related details were commented in the files comparison

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-01-05 15:50:47 +08:00
cai.zhang
a362bb1457
Support array datatype (#26369)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-09-19 14:23:23 +08:00
Xu Tong
9166011c4a
Add float16 vector (#25852)
Signed-off-by: Writer-X <1256866856@qq.com>
2023-09-08 10:03:16 +08:00
cai.zhang
a0198ce8ae
Support json contains feature (#25384)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-08-11 17:09:30 +08:00
Jiquan Long
6578a27fe7
Support empty expression (#26153)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-08-10 08:37:15 +08:00
Jiquan Long
5c1f79dc54
Push down the limit operator to segcore (#25959)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-08-01 20:29:05 +08:00
cai.zhang
a6dbcaeb7a
Add proto field to support reverse in (#24678)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-06-07 17:24:36 +08:00
xige-16
732fe54775
Support partition Key (#24047)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-06-06 10:24:34 +08:00
cai.zhang
9715a850fa
Support expr with json field (#23804)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-05-10 10:19:19 +08:00
yah01
e23af6406f
Add nested path for accessing json fields (#23651)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-04-24 17:14:34 +08:00
Jiquan Long
8139106b51
Feat: count entities by expression (#22765)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-03-16 19:31:55 +08:00
xige-16
a8829554eb
Merge multiple sets of expr's optypes in diff dirs (#17192)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-05-24 21:56:00 +08:00
Jiquan Long
98ceb162aa
Support antlr as plan parser (#16696)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>

Co-authored-by: xaxys <tpnnghd@163.com>

Co-authored-by: xaxys <tpnnghd@163.com>
2022-05-06 17:43:51 +08:00
xige-16
515d0369de
Support string type in segcore (#16546)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>

Co-authored-by: dragondriver <jiquan.long@zilliz.com>
2022-04-29 13:35:49 +08:00
Aivin V. Solatorio
04fffb08ef
Support arithmetic operations on numerical fields for scalar filtering (#16520)
Signed-off-by: Aivin V. Solatorio <avsolatorio@gmail.com>
2022-04-24 16:43:45 +08:00
xige-16
99984b88e1
Support delete varChar value (#16229)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-04-02 17:43:29 +08:00
elfisworking
d4232f88a2
For search funtion adding a round_decimal paramter to precision control (#8574)
Signed-off-by: elfisworking <zymustb@126.com>
2021-10-08 17:39:55 +08:00
yukun
94272bba87
Support query by expression (#7386)
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-09-03 17:12:55 +08:00
xaxys
e43b43e7d8
Refactor RangeExpr & CompareExpr (#6786)
* Changed `RangeExpr` proto to `UnaryRangeExpr` & `BinaryRangeExpr`
Several unit test unpassed.

Signed-off-by: xaxys <tpnnghd@163.com>

* Fix bugs to pass unit test. Fix format.

Signed-off-by: xaxys <tpnnghd@163.com>

* Remove debug information.

Signed-off-by: xaxys <tpnnghd@163.com>

* Fix format.
Remove debug information.
Unify variable name.
Add error information.
Remove `CompareExpr` test in `test_c_api.cpp`.

Signed-off-by: xaxys <tpnnghd@163.com>

* Fix code format.

Signed-off-by: xaxys <tpnnghd@163.com>

* Update `Plan.cpp`.

Signed-off-by: xaxys <tpnnghd@163.com>
2021-07-25 10:23:19 +08:00
xaxys
e5f8a77a2d
Extend grammar of boolean expression (#6460)
* Extend grammar of boolean expression

Signed-off-by: xaxys <tpnnghd@163.com>
2021-07-13 14:37:03 +08:00
yukun
b2eb596c23
Change search result to columnBased and return target_entry (#5896)
* Change search result to columnBased and return target_entry

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

Co-authored-by: fluorinedog <fluorinedog@gmail.com>

* synchronize the latest pymilvus-orm version

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Change pymilvus-orm version to 40

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

Co-authored-by: fluorinedog <fluorinedog@gmail.com>
2021-06-21 20:18:13 +08:00
FluorineDog
bec9f2c182
Split segcore and plan proto for future feature (#5767)
* Split segcore and plan proto for future feature

Signed-off-by: fluorinedog <fluorinedog@gmail.com>

* lint

Signed-off-by: fluorinedog <fluorinedog@gmail.com>
2021-06-15 14:43:57 +08:00
Xiangyu Wang
68f703f983
Add Query impl in proxynode (#5644)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-06-07 17:21:36 +08:00
FluorineDog
9a90313390
Support GetEntityByIDs in CGo, fix segcore bugs (#5563)
Signed-off-by: fluorinedog <fluorinedog@gmail.com>
2021-06-04 10:38:34 +08:00
FluorineDog
396b3f33e9
Support TermExpr, NotExpr, LogicalExpr (#5096)
1. Support Term, like `A in [1, 2, 3]`
2. Support Not, like `! A < 3`
3. Support logical combination, like `A < 3 && B > 5 or C == 0`

Type: Feature

Signed-off-by: fluorinedog <fluorinedog@gmail.com>
2021-04-30 07:19:52 +00:00
FluorineDog
88f5642603
Add plan proto and support basic boolean expr parser (#5088)
**What type of PR is this?**
- [x] Feature

**What this PR does / why we need it:**
This PR supports boolean expression as DSL.
1. The goal of this PR is to support predicates
    like `A > 3 && not B < 5 or C in [1, 2, 3]`. 
2. Defines `plan.proto`, as Intermediate Representation (IR) 
    used between go and cpp. 
3. Support expr parser, convert predicate expr to IR
    in proxynode, while doing static check there
4. Support IR to AST in cpp, enable the execution
2021-04-29 08:48:06 +00:00