Commit Graph

19259 Commits

Author SHA1 Message Date
PowderLi
58d7b9f902
fix: [restful v2] search & advanced_search API (#31113)
issue: #30688
former pr:  #30946

1. param `vector` is required #31012
2. param `annsField` is optional, for multiply vector fields #31010
3. support BinaryVector, Float16Vector, BFloat16Vector #31013
4. replace vector with data, to align with pymilvus milvus_client #31093
5. create collection quickly, to align with pymilvus milvus_client
#31149

---------

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-03-11 15:57:01 +08:00
wei liu
9cfe183253
enhance: remove duplicated target node id check (#31087)
issue: #31109
This PR remove duplicate target node id check, due to server id has
already been checked in rpc's interceptor

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-11 15:31:02 +08:00
Buqian Zheng
070dfc77bf
feat: [Sparse Float Vector] segcore basics and index building (#30357)
This commit adds sparse float vector support to segcore with the
following:

1. data type enum declarations
2. Adds corresponding data structures for handling sparse float vectors
in various scenarios, including:
* FieldData as a bridge between the binlog and the in memory data
structures
* mmap::Column as the in memory representation of a sparse float vector
column of a sealed segment;
* ConcurrentVector as the in memory representation of a sparse float
vector of a growing segment which supports inserts.
3. Adds logic in payload reader/writer to serialize/deserialize from/to
binlog
4. Adds the ability to allow the index node to build sparse float vector
index
5. Adds the ability to allow the query node to build growing index for
growing segment and temp index for sealed segment without index built

This commit also includes some code cleanness, comment improvement, and
some unit tests for sparse vector.

https://github.com/milvus-io/milvus/issues/29419

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-03-11 14:45:02 +08:00
Cai Yudong
a99143dd52
fix: Save traceID and spanID as hex string into search config (#31071)
Issue: #30961

Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-03-11 14:21:01 +08:00
Bingyi Sun
425da78b38
fix: alter index request's index name can not be empty (#31128)
issue: https://github.com/milvus-io/milvus/issues/31138

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-11 13:05:02 +08:00
congqixia
ff1e967e89
enhance: Add segment id short cut for WithSegmentID filter (#31144)
See also #31143

This PR add short cut for datanoe metacache `WithSegmentIDs` filter,
which could just fetch segment from map with provided segmentIDs. Also
add benchmark for new implementation vs old one.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-11 10:55:02 +08:00
yihao.dai
8cb06acfed
feat: Replacing the current import API with the v2 implementation (#31046)
Replacing the current import API v1 implementation with the v2
implementation.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-10 12:23:02 +08:00
congqixia
a944c1f912
enhance: Auto release in integration test tear down (#31147)
Now the integration tests have lots of time during tear down phase when
there are some segments or channel left in querynode(s). Querynode
cannot quick as soon as test finishes since graceful shut down logic
need querynode has no segment or channel in it. Auto releasing loaded
collection could reduce lots of running time for integration test.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-09 11:17:00 +08:00
sre-ci-robot
53af6d8c59
[automated] Update Knowhere Commit (#31151)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-03-09 01:55:02 +08:00
Ted Xu
987d9023a5
enhance: Enable binlog deserialize reader in datanode compaction (#31036)
See #30863

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-03-08 18:25:02 +08:00
wei liu
ddd918ba04
enhance: change frequency log to rated level (#31084)
This PR change frequency log of check shard leader to rated level

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-08 16:39:02 +08:00
wei liu
efe8cecc88
enhance: refactor segment dist manager interface (#31073)
issue: #31091
This PR add `GetByFilter` interface in segment dist manager, instead of
all kind of get func

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-08 16:29:01 +08:00
XuanYang-cn
ff80d2fd8c
enhance: Enable L0 by default (#30998)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-03-08 15:53:02 +08:00
wei liu
5016031dc7
enhance: add metrics for load segment progress (#31005)
This PR add metrics for load segment progress:
1. add metrics for load segment/index concurrency
2. add metrics for load index latency
3. change load segment latency's time unit to ms

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-08 15:39:02 +08:00
Cai Yudong
122981aeb9
fix: Disable knowhere trace as a quick fix (#31055)
Issue: #30961

Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-03-08 15:27:01 +08:00
congqixia
a1db0fc406
enhance: Set log level to INFO for integration test (#31136)
- Set log level to INFO for integration test
- Improve hello_milvus integration test runtime and error check

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-08 14:33:00 +08:00
Chun Han
3574bdf858
enhance: ban range-search iteration for search-group-by (#30824)
related: #30033

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-03-08 14:17:00 +08:00
wei liu
c8efed6562
fix: Balance param use duplicated key (#31112)
issue: #31115
This PR fix balance check interval  param use duplicated key

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-08 12:07:00 +08:00
wei liu
22df5061c1
fix: Leader checker can't update segment's load version (#31040)
issue: #30890

when leader checker find that leader view has an older load version of
segment, it will try to correct leader view. but the sync action doesn't
specify the latest load version. so the update operation will failed.

This PR fix leader checker can't update segment's load version and
keeping generate same task to scheduler.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-08 11:57:01 +08:00
zhuwenxing
cc51ab9cd9
test: add json and array datatype check in restful v1 (#31097)
* When the collection is created using an SDK and includes array and
JSON datatypes in the schema, data can be inserted using the RESTful
API.
* When the collection is created using the RESTful API and includes JSON
and array datatypes in dynamic fields, data can also be inserted using
the RESTful API.

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-03-08 10:27:01 +08:00
zhuwenxing
0a2655dba0
test: fix chaos apply time (#31076)
* increase waiting time 
* still apply chaos even timeout

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-03-08 10:25:10 +08:00
XuanYang-cn
7e17f24d45
fix: Skip unstable compaction test it (#31116)
See also: #31106

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-03-08 10:23:00 +08:00
zhagnlu
a65a9ce8a5
fix: fix oom because of wrong load mode for index's raw data (#31130)
#30962

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-03-07 12:14:26 -08:00
congqixia
ba8197fafb
fix: Filter channel level zero segments when build level delete cache (#31129)
See also #31125

Delegator shall build level zero delete cache from l0 segments belongs
to it. Previously it build cache from all existing level zero segments
in the querynode which may lead to high memory usage and even panicking
when pk types are not matched

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-07 23:01:02 +08:00
congqixia
c886aa29ff
enhance: Use ListIndexes instead of DescribeIndex for qc broker (#31122)
See also #31103

Since querycoord need index meta information from datacoord only, broker
shall use `ListIndexes` to skip segment index building check logic in
datacoord

This PR is also related to #30538, in which DescribeIndex caused lots of
memory usage and lead to OOM eventually

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-07 21:43:03 +08:00
presburger
19c64067af
enhance: jemalloc aarch64 platform use 64k pagesize. (#29522)
enhance: jemalloc aarch64 platform use 64k pagesize.
issue: #28843

Signed-off-by: Yusheng.Ma <Yusheng.Ma@zilliz.com>
2024-03-07 21:01:01 +08:00
yihao.dai
c411cb4a49
enhance: Prevent the backlog of channelCP update tasks, perform batch updates of channelCPs (#30941)
This PR includes the following adjustments:
1. To prevent channelCP update task backlog, only one task with the same
vchannel is retained in the updater. Additionally, the lastUpdateTime is
refreshed after the flowgraph submits the update task, rather than in
the callBack function.
2. Batch updates of multiple vchannel checkpoints are performed in the
UpdateChannelCheckpoint RPC (default batch size is 128). Additionally,
the lock for channelCPs in DataCoord meta has been switched from key
lock to global lock.
3. The concurrency of UpdateChannelCheckpoint RPCs in the datanode has
been reduced from 1000 to 10.

issue: https://github.com/milvus-io/milvus/issues/30004

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: congqixia <congqi.xia@zilliz.com>
2024-03-07 20:39:02 +08:00
nico
045000e5f3
enhance: update pymilvus version (#31001)
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-03-07 18:05:02 +08:00
aoiasd
d19a82e3f6
enhance: support stream call for grpc client (#30013)
relate: https://github.com/milvus-io/milvus/issues/29719

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-03-07 17:45:01 +08:00
congqixia
d81ba164c8
enhance: Add ListIndexes API from datacoord (#31104)
See also #31103

This PR add `listIndexes` API for datacoor server to list all indexes
for provided collection.
Comparing to the existing `DescribeIndex` API, the new one does NOT
check the segment index building progress to ease the burden when
invoking it

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-07 17:37:01 +08:00
wei liu
2a047103d6
fix: Dirty sealed segment won't release after channel balance (#31095)
issue: #31074
This PR fix dirty sealed segment doesn't release after channel balance,
dirty sealed segment means segment doesn't exist in targets.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-03-07 16:23:01 +08:00
Bingyi Sun
eb0cfe3e84
fix: set CollectionID field for AlterCollection request (#30814)
issue: https://github.com/milvus-io/milvus/issues/30800

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-07 14:49:02 +08:00
binbin
4863765521
test: skip test cases (#31079)
issue: #30793

Signed-off-by: binbin lv <binbin.lv@zilliz.com>
2024-03-07 14:13:01 +08:00
wayblink
875036b81b
feat: Define FieldValue, FieldStats and PartitionStats (#30286)
Define FieldValue, FieldStats, PartitionStats
FieldValue is largely copied from PrimaryKey
FieldStats is largely copied from PrimaryKeyStats
PartitionStats is map[segmentid][]FieldStats
Each partition can have a PartitionStats file

/kind feature
related: #30287
related: #30633

---------

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-03-06 20:42:37 -08:00
sre-ci-robot
2d9de233fc
[automated] Update Knowhere Commit (#31089)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-03-07 12:05:02 +08:00
cai.zhang
951c91c9b4
fix: [skip e2e]Increase graceful time to ensure that bounded search executing normally (#31064)
issue: #31063 
Increase the graceful time to ensure that bounded query requests can be
executed normally in the event of RootCoord failure.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-03-07 10:53:00 +08:00
yihao.dai
8ff8fefdc1
fix: Fix errors in the Index service APIs are ignored (#31077)
In Index service APIs, return error if occurs instead of always
returning nil. Additionally, add more tests to cover this scenario.

issue: https://github.com/milvus-io/milvus/issues/31069,
https://github.com/milvus-io/milvus/issues/31027

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-06 22:11:00 +08:00
congqixia
196f0c1e1d
fix: Skip invalid compaction plan (#31045)
See also #31044

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-06 21:36:59 +08:00
congqixia
007fab183c
enhance: Skip partition key name check if feature not enabled (#31067)
Skip partition key naming & hash value pre process if collection schema
does not have partition key
The PR removes mislead warning when collection has no partition key

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-06 19:33:00 +08:00
MrPresent-Han
d0eeea4b44
fix: reduce incorrectly for group-by with offset(#30828) (#30882)
related: #30828

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-03-06 16:47:00 +08:00
Bingyi Sun
fd17a5f050
fix: check collection lazy load prop using schema (#30992)
issue: https://github.com/milvus-io/milvus/issues/30361

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-06 16:19:01 +08:00
Bingyi Sun
df7aafa3ec
fix: filter mmap key when checking index params (#31030)
issue: https://github.com/milvus-io/milvus/issues/31031

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-06 16:03:00 +08:00
Jiquan Long
a88c896733
enhance: purge client infos periodically (#31037)
https://github.com/milvus-io/milvus/issues/31007

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-03-06 12:50:59 +08:00
sre-ci-robot
3c9ffdedf3 Update all contributors
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2024-03-05 12:00:52 +00:00
XuanYang-cn
def72947c7
fix: Trigger l0 compaction when l0 views don't change (#30729)
Trigger l0 compaction when l0 views don't change

So that leftover l0 segments would be compacted in the end.

1. Refresh LevelZero plans in comactionPlanHandler, remove the meta
dependency
of compaction trigger v2
2. Add ForceTrigger method for CompactionView interface
3. rename mu to taskGuard
4. Add a new TriggerTypeLevelZeroViewIDLE
5. Add an idleTicker for compaction view manager

See also: #30098, #30556

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-03-05 16:37:00 +08:00
congqixia
30398d4b71
enhance: Fix misleading log content & possible nil panic (#31021)
- Change load field log from "dy pool" to "load pool"
- Also defer delete when there is no error

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-05 16:33:00 +08:00
sre-ci-robot
c047f09110
[automated] Update Knowhere Commit (#31015)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-03-05 16:31:00 +08:00
Bingyi Sun
e3cce11dd9
fix: data race in querynode task test (#31019)
issue: https://github.com/milvus-io/milvus/issues/31022

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-05 16:26:59 +08:00
yihao.dai
3b66c17279
feat: Revise the RESTful bulk insert API (#29698)
Revise the RESTful bulk insert API from version 1 to version 2.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-05 15:03:00 +08:00
congqixia
8c2615f840
enhance: Add unit(seconds) for new added connection manager param (#31023)
See also #31007 #31008 #31009

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-05 14:50:59 +08:00