Commit Graph

19257 Commits

Author SHA1 Message Date
Xiaofan
9488ac0939
Update README.md 2024-01-25 16:50:44 +08:00
yah01
a27c0e86fd
enhance: reduce many I/O operations while loading disk index (#30189)
before this, every time writting the index chunk data into the disk,
there are 4 I/O operations:
- open the file
- seek to the offset
- write the data
- close the file

this optimized this to open only once and continiously write all data.

This also makes it concurrent to load the files from object storage

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-25 15:23:02 +08:00
SimFG
aa7014a360
enhance: move the cgo code in the pkg dir to interal dir (#30261)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-01-25 15:15:01 +08:00
zhuwenxing
78562d0246
test: add multi vector for bulk insert test (#30223)
add multi vector for bulk insert test

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-01-25 15:03:01 +08:00
yah01
51fe4743f1
enhance: hint the invalid metric type in error message (#30263)
this hints the user passed invalid metric type in error message, so
users could know what's wrong

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-25 14:15:00 +08:00
congqixia
8e8ac213aa
enhance: Utilize partition key optimization in reQuery (#30253)
See also #30250

This PR add requery flag in query task. When reQuery flag is true, query
task shall skip partition name conversion and use pre-calculated
partitionIDs passed from search task.

TODO: hybrid search does not have partition id information. we shall
apply same logic for hybrid search later.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-25 11:05:07 +08:00
xige-16
4f25066aa7
fix: Fix requery without limit during HybridSearch (#30220)
issue: #29990 

/kind bug

Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-01-25 11:03:00 +08:00
aoiasd
da749c0df2
enhance: [skip e2e] fix integration block UT 30 minutes after failed and use merr (#30125)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-01-25 10:09:02 +08:00
zhagnlu
8c58d9af67
enhance: optimize marisa trie range search for performance (#30079)
#30078
#29986

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-01-25 10:07:00 +08:00
nico
ba862ef91d
enhance: update pymilvus version and update cases (#30012)
1. modify test case: test_search_repeatedly_ivf_index_different_limit
2. update pymilvus version from 2.4.0rc19 to 2.4.0rc24
3. Before, insert will return a pk list. In the latest milvus client,
insert will return a number that is inserted successfully
4. In the latest milvus client, flush and num_entities have been removed
5. Before, the default consistency level of a new collection is strong.
In the latest milvus client, it becomes bounded. So related cases have
been modified correspondingly, or immediate search after insert will
return no results.
6. In the latest pymilvus, new data type FLOAT16_VECTOR and
BFLOAT16_VECTOR have been added.

Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-01-24 21:13:01 +08:00
sammy.huang
3e741cc3e3
enhance: [skip e2e]increase timeout for building gpu image (#30238)
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-01-24 15:35:13 +08:00
XuanYang-cn
36b8fbbadc
fix: Donot set metrics for compactTo 0 rows seg (#30126)
See also: #29204

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-24 14:23:00 +08:00
congqixia
6445880753
fix: prevent segments got flushed multiple times (#30240)
See also #30111

Segments could be "Flushed" only by `FlushSegments` grpc call from
datacoord by design. There are two possible reason to cause one segment
got flushed multiple times.

- Segment is in flushing state during multiple epoch in flowgraph
- Segment is flushed by flushTs & Flush segments

So this pr fix:

- Remove state change logic form FlushTs policy
- Change Flush segment into three stage way: Sealed->Flushing->Flushed
preventing multiple Flushed=true operations.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-24 14:19:00 +08:00
PowderLi
08ca0a2ca5
feat: support etcd authentication (#30226)
issue: #28895
add 3 configuration for ETCD config

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-24 11:35:00 +08:00
SimFG
b62748a1a6
fix: wrong format expr for the delete rest api (#30217)
/kind improvement
issue: #30092

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-01-24 11:27:06 +08:00
Patrick Weizhi Xu
0907d76253
enhance: pass partition key scalar info if enabled when build vector index (#29931)
issue: #29892 

Pass optional scalar IVF offsets to Cardinal

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-01-24 00:04:55 +08:00
aoiasd
38746dfc2b
fix: Remove useless lock which cause porxy meta cache recursive lock (#30203)
relate: https://github.com/milvus-io/milvus/issues/30193

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-01-23 23:56:55 +08:00
zhuwenxing
cad8bf6c8d
test: add restful client test in ci (#30036)
add restful client test in ci

---------

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-01-23 21:34:55 +08:00
congqixia
c9860e3d2a
fix: Init paramtable after milvus role setup (#30211)
See also #30176

Move paramtable.Init after env setup in roles.Run. Also introduced a
flag for mixture run to set role correctly for mixture mode.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-23 19:10:56 +08:00
congqixia
d73b534f1e
fix: use atomic.Pointer to store EventHandler in case of data race (#30205)
Resolves #30204

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-23 19:09:04 +08:00
zhuwenxing
24bbb12166
test: add assert message if assert failed (#30186)
add assert message if assert failed

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-01-23 16:07:01 +08:00
SimFG
463765922e
enhance: support related privilege for grant api (#30153)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-01-23 15:56:54 +08:00
cqy123456
42bb4e37e5
fix:diskann search crash when search list = 9999999999 (#30185)
issue: https://github.com/milvus-io/milvus/issues/29020
Json can't not pass a max_int32 value to int32_t, so let knowhere check
value range by itself.
After fix this, pymilvus will report:
pymilvus.exceptions.MilvusException: <MilvusException: (code=65535,
message=fail to search on QueryNode 6: worker(6) query failed: => failed
to search: arithmetic overflow: param search_list_size should be at most
2147483647)>

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-01-23 14:46:55 +08:00
cai.zhang
6cf2f09b60
feat: Support tencent cloud object storage for milvus (#30163)
issue: #30162

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-01-23 11:28:56 +08:00
congqixia
6a73860815
enhance: Add open telemetry tracing for compaction (#30168)
Resolves #30167

This PR add tracing for all compaction from the task start in datacoord
and execution procedures in datanode.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-23 10:37:00 +08:00
sre-ci-robot
3d3101796b Update all contributors
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2024-01-23 00:01:33 +00:00
yah01
a77693aa19
enhance: convert the GetObject util to async (#30166)
This makes it much easier to use

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-22 19:20:57 +08:00
XuanYang-cn
0ac37a4a0c
fix: Use longTime bucket for longer latency (#30175)
See also: #30174

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-22 19:18:55 +08:00
sre-ci-robot
e967949cc5
[automated] Update Knowhere Commit (#30120)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-01-22 18:40:54 +08:00
aoiasd
4142743128
enhance: fix access log can't print SDK version when client connect (#29680)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-01-22 17:22:54 +08:00
cai.zhang
6bfa826320
fix: Fix bug for read data from azure (#30007)
issue: #30005

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-01-22 15:44:54 +08:00
chyezh
5ee9f734c1
fix: Use determined order to lock in BlockAll to avoid deadlock (#29246)
issue: #29104

Signed-off-by: chyezh <ye.zhen@zilliz.com>
2024-01-22 14:50:56 +08:00
congqixia
8a6de3d2b1
fix: decompress deltelog path for level zero compaction (#30164)
Resolves: #30161
See also: #28873

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-22 14:44:55 +08:00
yihao.dai
8780d65b66
fix: Use channel cp as the dml&start position for import segments (#30107)
This PR discontinuing the subscription to the mq and, instead, employing
the channel checkpoint as the DML and starting position for the import
segments.

issue: https://github.com/milvus-io/milvus/issues/30106

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-22 14:36:55 +08:00
aoiasd
a81d2b4780
enhance: refine proxy metacache for concurrent safe (#29872)
relate: https://github.com/milvus-io/milvus/issues/29675

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-01-22 14:28:55 +08:00
MrPresent-Han
4436effdc3
enhance: support groupby based on scalar-index(#29965) (#30091)
related: #29965

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-01-22 10:50:54 +08:00
sammy.huang
ae4f62ab4b
enhance: [skip e2e]Revert changes from pr 29890 (#29957)
Revert changes from pr #29890

This reverts commit 61acda4948.

Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-01-22 10:00:55 +08:00
sre-ci-robot
922414c59a
[automated] Update Pytest image changes (#30104)
Update Pytest image changes
See changes:
c91254f762
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-01-22 09:26:55 +08:00
sammy.huang
8a6bc2a804
enhance: [skip e2e]enable serviceMonitor for a nightly ci (#30141)
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-01-22 09:24:55 +08:00
yihao.dai
c8a78465b4
enhance: Add a counter monitoring for the rate-limit requests (#30109)
Add a counter monitoring metric for the ratelimited rpc requests with
labels: proxy nodeID, rpc request type, and state.

issue: https://github.com/milvus-io/milvus/issues/30052

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-21 14:44:59 +08:00
PowderLi
4f44942c80
enhance: support dataType: array & json (#30076)
issue: #30075

deal with the array<?> field data correctly

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-21 13:56:55 +08:00
wei liu
f69f65ff68
fix: Leader checker can't remove segment from leader view (#30151)
issue: #30150

This PR fix three problems:
1. leader checker use wrong node id when generate release task, which
cause the release task finished immediately
2. the release request generated by leader_checker doesn't set the
`force` flag, the operation to clean leader view on delegator will fail.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-20 18:58:58 +08:00
congqixia
e3b8c3f60b
fix: collection properties not saved for alter collection (#30145)
Resolves: #30144

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-19 23:06:54 +08:00
yihao.dai
ddd741a5d4
fix: Fix closing closed chan in proxy watcher (#30143)
issue: https://github.com/milvus-io/milvus/issues/30142

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-19 23:02:54 +08:00
Bingyi Sun
d8025177fa
fix: return correct compaction plan count by datacoord (#29980)
issue: #29943

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-01-19 21:06:55 +08:00
nico
a762561730
test: update test cases for error code modification (#30124)
#23075

Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-01-19 21:04:55 +08:00
nico
ba0e980db9
test: add test cases (#29924)
issue: #28499

Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-01-19 20:58:53 +08:00
XuanYang-cn
3d46096f86
fix: Set segment level for comapct to segment (#30129)
See also: #29204

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-19 18:52:53 +08:00
xige-16
aee19dcd6b
enhance: Opt vector dimension mismatch error message (#29928)
issue: https://github.com/milvus-io/milvus/issues/29791
/kind improvement

Signed-off-by: xige-16 <xi.ge@zilliz.com>

---------

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-01-19 17:52:54 +08:00
congqixia
0d356b0545
enhance: only buffer delete if it match insert has smaller timestamp (#30122)
See also: #30121 #27675

This PR changes the delete buffering logic:
- Write buffer shall buffer insert first
- Then the delete messages shall be evaluated
  - Whether PK matches previous Bloom filter, which ts is always smaller
  - Whether PK matches insert data which has smaller timestamp
- Then the segment bloom filter is updates by the newly buffered pk rows

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-19 17:28:53 +08:00