Commit Graph

19168 Commits

Author SHA1 Message Date
congqixia
36d78e3dd0
fix: Use localStorage path to check disk cap (#30944)
See also #30943

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-01 10:17:01 +08:00
yihao.dai
a5a4ca8459
enhance: Remove debug log (#30955)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-03-01 10:02:59 +08:00
MrPresent-Han
17a2fd048e
feat: support set up knowhere-build-pool-size on querynode(#29650) (#30922)
related: #29650

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-02-29 18:15:00 +08:00
yanliang567
d930666b3e
test: Add a test for issue #30607 (#30933)
related issue: #30607 
and update some test for groupby

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2024-02-29 18:05:00 +08:00
chyezh
0c7474d7e8
enhance: add graceful stop timeout to avoid node stop hang under extreme cases (#30317)
1. add coordinator graceful stop timeout to 5s
2. change the order of datacoord component while stop
3. change querynode grace stop timeout to 900s, and we should
potentially change this to 600s when graceful stop is smooth

issue: #30310
also see pr: #30306

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-02-29 17:01:50 +08:00
Bingyi Sun
816ed671aa
fix: alter_index should return error if index not found (#30786)
issue: https://github.com/milvus-io/milvus/issues/30932

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-02-29 15:55:01 +08:00
Jiquan Long
dcfc35316f
fix: underscore matching not work (#30826)
issue: https://github.com/milvus-io/milvus/issues/30721

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-29 10:41:00 +08:00
zhuwenxing
b023bd09b3
test: skip bulk insert test for one pod standalone (#30780)
skip bulk insert test for one pod standalone

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-02-29 10:31:08 +08:00
chyezh
e59bc1e526
enhance: use latest milvus proto, add unimplemented server (#30889)
issue: #30647

Signed-off-by: chyezh <chyezh@outlook.com>
2024-02-29 10:13:00 +08:00
PowderLi
50a78b682e
fix: set proxy.http.acceptTypeAllowInt64: true as default (#30720)
issue: #30680

also let the parameter item to be refreshable

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-02-29 09:59:07 +08:00
yihao.dai
3775464b7c
enhance: Support varchar autoid for bulkinsertV1 (#30896)
This PR is a supplement to PR
https://github.com/milvus-io/milvus/pull/30377.

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-02-28 21:20:59 +08:00
sre-ci-robot
095cdbed59 Update all contributors
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2024-02-28 12:01:10 +00:00
XuanYang-cn
cdc5ce5d6f
fix: Donot set LogPath when executing compaction (#30537)
Compaction would copy logPaths from comapctFrom segA to compactTo segB,
and previous code would copy the logPath directly, causing there're
full-logPaths-of-segA in compactTo segB's meta. So, for the next
compaction of segB, if segA has been GCed, Download would report error
"The sperified key not found".

This PR makes sure compactTo segment's meta contains logID only. And
this PR also refines CompleteComapctionMutation, increasing some
readability and merge two methods into one.

See also: #30496

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-02-28 19:03:01 +08:00
congqixia
4082315bd0
enhance: Add ParseCTraceContext util function for tracing (#30883)
See also #29803

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-28 18:59:00 +08:00
aoiasd
81a2d9ced6
enhance: access log support get sdk type by user agent (#30760)
Support get sdk type by user agent when we can't get sdk version by
connection in access log.

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-02-28 18:47:00 +08:00
groot
ba6d33cd57
fix: Support TLS for kafka connection (#30468)
#27977

Add extra configurations in milvus.yaml to pass certificates for kafka.

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2024-02-28 18:43:07 +08:00
Jiquan Long
e2f35954d4
enhance: support pattern matching on json field (#30779)
issue: https://github.com/milvus-io/milvus/issues/30714

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-28 18:31:00 +08:00
sammy.huang
ed1197ea50
enhance: Revert[skip e2e] bump milvus version for scripts/standalone_embed.sh (#30892)
This reverts commit cdb3f61fc8.

Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
2024-02-28 16:55:06 +08:00
Jiquan Long
16b785e149
enhance: optimize the memory usage and speed up loading variable length data (#30787)
/kind improvement
this removes the 1x copying while loading variable length data, also
avoids constructing std::string, which could lead to memory
fragmentation

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: yah01 <yah2er0ne@outlook.com>
2024-02-28 16:45:00 +08:00
Jiquan Long
4459078e0b
fix: wrong num_entities used when mmap variable length data (#30848)
https://github.com/milvus-io/milvus/issues/30728

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-28 16:38:56 +08:00
Guillaume Moutier
d15db7596d
fix: fix permissions on /milvus for OpenShift compatibility (#30775)
OpenShift runs Pods with a random uid and gid 0.
As Milvus needs to write into the /milvus directory, this fix modifies
the group permissions to allow the root group (gid 0) to write into it.
This is a simple solution that allows to run Milvus on OpenShift without
adding security constraints, use special ServiceAccounts or whatever.
This should also not impact any other kubernetes platform as it's a
simple group permission change, nothing regarding the userid.

Note: Other actions are necessary to properly deploy the full stack
(Minio, Pulsar,...) on OpenShift. I will document them in the helm chart
project.

Fixes #25565

Signed-off-by: Guillaume Moutier <guillaume.moutier@gmail.com>
2024-02-28 16:29:05 +08:00
SimFG
229fc4f755
enhance: retry to read when the s3 get the unexpect eof error (#30861)
/kind improvement
issue: #30877

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-02-28 16:28:53 +08:00
congqixia
a115b731ed
enhance: fix old pr cpp format issue (#30894)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-28 16:28:20 +08:00
Buqian Zheng
f658dd5faa
enhance: update knowhere version to 60a5c9c (#30788)
/kind improvement

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-02-28 14:18:55 +08:00
zhuwenxing
d2c286c536
test: skip building index when field already has index (#30820)
skip building index when field already has index

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-02-28 14:08:58 +08:00
wei liu
545e8de401
fix: promote leader task failed when segment only exist on current target (#30794)
issue: #30150

`checkLeaderTaskStale` will check segment whether exist on next current
for leaderTask's growing action, which will cause promote leader task
failed when segment only exist on current target

This PR will check segment for both current or next target.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-02-28 13:14:59 +08:00
wei liu
382b7485a6
enhance: Correct misleading nodeID in GetComponentStates's log (#30731)
This PR corrects the misleading nodeId in GetComponentStates's log

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-02-28 13:13:12 +08:00
Cai Yudong
8a219e0102
feat: Support knowhere trace using OpenTelemetry (#30750)
Issue: #21508

Signed-off-by: Yudong Cai <yudong.cai@zilliz.com>
2024-02-28 12:29:00 +08:00
smellthemoon
a4f3e01a3a
fix: add the range search params check in proxy (#30423)
if check in Segcore, will not do the it when not insert data.
so, check "radius" and "range_filter" in proxy.
related with #30365

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-02-28 11:24:58 +08:00
sre-ci-robot
6e9f3ea531
[automated] Update Knowhere Commit (#30744)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-02-28 10:50:57 +08:00
sammy.huang
74c82b48b4
enhance: [skip e2e] disable macos code checker (#30874)
Signed-off-by: Liang Huang <sammy.huang@zilliz.com>
2024-02-28 10:46:57 +08:00
chyezh
77477d6340
fix: wrong context passing into NewClient, error handling lost in session_util (#30817)
issue: #30799

Signed-off-by: chyezh <chyezh@outlook.com>
2024-02-28 10:40:09 +08:00
PowderLi
6548182a4a
fix: [restful]vector field cannot be empty while insert (#30871)
issue: #30870 

did a check whether the vector field is empty in advance

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-02-28 10:36:57 +08:00
congqixia
af315539d4
fix: Remove time tick delay metrics when nodes go offline (#30833)
See also #30832

This PR removes time tick delay metrics when rootcoord GetMetrics
response does not have previously existed querynode/datanode

Also add unit tests for this case

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Signed-off-by: Congqi.Xia <congqi.xia@zilliz.com>
2024-02-28 10:10:56 +08:00
yah01
57397b1307
enhance: add new LRU cache impl (#30360)
- remove  the unused LRU cache
- add new LRU cache impl which wraps github.com/karlseguin/ccache

related #30361

---------

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-02-27 20:58:40 +08:00
zhuwenxing
f87a3a13d0
test: remove concurrent test in ci (#30862)
remove concurrent test in ci

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-02-27 19:38:56 +08:00
PowderLi
19911a411f
fix: [restful v2] bug list (#30740)
bug list
1. element data type is needed while create a collection with an array
field #30638
2. spelling mistake about metricsType #30643
3. new collection enable dynamic field / auto id as user defined #30665
4. convert rowCount from string to int #30661
5. try it's best to create a new collection #30652
6. int64 percious #20415
7. insert into collection which has multi vector fields #30674
8. cannot rename a collection to other database #30700
9. update the request body of "indexes/create" #30769
10. got [] while list indexes of a collection which has no index #30722
11. restful need encode password before call
CreateCredential/UpdateCredential #30730
12. some parameters missed the required label #30737
13.define the field to be or not to be a partition key while create a
collection #30797

enhance: support opentelemetry
enhance: support dataType: Float16Vector & BFloat16Vector #22837
enhance: describe collection will show the field is partition key or not
#30789

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-02-27 16:48:56 +08:00
zhuwenxing
7f78e9d40d
test: use jax.numpy to generate brain float16 datatype (#30825)
JAX is a more lightweight lib that can also generate bfloat16 datatype

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-02-27 12:48:55 +08:00
sammy.huang
7a49cf2104
enhance: [skip e2e] How to build Milvus v2.3.4 arm image to support ky10 sp3 (#30813)
issue: #30843

Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-02-27 10:52:57 +08:00
Xiaofan
0661d33879
enhance: change frequent log to debug (#30782)
change the "pipeline fetch insert msg" log to debug

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-02-27 10:30:56 +08:00
Jiquan Long
3e82d21ca1
enhance: reduce 1x memory copy when loading json (#30753)
/kind improvement

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-02-27 10:18:55 +08:00
congqixia
637dcffb6b
fix: Disk resource is not requested for index loaded with disk (#30757)
See also #30756

This PR:
- Request disk resource when index type, version loaded with disk
- Add attribute cache for index utility
- Add `typeutil.Pair`

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-02-27 08:50:56 +08:00
sre-ci-robot
bea4c3a418 Update all contributors
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2024-02-26 12:00:49 +00:00
foxspy
e1e87d572b
fix: compatibility for diskann cache param (#30119)
patch search cache param from index configs when index meta could not
get the search cache size key
#issue: #30113

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-02-26 16:54:55 +08:00
cqy123456
b31d1a1eb5
enhance: add addition index params for raft index (#30179)
issue: https://github.com/milvus-io/milvus/issues/29230

Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
Co-authored-by: yusheng.ma <yusheng.ma@zilliz.com>
2024-02-26 14:50:54 +08:00
zhagnlu
a0531b72aa
fix: reorder hybridsearch result for L2 metric (#30739)
#30694

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-02-26 14:18:55 +08:00
Bingyi Sun
ece9d273a7
enhance: some patches for #30636 (#30664)
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-02-26 11:42:55 +08:00
smellthemoon
9815cf50c9
fix: close rocksmq and natsmq when shutdown (#30706)
related: #30703 #30355

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-02-26 11:38:55 +08:00
wei liu
befe0e21fd
fix: Set indexInfo when try to set segment to leader view (#30758)
issue: #30150
see also: #30258

cause `SyncDataDistribution` will try to load delta for segment. if miss
indexInfo in request, sync action will failed due to lack of index info.

This PR set indexinfo when try to set segment to leader view

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-02-26 11:02:55 +08:00
Xiaofan
d970a9fc6c
enhance: refine error log of json parse error (#30671)
add more error info when json format does not match.
relate to https://github.com/milvus-io/milvus/issues/29837

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-02-24 12:32:14 +08:00