tinswzy
1dbb6cd7cb
enhance: refine the datacoord meta related interfaces ( #37957 )
...
Code Checker / Code Checker AMD64 Ubuntu 22.04 (push) Waiting to run
Code Checker / Code Checker Amazonlinux 2023 (push) Waiting to run
Code Checker / Code Checker rockylinux8 (push) Waiting to run
Mac Code Checker / Code Checker MacOS 13 (push) Waiting to run
Build and test / Build and test AMD64 Ubuntu 22.04 (push) Waiting to run
Build and test / UT for Cpp (push) Blocked by required conditions
Build and test / UT for Go (push) Blocked by required conditions
Build and test / Integration Test (push) Blocked by required conditions
Build and test / Upload Code Coverage (push) Blocked by required conditions
issue: #35917
This PR refines the meta-related APIs in datacoord to allow the ctx to
be passed down to the catalog operation interfaces
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-11-26 19:46:34 +08:00
SimFG
2208b7c2ef
fix: the too long default root password does not take effect ( #37983 )
...
- issue: #36987
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-11-26 17:24:35 +08:00
Zhen Ye
2b4f211d84
enhance: add switch for local rpc enabled ( #37985 )
...
issue: #33285
- Add switch for local rpc
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-26 17:00:54 +08:00
aoiasd
db3c002788
fix:wrong warning log when idf oracle find L0 segment lacks ( #37873 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-11-26 16:26:41 +08:00
congqixia
cb6542339e
enhance: Mark cgo thread with tag name ( #38000 )
...
Related to #37999
This PR add `SetThreadName` API for marking cgo thread and utilize it
when initializing cgo worker.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-26 11:22:35 +08:00
tinswzy
e76802f910
enhance: refine querycoord meta/catalog related interfaces to ensure that each method includes a ctx parameter ( #37916 )
...
issue: #35917
This PR refine the querycoord meta related interfaces to ensure that
each method includes a ctx parameter.
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-11-25 11:14:34 +08:00
Zhen Ye
fbb68ca370
enhance: make all index operation async scheduled by tokio ( #37946 )
...
issue: #37851
related pr: https://github.com/milvus-io/tantivy/pull/3
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-25 10:12:34 +08:00
sre-ci-robot
ed73dfca3f
[automated] Update Knowhere Commit ( #37965 )
...
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-11-25 01:28:32 +08:00
Yinzuo Jiang
5a06faca39
feat: fp32 vector to fp16/bf16 vector conversion for RESTful API ( #37556 )
...
RESTful API. The influenced API are as follows:
- Handler. insert
- HandlerV1. insert/upsert
- HandlerV2. insert/upsert/search
We do not modify search API in Handler/HandlerV1 because they do not
support fp16/bf16 vectors.
module github.com/milvus-io/milvus/pkg:
Add `Float32ArrayToBFloat16Bytes()`, `Float32ArrayToFloat16Bytes()` and
`Float32ArrayToBytes()`. These method will be used in GoSDK in the
future.
issue: #37448
Signed-off-by: Yinzuo Jiang <yinzuo.jiang@zilliz.com>
Signed-off-by: Yinzuo Jiang <jiangyinzuo@foxmail.com>
2024-11-24 17:46:33 +08:00
zhagnlu
62af24c1a1
fix: change search latency metric from us unit to ms unit ( #37806 )
...
#37805
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-11-24 17:26:33 +08:00
congqixia
1e76d2b9f1
fix: Use correct policy merging growing&l0 and add unit tests ( #37950 )
...
Related to #37574
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-24 17:10:33 +08:00
Zhen Ye
b9a10a2f68
enhance: remove the rpc layer of coordinator when enabling standalone or mixcoord ( #37815 )
...
issue: #37764
- add a local client to call local server directly for
querycoord/rootcoord/datacoord.
- enable local client if milvus is running mixcoord or standalone mode.
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-23 21:36:33 +08:00
sthuang
bd0811434b
fix: fix grant v2 service ( #37945 )
...
issue: https://github.com/milvus-io/milvus/issues/37031
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-22 19:42:32 +08:00
smellthemoon
df08d91c0f
enhance: error msg ( #37901 )
...
#37900
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-11-22 18:40:32 +08:00
jaime
7bbfe86bcd
enhance: add list index and segment index retrieval API for WebUI ( #37861 )
...
issue: #36621
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-22 16:58:34 +08:00
SimFG
7c5a8012cf
enhance: remove the collectionBasicInfo class in the proxy metacache ( #37874 )
...
/kind improvement
- issue: #37928
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-11-22 16:10:33 +08:00
Bingyi Sun
700a448a54
fix: Escape prefix before search in inverted index ( #37925 )
...
issue: https://github.com/milvus-io/milvus/issues/37912
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-22 14:10:33 +08:00
Bingyi Sun
06d73cf2e2
enhance: Remove raw tokenizer register. ( #37886 )
...
tantivy already register raw tokenizer by default
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-11-22 12:02:32 +08:00
congqixia
b34bfb98a0
enhance: Refine Replica manager colle2Replicas secondary index ( #37906 )
...
Related to #37630
This PR add a new util coll2Replicas secondary index to reduce map
access & iteration while get replicas by collection
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-22 11:54:32 +08:00
congqixia
1ed686783f
enhance: Use PrimaryKeys to replace interface slice for segment delete ( #37880 )
...
Related to #35303
Reduce temporary memory usage for PK interface for segment delete.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-22 11:52:33 +08:00
SimFG
923a661dfe
enhance: filter the fields instead of create a new response obj ( #37845 )
...
/kind improvement
Here you only need to filter out the system fields, and you don’t need
to recreate a response, because recreating the response will cause this
part to be easily missed when adding fields later.
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-11-22 11:42:32 +08:00
congqixia
cfa1f1f141
enhance: Add thread watcher to provide actual thread num ( #37905 )
...
Related to #37904
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-22 10:20:33 +08:00
congqixia
92e6ee6285
enhance: Use load pool for CreateTextIndex ( #37898 )
...
Related to #37895
Only resolves the starving issue which caused goroutine leakage
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-22 10:06:33 +08:00
SimFG
54aaeda63f
fix: add the request ctx for stream pipeline interface ( #37835 )
...
- issue: #37834
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-11-22 03:10:34 +08:00
sthuang
19572f5b06
enhance: RBAC new grant/revoke privilege ( #37785 )
...
issue: https://github.com/milvus-io/milvus/issues/37031
also fix issues: https://github.com/milvus-io/milvus/issues/37843 ,
https://github.com/milvus-io/milvus/issues/37842 ,
https://github.com/milvus-io/milvus/issues/37887
Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-11-21 22:20:34 +08:00
wei liu
5f3601a6a5
fix: unstable integration test caused by paramtable.GetNodeID ( #37909 )
...
issue: #37908
cause paramtable is global single instance, which cause
paramtable.GetNodeID may return wrong server id in integration test.
This PR use node.GetNodeID to replace paramtable.GetNodeID
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-21 22:16:33 +08:00
Zhen Ye
1dc1a97e65
fix: use different thread pool for scheduler and merger ( #37911 )
...
issue: #37895
related pr: https://github.com/milvus-io/tantivy/pull/2
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-21 21:34:33 +08:00
MimoN
3cb9511c57
fix: update webui filters ( #37885 )
...
issue: #36621
- fix the empty segment source
- remove http request timeout limit
- support more filters in seg and channel table
Signed-off-by: mimo.oyn <ning.ouyang@zilliz.com>
2024-11-21 20:04:33 +08:00
tinswzy
e247ff9ee7
enhance: refine rootcoord/metatable interfaces to ensure that each method includes a ctx parameter ( #37846 )
...
issue: #35917
Before enhancing log trace information, it's necessary to pass the
context to the method entry point.
This PR first refine the rootcoord/metatable interfaces to ensure that
each method includes a ctx parameter.
Signed-off-by: tinswzy <zhenyuan.wei@zilliz.com>
2024-11-21 19:30:32 +08:00
wei liu
965bda6e60
enhance: Add channel name to shard leader log in meta cache ( #37856 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-21 19:24:31 +08:00
wei liu
97a44b62fd
fix: Data race in datacoord channel manager ( #37866 )
...
issue: #37865
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-21 19:00:32 +08:00
XuanYang-cn
d7dcc752f1
enhance: Increase task capacity and clean illegal task ( #37896 )
...
1. taskQueueCapacity 256 is too small for production when we want to
re-write the entire collection
2. tasks should be cleaned when unable to recover, or the meta will
remain in etcd forever later.
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-11-21 18:48:32 +08:00
wei liu
0a440e0d38
fix: Prevent simultaneous balance of segments and channels ( #37850 )
...
issue: #33550
balance segment and balance segment execute at same time, which will
cause bounch of corner case.
This PR disable simultaneous balance of segments and channels
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-21 17:56:55 +08:00
wei liu
2159526e80
fix: Deadlock in compaction handler ( #37868 )
...
issue: #37803
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-21 17:26:46 +08:00
XuanYang-cn
70e6a00ba1
fix: Replace outer lock with concurrent map ( #37817 )
...
See also: #37493
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-11-21 16:08:33 +08:00
wei liu
7817ea6b8b
enhance: Remove unnecessary stack trace in error ( #37816 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-21 15:10:32 +08:00
cai.zhang
c07f056b17
fix: Use the ID to retrieve the real name when collectionName is empty ( #37859 )
...
issue: #36989
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-11-21 14:28:32 +08:00
wei liu
b983ef9fca
fix: Channel may be released after balance ( #37862 )
...
Code Checker / Code Checker AMD64 Ubuntu 22.04 (push) Has been cancelled
Code Checker / Code Checker Amazonlinux 2023 (push) Has been cancelled
Code Checker / Code Checker rockylinux8 (push) Has been cancelled
Mac Code Checker / Code Checker MacOS 13 (push) Has been cancelled
Build and test / Build and test AMD64 Ubuntu 22.04 (push) Has been cancelled
Update Knowhere Commit / update-knowhere-commit (push) Has been cancelled
Build and test / UT for Cpp (push) Has been cancelled
Build and test / UT for Go (push) Has been cancelled
Build and test / Integration Test (push) Has been cancelled
Build and test / Upload Code Coverage (push) Has been cancelled
issue: #37830
casue dist handler doesn't set channel's version, so if channel checker
try to dedup channel, it may release the new delegator after balance
finished.
this PR fix the way to set proper version for channel.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-11-21 10:40:31 +08:00
Zhen Ye
f3a36f8a29
fix: use global pool but not dedicated pool for every index ( #37852 )
...
Code Checker / Code Checker AMD64 Ubuntu 22.04 (push) Waiting to run
Code Checker / Code Checker Amazonlinux 2023 (push) Waiting to run
Code Checker / Code Checker rockylinux8 (push) Waiting to run
Mac Code Checker / Code Checker MacOS 13 (push) Waiting to run
Build and test / Build and test AMD64 Ubuntu 22.04 (push) Waiting to run
Build and test / UT for Cpp (push) Blocked by required conditions
Build and test / UT for Go (push) Blocked by required conditions
Build and test / Integration Test (push) Blocked by required conditions
Build and test / Upload Code Coverage (push) Blocked by required conditions
all-contributors / contributor (push) Has been cancelled
issue: #37851
- make a global thread pool at tantivy temporally.
- set 1 but not 4 threads for inverted text index.
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-20 20:44:32 +08:00
Buqian Zheng
511edd29fd
enhance: disallow get raw vector data of a BM25 Function output field ( #37800 )
...
issue: https://github.com/milvus-io/milvus/issues/35853
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-11-20 14:22:30 +08:00
congqixia
b8d31ebed8
enhance: Remove unnecessary segment clone updating dist ( #37797 )
...
Related to #37630
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-20 11:26:31 +08:00
nish112022
484c6b5c44
feat: Added code for Internal-tls ( #36865 )
...
issue : https://github.com/milvus-io/milvus/issues/36864
I have a few questions regarding my approach.I will consolidate them
here for feedback and review.Thanks
---------
Signed-off-by: Nischay Yadav <nischay.yadav@ibm.com>
Signed-off-by: Nischay <Nischay.Yadav@ibm.com>
2024-11-20 06:00:32 +08:00
congqixia
83c902992c
enhance: Prevent generate "null" search params ( #37812 )
...
Code Checker / Code Checker AMD64 Ubuntu 22.04 (push) Waiting to run
Code Checker / Code Checker Amazonlinux 2023 (push) Waiting to run
Code Checker / Code Checker rockylinux8 (push) Waiting to run
Mac Code Checker / Code Checker MacOS 12 (push) Waiting to run
Build and test / Build and test AMD64 Ubuntu 22.04 (push) Waiting to run
Build and test / UT for Cpp (push) Blocked by required conditions
Build and test / UT for Go (push) Blocked by required conditions
Build and test / Integration Test (push) Blocked by required conditions
Build and test / Upload Code Coverage (push) Blocked by required conditions
Preventing generating null search params in restful search request
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-19 19:54:32 +08:00
yihao.dai
b3fc53015d
fix: Avoid memory copy in RESTful search and query ( #37674 )
...
Code Checker / Code Checker AMD64 Ubuntu 22.04 (push) Waiting to run
Code Checker / Code Checker Amazonlinux 2023 (push) Waiting to run
Code Checker / Code Checker rockylinux8 (push) Waiting to run
Mac Code Checker / Code Checker MacOS 12 (push) Waiting to run
Build and test / Build and test AMD64 Ubuntu 22.04 (push) Waiting to run
Build and test / UT for Cpp (push) Blocked by required conditions
Build and test / UT for Go (push) Blocked by required conditions
Build and test / Integration Test (push) Blocked by required conditions
Build and test / Upload Code Coverage (push) Blocked by required conditions
Publish Test Images / PyTest (push) Has been cancelled
Custom `jsonRender` that encodes JSON data directly into the response
stream, it uses less memory since it does not buffer the entire JSON
structure before sending it, unlike `c.JSON` in `HTTPReturn`, which
serializes the JSON fully in memory before writing it to the response.
Benchmark testing shows that, using the custom render incurs no
performance loss and reduces memory consumption by nearly 50%.
issue: https://github.com/milvus-io/milvus/issues/37671
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-11-19 17:24:31 +08:00
SimFG
3bbbf3cd95
enhance: improve the lock key of the create/drop collection task ( #37759 )
...
/kind improvement
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-11-19 15:50:32 +08:00
cqy123456
8216345b07
enhance: reduce copy of bitset and id conversion of brurtforce search ( #37675 )
...
issue: https://github.com/milvus-io/milvus/issues/37798
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-11-19 15:48:40 +08:00
yihao.dai
b6612e02b4
enhance: Reduce GetIndexInfos calls ( #37695 )
...
Batch `GetIndexInfos` calls for segments to reduce RPC calls.
issue: https://github.com/milvus-io/milvus/issues/37634
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-11-19 14:24:31 +08:00
cai.zhang
dae4160466
enhance: Whether to enable mergeSort mode when performing mixCompaction ( #37664 )
...
issue: #37579
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-11-19 11:28:31 +08:00
Xiaofan
33bfb25c73
enhance: refine meta cache log and logic ( #37318 )
...
Code Checker / Code Checker AMD64 Ubuntu 22.04 (push) Waiting to run
Code Checker / Code Checker Amazonlinux 2023 (push) Waiting to run
Code Checker / Code Checker rockylinux8 (push) Waiting to run
Mac Code Checker / Code Checker MacOS 12 (push) Waiting to run
Build and test / Build and test AMD64 Ubuntu 22.04 (push) Waiting to run
Build and test / UT for Cpp (push) Blocked by required conditions
Build and test / UT for Go (push) Blocked by required conditions
Build and test / Integration Test (push) Blocked by required conditions
Build and test / Upload Code Coverage (push) Blocked by required conditions
related to #36989
add more logs in proxy meta cache and make it clearer
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-11-19 10:50:33 +08:00
congqixia
c79fbd5eab
fix: Load l0 delta for growings when using RemoteLoad ( #37771 )
...
Related to #37574
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-11-19 10:38:31 +08:00
