Zhen Ye
fca946dee1
enhance: move the search utilities from querynode into new module ( #37531 )
...
issue: #33285
- The search utilities will be shared between query node and streaming
node.
Signed-off-by: chyezh <chyezh@outlook.com>
2024-11-11 11:10:27 +08:00
jaime
f348bd9441
feat: add segment,pipeline, replica and resourcegroup api for WebUI ( #37344 )
...
issue: #36621
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-11-07 11:52:25 +08:00
jaime
9d16b972ea
feat: add tasks page into management WebUI ( #37002 )
...
issue: #36621
1. Add API to access task runtime metrics, including:
- build index task
- compaction task
- import task
- balance (including load/release of segments/channels and some leader
tasks on querycoord)
- sync task
2. Add a debug model to the webpage by using debug=true or debug=false
in the URL query parameters to enable or disable debug mode.
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-10-28 10:13:29 +08:00
Yinzuo Jiang
3628593d20
feat: Implement custom function module in milvus expr ( #36560 )
...
OSPP 2024 project:
https://summer-ospp.ac.cn/org/prodetail/247410235?list=org&navpage=org
Solutions:
- parser (planparserv2)
- add CallExpr in planparserv2/Plan.g4
- update parser_visitor and show_visitor
- grpc protobuf
- add CallExpr in plan.proto
- execution (`core/src/exec`)
- add `CallExpr` `ValueExpr` and `ColumnExpr` (both logical and
physical) for function call and function parameters
- function factory (`core/src/exec/expression/function`)
- create a global hashmap when starting milvus (see server.go)
- the global hashmap stores function signatures and their function
pointers, the CallExpr in execution engine can get the function pointer
by function signature.
- custom functions
- empty(string)
- starts_with(string, string)
- add cpp/go unittests and E2E tests
closes : #36559
Signed-off-by: Yinzuo Jiang <jiangyinzuo@foxmail.com>
2024-10-25 15:25:30 +08:00
yihao.dai
763fd0dfc5
enhance: Use a separate mmap config for chunk cache ( #36276 )
...
issue: https://github.com/milvus-io/milvus/issues/35273
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-15 16:23:09 +08:00
aoiasd
c22a2cebb6
fix: split stream query result to avoid grpc response too large error ( #36090 )
...
relate: https://github.com/milvus-io/milvus/issues/36089
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-13 15:07:09 +08:00
Zhen Ye
a773836b89
enhance: optimize milvus core building ( #35610 )
...
issue: #35549,#35611,#35633
- remove milvus_segcore milvus_indexbuilder..., add libmilvus_core
- core building only link once
- move opendal compilation into cmake
- fix odr
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-23 12:35:02 +08:00
SimFG
731d45abbe
enhance: provide more general configuration to control mmap behavior ( #35359 )
...
- issue: #35273
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-08-21 00:22:54 +08:00
congqixia
c64a078458
enhance: Support proxy/delegator qn client pooling ( #35194 )
...
See also #35196
Add param item for proxy/delegator query node client pooling and
implement pooling logic
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-02 11:24:19 +08:00
zhenshan.cao
aa247f192d
enhance: remove unused code for StorageV2 ( #35132 )
...
issue: https://github.com/milvus-io/milvus/issues/34168
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-08-01 12:08:13 +08:00
cai.zhang
2372452fac
enhance: Optimized the GC logic to ensure that memory is released in time ( #34949 )
...
issue: #34703
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-28 23:53:47 +08:00
cqy123456
32f685ff12
enhance: growing segment support mmap ( #32633 )
...
issue: https://github.com/milvus-io/milvus/issues/32984
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-06-18 14:42:00 +08:00
aoiasd
59a7a46904
enhance: Merge query stream result for reduce delete task ( #32855 )
...
relate: https://github.com/milvus-io/milvus/issues/32854
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-05-27 18:15:43 +08:00
wei liu
39f56678a0
enhance: Reduce bloom filter lock contention between insert and delete in query coord ( #32643 )
...
issue: #32530
cause ProcessDelete need to check whether pk exist in bloom filter, and
ProcessInsert need to update pk to bloom filter, when execute
ProcessInsert and ProcessDelete in parallel, it will cause race
condition in segment's bloom filter
This PR execute ProcessInsert and ProcessDelete in serial to avoid block
each other
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-22 19:11:40 +08:00
wei liu
f1c9986974
enhance: Skip return data distribution if no change happen ( #32814 )
...
issue: #32813
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-05-17 10:11:37 +08:00
foxspy
f6777267e3
enhance: add score compute consistency config for knowhere ( #32997 )
...
issue: https://github.com/milvus-io/milvus/issues/32583
related: #32584
Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-05-13 14:21:31 +08:00
Bingyi Sun
fecd9c21ba
feat: LRU cache implementation ( #32567 )
...
issue: https://github.com/milvus-io/milvus/issues/32783
This pr is the implementation of lru cache on branch lru-dev.
Signed-off-by: sunby <sunbingyi1992@gmail.com>
Co-authored-by: chyezh <chyezh@outlook.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
2024-05-06 20:29:30 +08:00
wei liu
c0555d4b45
fix: Remove read only node from replica immedaitely after node down ( #32666 )
...
issue: #32665
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-28 20:25:25 +08:00
Bingyi Sun
8e661f791a
fix: lazy load index data in cache ( #31094 )
...
issue: https://github.com/milvus-io/milvus/issues/31571
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-25 15:43:07 +08:00
Bingyi Sun
0fe2ac4c3a
fix: fix mmap enable log ( #31439 )
...
issue: https://github.com/milvus-io/milvus/issues/31443
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-20 19:17:05 +08:00
jaime
db79be3ae0
fix: ctx cancel should be the last step while stopping server ( #31220 )
...
issue: #31219
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-03-15 10:33:05 +08:00
Chun Han
3298e64bd3
enhance: cache config values for saving cpu cycles to parse config item ( #30947 )
...
related: #30958
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-03-12 11:09:04 +08:00
MrPresent-Han
17a2fd048e
feat: support set up knowhere-build-pool-size on querynode( #29650 ) ( #30922 )
...
related: #29650
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-02-29 18:15:00 +08:00
chyezh
0c7474d7e8
enhance: add graceful stop timeout to avoid node stop hang under extreme cases ( #30317 )
...
1. add coordinator graceful stop timeout to 5s
2. change the order of datacoord component while stop
3. change querynode grace stop timeout to 900s, and we should
potentially change this to 600s when graceful stop is smooth
issue: #30310
also see pr: #30306
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-02-29 17:01:50 +08:00
chyezh
77477d6340
fix: wrong context passing into NewClient, error handling lost in session_util ( #30817 )
...
issue: #30799
Signed-off-by: chyezh <chyezh@outlook.com>
2024-02-28 10:40:09 +08:00
yiwangdr
c6665c2a4c
test: support multiple data/querynodes in integration test ( #30618 )
...
issue: https://github.com/milvus-io/milvus/issues/29507
Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-02-21 11:54:53 +08:00
Bingyi Sun
e1258b8cad
feat: integrate storagev2 into loading segment ( #29336 )
...
issue: #29335
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-01-12 18:10:51 +08:00
congqixia
b251c3a682
enhance: add ctx for HandleCStatus and callers ( #29517 )
...
See also #29516
Make `HandleCStatus` print trace id for better logging
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-27 16:10:47 +08:00
cqy123456
4c979538a4
enhance: update cagra index params in config and add params check ( #29045 )
...
issue:https://github.com/milvus-io/milvus/issues/29230
this pr do two things about cagra index:
a.milvus yaml config support gpu memory settings
b.add cagra-params check
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
Co-authored-by: yusheng.ma <yusheng.ma@zilliz.com>
2023-12-26 11:04:47 +08:00
SimFG
dd9c61831d
enhance: Support to get the param value in the runtime ( #29297 )
...
/kind improvement
issue: #29299
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-12-22 18:36:44 +08:00
yah01
1997fd150c
enhance: use local path as mmap file path if no mmap path provided ( #28992 )
...
the mmap mode isn't controlled by the config anymore, so we don't
require user to set it when they enabled mmap mode.
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-21 18:12:45 +08:00
wei liu
cb0676153f
enhance: add metrics for stopping querynode balance progress ( #29201 )
...
This PR add three metrics to track the stopping balance progress.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-21 10:14:52 +08:00
zhagnlu
a602171d06
enhance: Refactor runtime and expr framework ( #28166 )
...
#28165
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2023-12-18 12:04:42 +08:00
yihao.dai
b4353ca4ce
enhance: Remove vector chunk manager ( #28569 )
...
We have implemented the chunkcache (in cpp) to retrieve vectors, hence
rendering the vectorchunkcache (in golang) obsolete.
issue: https://github.com/milvus-io/milvus/issues/28568
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-11-30 18:00:33 +08:00
SimFG
e3b7fdac61
Delay the cancellation of ctx when stopping the node ( #28247 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-11-08 03:20:17 +08:00
cqy123456
4fbe3c9142
replace loaded binlog with binlog index for search performance ( #27673 )
...
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2023-11-01 02:20:15 +08:00
Filip Haltmayer
6b1a106a31
Moving etcd client into session ( #27069 )
...
Signed-off-by: Filip Haltmayer <filip.haltmayer@zilliz.com>
2023-10-27 07:36:12 +08:00
congqixia
bcbe98aba1
Add querynode client wrapper and avoid grpc in standalone mode ( #27781 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-10-19 11:10:07 +08:00
jaime
ec1fe3549e
Add a stop hook to clean session ( #27564 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-10-16 10:24:10 +08:00
yihao.dai
7d0dd0047d
Use path.Join when init ChunkCache path ( #27433 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-30 17:05:27 +08:00
yihao.dai
106c17f304
Make read ahead policy in ChunkCache configurable ( #27291 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-28 15:47:27 +08:00
congqixia
258e1ccd66
Refine querynode scheduler lifetime ( #26915 )
...
This PR refines scheduler lifetime control:
- Move private tri-state into lifetime package
- Make scheduler block incoming "Add" task
- Make scheduler Stop wait until all previously accepted task done
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-28 10:21:26 +08:00
congqixia
8c59dba329
Refine queryHook mockery ( #27394 )
...
This PR move `QueryHook` interface to `optimizers` pkg
Update all mockery generated files to latest
Add makefile entry for `QueryHook`
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-28 10:01:26 +08:00
foxspy
5db4a0489e
dynamic index version control ( #27335 )
...
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
2023-09-25 21:39:27 +08:00
SimFG
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
yihao.dai
bb6711f28c
Add ChunkCache: support get vector from storage ( #26142 )
...
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-09-15 10:21:20 +08:00
Enwei Jiao
fb0705df1b
Decouple basetable and componentparam ( #26725 )
...
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-05 10:31:48 +08:00
congqixia
4b58c71908
Add ctx parameter for organizeTask and GetWorker method ( #26835 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-05 10:05:48 +08:00
Gao
7e36f819b4
Make autoindex config clean ( #26732 )
...
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-08-31 10:25:01 +08:00
wei liu
7af0f7d90c
avoid concurrent sub/unsub on same channel ( #26454 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-23 10:10:22 +08:00
