Bingyi Sun
370562b4ec
fix: fix partition loaded num metric ( #33316 )
...
issue: https://github.com/milvus-io/milvus/issues/32108
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-05-24 15:31:42 +08:00
chyezh
1c84a1c9b6
fix: lru related issue fixup patch ( #32916 )
...
issue: #32206 , #32801
- search failure with some assertion, segment not loaded and resource
insufficient.
- segment leak when query segments
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-05-10 19:17:30 +08:00
congqixia
40728ce83d
enhance: Add metautil.Channel to convert string compare to int ( #32749 )
...
See also #32748
This PR:
- Add `metautil.Channel` utiltiy which convert virtual name to physical
channel name, collectionID and shard idx
- Add channel mapper interface & implementation to convert limited
physical channel name into int index
- Apply `metautil.Channel` filter in querynode segment manager logic
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-05-07 19:13:35 +08:00
chyezh
641f702f64
fix: add request resource timeout for lazy load, refactor context usage in cache ( #32709 )
...
issue: #32663
- Use new param to control request resource timeout for lazy load.
- Remove the timeout parameter of `Do`, remove `DoWait`. use `context`
to control the timeout.
- Use `VersionedNotifier` to avoid notify event lost and broadcast,
remove the redundant goroutine in cache.
related dev pr: #32684
Signed-off-by: chyezh <chyezh@outlook.com>
2024-05-07 16:33:30 +08:00
Bingyi Sun
fecd9c21ba
feat: LRU cache implementation ( #32567 )
...
issue: https://github.com/milvus-io/milvus/issues/32783
This pr is the implementation of lru cache on branch lru-dev.
Signed-off-by: sunby <sunbingyi1992@gmail.com>
Co-authored-by: chyezh <chyezh@outlook.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
2024-05-06 20:29:30 +08:00
chyezh
c9faa6d936
enhance: add more metrics for cache and search ( #31777 )
...
issue: #30931
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-10 10:55:17 +08:00
chyezh
73adf2a5cc
fix: use stateful lock to avoid load and release on LocalSegment concurrently ( #31606 )
...
issue: #31605
---------
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-08 17:09:16 +08:00
chyezh
7b400252ff
fix: add configuration disk capacity config for lru and fix some bug ( #31977 )
...
issue: #30361
- Add configurable disk capacity limit
- fix bitset reset logic
- make insert record reinsert after clear
Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-08 15:55:16 +08:00
Bingyi Sun
8e661f791a
fix: lazy load index data in cache ( #31094 )
...
issue: https://github.com/milvus-io/milvus/issues/31571
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-25 15:43:07 +08:00
Ted Xu
09281a07f4
enhance: disk cache to enable size based eviction ( #31469 )
...
See #31262
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-03-21 18:57:07 +08:00
Ted Xu
c13c96e321
feat: adding cache scavenger ( #31264 )
...
See #31262
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-03-21 14:33:06 +08:00
congqixia
57a5e44b88
enhance: Add id,type filter shortcut for QueryNode segment manager ( #31179 )
...
See also #31143
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-13 20:15:03 +08:00
Bingyi Sun
7b7187b465
fix: fix mmap load ( #31171 )
...
issue: https://github.com/milvus-io/milvus/issues/31101
---------
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-13 10:49:04 +08:00
yah01
57397b1307
enhance: add new LRU cache impl ( #30360 )
...
- remove the unused LRU cache
- add new LRU cache impl which wraps github.com/karlseguin/ccache
related #30361
---------
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-02-27 20:58:40 +08:00
yah01
d357139064
fix: the entities num metric may be contributed more than once ( #29767 )
...
the growing segments contribute to this metric while inserting and
putting into the manager, but the current impl inserts data before
putting the segments into manager, which leads to double contributions
fix : #29766
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2024-01-10 10:00:51 +08:00
yah01
a0cec4047a
fix: make the entity num metric accurate ( #29643 )
...
fix #29642
Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-05 18:24:47 +08:00
yah01
9b3e06ae86
enhance: add more metrics for level zero segments ( #29029 )
...
- Add SegmentNum metric for level zero segments
- Add level zero segments size metirc
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-07 14:34:35 +08:00
yah01
cc952e0486
enhance: optimize forwarding level0 deletions by respecting partition ( #28456 )
...
- Cache the level 0 deletions after loading level0 segments
- Divide the level 0 deletions by partition
related: #27349
---------
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-21 18:24:22 +08:00
aoiasd
13a5b9f64a
fix: query l0 segment bugs ( #28558 )
...
relate: https://github.com/milvus-io/milvus/issues/27675
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-11-20 17:26:23 +08:00
yah01
ece592a42f
Deliver L0 segments delete records ( #27722 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-07 01:44:18 +08:00
yah01
50672f0d60
Skip empty segment while query/search ( #27711 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-15 16:10:04 +08:00
SimFG
26f06dd732
Format the code ( #27275 )
...
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
congqixia
cc9974979f
Add staticcheck linter and fix existing problems ( #27174 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-19 10:05:22 +08:00
congqixia
c6116d1819
Remove segment to LocalSegment type assertion ( #26931 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-09 10:35:16 +08:00
congqixia
95dcf7fa06
Fix memory leak when Put duplicated segments ( #26693 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-30 11:37:00 +08:00
congqixia
1cf6e00fa6
Improve segment manager interface ( #26637 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-29 15:46:27 +08:00
Xiaofan
7b3a1dc93f
Fix load index panic ( #26457 )
...
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-08-21 09:56:20 +08:00
wei liu
3b11ad2695
fix dead lock in concurrent search/release segment ( #26435 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-18 15:28:18 +08:00
yah01
5dae6a654a
Protect segment from being released while query/search ( #26322 )
...
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-08-16 18:38:17 +08:00
wei liu
b47a72bfcf
fix set dirty segment distribution to leader view ( #26180 )
...
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-08-11 11:21:32 +08:00
congqixia
0bc03ede0d
Add eventlog pkg and support grpc streaming event observation ( #25812 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-07-25 17:23:01 +08:00
yah01
d216f9abda
Clear collection meta after all channels/segments released ( #25486 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-07-14 10:28:30 +08:00
yah01
7b5013664d
Support graceful stop ( #25226 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-30 17:28:23 +08:00
yah01
400364483d
Improve predict the resource usage ( #23968 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-05-11 15:33:24 +08:00
jaime
c9d0c157ec
Move some modules from internal to public package ( #22572 )
...
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
yah01
081572d31c
Refactor QueryNode ( #21625 )
...
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: aoiasd <zhicheng.yue@zilliz.com>
2023-03-27 00:42:00 +08:00
