Commit Graph

438 Commits

Author SHA1 Message Date
Chun Han
ac82cef04d
enhance: disable reload partstats by config (#32702)
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-04-29 19:11:26 +08:00
wei liu
c0555d4b45
fix: Remove read only node from replica immedaitely after node down (#32666)
issue: #32665

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-28 20:25:25 +08:00
chyezh
2586c2f1b3
enhance: use WalkWithPrefix api for oss, enable piplined file gc (#31740)
issue: #19095,#29655,#31718

- Change `ListWithPrefix` to `WalkWithPrefix` of OOS into a pipeline
mode.

- File garbage collection is performed in other goroutine.

- Segment Index Recycle clean index file too.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-25 20:41:27 +08:00
Jiquan Long
c002745902
enhance: retrieve output fields after local reduce (#32346)
issue: #31822

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-04-25 09:49:26 +08:00
congqixia
faa559592d
enhance: Make applyDelete work in paralell in segment level (#32291)
`applyDelete` used to be serial for delete entries on each segments.
This PR make it work in parallel with errgroup to improve performance

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-24 17:01:24 +08:00
yihao.dai
281a583eda
fix: Correct the negative queryable num entities metric (#32361)
issue: https://github.com/milvus-io/milvus/issues/32281

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-04-24 15:55:24 +08:00
Cai Yudong
16b8b7b35d
enhance: Add get_vector unittest for float16 & bfloat16 (#32153)
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-23 16:15:23 +08:00
chyezh
e19d17076f
fix: delete may lost when enable lru cache, some field should be reset when ReleaseData (#32012)
issue: #30361

- Delete may be lost when segment is not data-loaded status in lru
cache. skip filtering to fix it.

- `stats_` and `variable_fields_avg_size_` should be reset when
`ReleaseData`

- Remove repeat load delta log operation in lru.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-16 11:17:20 +08:00
Gao
55d894bd5e
enhance: support disable search optimization (#32141)
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-04-16 10:51:20 +08:00
SimFG
c012e6786f
feat: support rate limiter based on db and partition levels (#31070)
issue: https://github.com/milvus-io/milvus/issues/30577
co-author: @jaime0815

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-04-12 16:01:19 +08:00
wei liu
68dec7dcd4
fix: Use correct ts to avoid exclude segment list leak (#31991)
issue: #31990

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-12 10:39:19 +08:00
wei liu
1a98ce39f5
enhance: Remove useless logic about FromShardLeader (#32029)
issue: #32047

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-10 20:11:19 +08:00
Xiaofan
dbab9c5096
fix: reduce didn't handle offset without limit and reduceStopForBest correctly (#32089)
fix https://github.com/milvus-io/milvus/issues/32059

this pr fix two issues:

offset is not handled correctly without specify a limit
reduceStopForBest doesn't guarantee to return limit result even if there
are more result when there is small segment

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-04-10 16:01:18 +08:00
wei liu
df208d538c
fix: Check exclude segment before add new growing segment (#31803)
issue: #31479 #31797

milvus will add released segment to excluded info, and filter out it's
stream data in filter_node. but for data buffered in insert_node's
channel, if it belongs to growing segment which already be released,
then it will all the growing segment back again.

This PR maintain `excluded segments` in delegator, and check excluded
segment before new growing segment.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-10 15:29:17 +08:00
Chun Han
f3f2a5a7e9
fix: evicted segments in the serverlss mode(#31959) (#31961)
related: #31959
1. reset segment index status after evicting to lazyload=true
2. reset num_rows to null_opt

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-04-10 15:15:19 +08:00
SimFG
90bed1caf9
enhance: add the related data size for the read apis (#31816)
issue: #30436
origin pr: #30438
related pr: #31772

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-10 15:07:17 +08:00
chyezh
c9faa6d936
enhance: add more metrics for cache and search (#31777)
issue: #30931

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-10 10:55:17 +08:00
aoiasd
5b693c466d
fix: delegator filter out all partition's delete msg when loading segment (#31585)
May cause deleted data queryable a period of time.
relate: https://github.com/milvus-io/milvus/issues/31484
https://github.com/milvus-io/milvus/issues/31548

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-04-09 15:21:24 +08:00
zhenshan.cao
089c805e0a
enhance:Refactor hybrid search (#32020)
issue: https://github.com/milvus-io/milvus/issues/25639
https://github.com/milvus-io/milvus/issues/31368

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-04-09 14:21:18 +08:00
congqixia
1f7f3993a1
fix: Validate PlaceholderGroups before combine them (#32016)
See also #32015

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-09 11:33:17 +08:00
chyezh
73adf2a5cc
fix: use stateful lock to avoid load and release on LocalSegment concurrently (#31606)
issue: #31605

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-08 17:09:16 +08:00
chyezh
7b400252ff
fix: add configuration disk capacity config for lru and fix some bug (#31977)
issue: #30361

- Add configurable disk capacity limit

- fix bitset reset logic

- make insert record reinsert after clear

Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-08 15:55:16 +08:00
congqixia
0feee53631
enhance: Add back unit test for compactor and fix some TODOs (#31829)
This PR adds back compactor "Unhandled" data type unit test and fixes
some TODOs behvaior

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-02 20:35:14 +08:00
jaime
bd853be8c7
enhance: Add db label for some usual metrics (#30956)
issue: #31782

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-04-02 14:27:13 +08:00
wei liu
bb500d66c7
fix: Remove segment from leader view can't be executed (#31663)
issue: #31664

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-01 10:39:12 +08:00
wei liu
c311932d5f
fix: Update segment's version in leader task (#31643)
issue: #31468

1. when segment's version in leader view doesn't match segment's version
in dist, should update leader view
2. after call loadDeltalog, should update segment's load version with
latest ts
3. change leader task's priority from high to low, to avoid leader task
replace segment task and balance task

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-01 10:37:21 +08:00
chyezh
1ad5ccc50f
enhance: add rg and db interface for segment and db/rg metric label (#31715)
issue: #30931

Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-01 10:21:21 +08:00
cqy123456
976928ecd1
fix: fix fp16/bf16 some code missing and add more fp16/bf16 test (#31612)
issue: #31534

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-03-28 14:11:10 +08:00
Bingyi Sun
3d66670619
fix: add lazy load field to mark segment load type (#31591)
issue: https://github.com/milvus-io/milvus/issues/31673

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-28 11:23:10 +08:00
SimFG
b1a1cca10b
feat: add more operation detail info for better allocation (#30438)
issue: #30436

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-03-28 06:33:11 +08:00
Bingyi Sun
fbff46a005
enhance: add lazyload global config (#31610)
Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-27 20:23:10 +08:00
Jiquan Long
4eb4df1e81
fix: predict inverted index resource usage more reasonably (#31615)
/kind improvement
issue: #31617

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-03-27 19:33:09 +08:00
Chun Han
bd44bd5ae2
enhance: add default value config for segment prune filterRatio(#31003) (#31580)
related: #31003

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-03-27 16:13:10 +08:00
John
ced7629d7d
doc: remove repetitive words (#31105)
Signed-off-by: hishope <csqiye@126.com>
2024-03-25 17:27:09 +08:00
Bingyi Sun
8e661f791a
fix: lazy load index data in cache (#31094)
issue: https://github.com/milvus-io/milvus/issues/31571

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-25 15:43:07 +08:00
Bingyi Sun
0c868b654d
fix: fix mmap enabled check in resource estimation (#31536)
issue: https://github.com/milvus-io/milvus/issues/31569

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-25 14:29:07 +08:00
congqixia
7b38274cce
fix: Use mutex protection when accessing pks of l0 segments (#31505)
See also #31504

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-22 14:11:06 +08:00
Chun Han
c3264ca3e3
feat: support segment pruner (#31003)
related: #30376
2024-03-22 13:57:06 +08:00
congqixia
d90e01532f
enhance: Release level zero segments when channel unsub (#31486)
Related to #27349
See also #30816

Level zero is not allowed to balance among delegators, they shall always
serve current delegator. This PR releases all level zero segments after
channel is unsubscribed and preventing level zero segment blocking
graceful stop.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-22 10:27:17 +08:00
Ted Xu
09281a07f4
enhance: disk cache to enable size based eviction (#31469)
See #31262

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-03-21 18:57:07 +08:00
Ted Xu
c13c96e321
feat: adding cache scavenger (#31264)
See #31262

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-03-21 14:33:06 +08:00
chyezh
9f9ef8ac32
enhance: transfer resource group and dbname to querynode when load (#30936)
issue: #30931

Signed-off-by: chyezh <chyezh@outlook.com>
2024-03-21 11:59:12 +08:00
groot
c81909bfab
enhance: Support MinIO TLS connection (#31311)
issue: https://github.com/milvus-io/milvus/issues/30709
pr: #31292

Signed-off-by: yhmo <yihua.mo@zilliz.com>
Co-authored-by: Chen Rao <chenrao317328@163.com>
2024-03-21 11:15:20 +08:00
Bingyi Sun
0fe2ac4c3a
fix: fix mmap enable log (#31439)
issue: https://github.com/milvus-io/milvus/issues/31443

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-20 19:17:05 +08:00
congqixia
a647b84f3e
enhance: Add AllPartitionsID const to replace InvalidPartitionID (#31438)
"-1" as `InvalidPartitionID` previously used as All partition place
holder in delete cases. It's confusing and hard to maintain when a const
var has more than one meaning.

This PR add `AllPartitionsID` to replace these usages in delete
scenarios.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-20 19:01:05 +08:00
Xiaofan
a63b4cedcf
fix: remove some unnecessary unrecoverable errors (#31327)
use retry.handle when request is not able to service but don't throw
unrecoverable erros
fix #31323

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-03-20 11:35:07 +08:00
Jiquan Long
dc2cdbe387
enhance: add more metrics (#31271)
/kind improvement
fix: #31272 

This pr add more metrics, which are:
- Slow query count, which the duration considered as slow can be
configurable;
- Number of deleted entities;
- Number of entities imported;
- Number of entities per collection;
- Number of loaded entities per collection;
- Number of indexed entities;
- Number of indexed entities, per collection, per index and whether it's
a vetor index;
- Quota states (LongTimeTickDelay, MemoryExhuasted, DiskQuotaExhuasted)
per database;

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-03-19 15:23:06 +08:00
chyezh
8e293dc1ce
enhance: add resource usage estimate for segment interface (#31050)
issue: #30931

- move resource estimate function outside from segment loader.
- add load info and collection to base segment.
- add resource usage method for sealed segment.

Signed-off-by: chyezh <chyezh@outlook.com>
2024-03-19 11:53:05 +08:00
Bingyi Sun
bdc70dfc6a
feat: Add global mmap enable configuration (#31267)
https://github.com/milvus-io/milvus/issues/31279

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-03-18 15:17:10 +08:00
congqixia
243e311515
fix: Report offline info in GetDataDistribition (#31347)
See also #31345

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-03-18 14:51:04 +08:00