test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-05 05:18:52 +08:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity
20,008 Commits 96 Branches 105 Tags 186 MiB
4ebdccd160
Commit Graph

260 Commits

Author SHA1 Message Date
wei liu
c344083f22
enhance: Optimize grow slice cost during query (#34254) 
issue: #32252
pr: #34253

This PR try to pre-allocate FieldData for Reduce operations in the Query
chain using typeutil.PrepareResultFieldData to avoid the overhead of
dynamically growing the slice during appendFieldData process.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-01 15:18:11 +08:00
jaime
b37d6fa0f9
enhance: decrease cpu overhead during filter segments on datacoord (#34231) 
issue: https://github.com/milvus-io/milvus/issues/33129
pr: #33130 
pr: #33373

---------

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-01 10:20:08 +08:00
Bingyi Sun
460815ceab
fix: fix partition loaded num metric (#33316) (#34195) 
issue: https://github.com/milvus-io/milvus/issues/32108
related pr: #33316

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-06-28 14:58:05 +08:00
cqy123456
f5344abdaf
enhance: [cherry-pick]growing segment support mmap (#34110) 
issue: issue: https://github.com/milvus-io/milvus/issues/32984
related pr: https://github.com/milvus-io/milvus/pull/32633,
https://github.com/milvus-io/milvus/pull/33951,
https://github.com/milvus-io/milvus/pull/33993

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-25 14:52:07 +08:00
Jiquan Long
22e6807e9a
feat: support inverted index for array (#33452) (#34053) 
pr: https://github.com/milvus-io/milvus/pull/33184
pr: https://github.com/milvus-io/milvus/pull/33452
pr: https://github.com/milvus-io/milvus/pull/33633
issue: https://github.com/milvus-io/milvus/issues/27704
Co-authored-by: xiaocai2333 <cai.zhang@zilliz.com>

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: cai.zhang <cai.zhang@zilliz.com>
 2024-06-24 10:50:03 +08:00
congqixia
e02a95e3c2
fix: [2.4] Return record with largest timestamp for entires with same PK (#33936) (#34024) 
Cherry-pick from master
pr: #33936
See also #33883

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-21 14:14:01 +08:00
chyezh
dd6c982bdb
fix: load operation when segment is on releasing (#33699) 
issue: #30857
pr: #31340

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-14 11:47:57 +08:00
wei liu
25d8b74f71
enhance: Execute bloom filter apply in parallel to speed up segment predict (#33793) 
issue: #33610
pr: #33792

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-13 14:14:04 +08:00
wei liu
54feef30e7
enhance: Use BatchPkExist to reduce bloom filter func call cost (#33752) 
issue: #33610
pr: #33611

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-12 17:45:58 +08:00
chyezh
93348af5c0
fix: async warmup will be blocked by state lock (#33687) 
issue: #33685
pr: #33686

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-07 14:23:54 +08:00
Chun Han
627b787aed
fix: query iterator lack results(#33137) (#33422) (#33506) 
related: #33137 
pr: https://github.com/milvus-io/milvus/pull/33422

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-06-05 18:51:52 +08:00
yihao.dai
7384bfe3f8
fix: use seperate warmup pool and disable warmup by default (#33348) (#33349) 
1. use a small warmup pool to reduce the impact of warmup
2. change the warmup pool to nonblocking mode
3. disable warmup by default
4. remove the maximum size limit of 16 for the load pool

issue: https://github.com/milvus-io/milvus/issues/32772

pr: https://github.com/milvus-io/milvus/pull/33348

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: xiaofanluan <xiaofan.luan@zilliz.com>
 2024-05-28 19:27:43 +08:00
jaime
8990b8b051
fix: correct error of metrics stats (#33305) 
issue: #32980
cherry pick from master
pr:  #33075 #33255

---------

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-05-24 09:15:41 +08:00
Bingyi Sun
d4a146ef1a
enhance: mmap load raw data if scalar index does not have raw data (#… (#33317) 
pr: #33175

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-05-24 09:09:42 +08:00
Jiquan Long
d98e1f6ff5
fix: two-phase retrieval on lru-segment (#32945) (#33313) 
Cherry-pick from master
pr: #32945 
issue: #31822

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-05-23 16:25:40 +08:00
cai.zhang
6ea7633bd5
enhance: Add memory size for binlog (#33025) 
issue: #33005
1. add `MemorySize` field for insert binlog.
2. `LogSize` means the file size in the storage object.
3. `MemorySize` means the size of the data in the memory.

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2024-05-15 12:59:34 +08:00
SimFG
1d48d0aeb2
enhance: use different value to get related data size according to segment type (#33017) 
issue: #30436

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-05-14 14:59:33 +08:00
Cai Yudong
4fc7915c70
enhance: unify data generation test APIs (#32955) 
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-05-14 14:33:33 +08:00
chyezh
96489b814d
fix: remove busy log (#33042) 
issue: #32963

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-05-14 14:20:32 +08:00
chyezh
1c84a1c9b6
fix: lru related issue fixup patch (#32916) 
issue: #32206, #32801

- search failure with some assertion, segment not loaded and resource
insufficient.

- segment leak when query segments

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-05-10 19:17:30 +08:00
wei liu
25689859a1
fix: Load index metric use wrong time unit (#32935) 
issue:#32899

This PR fix the wrong metric value of load index, which introduced by
pr#32567, use wrong time unit for load index metrics

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-05-10 18:07:30 +08:00
Jiquan Long
0783582e2e
fix: temporarily disable two-phase retrieval when lru is enabled (#32927) 
issue: #31822

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-05-10 14:19:45 +08:00
Bingyi Sun
5cbf081111
fix: fix index resource estimation (#32842) 
issue: #32820

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-05-10 11:53:30 +08:00
Bingyi Sun
17a79f4ca9
enhance: The LRU cache evicts items and retries loading if the disk limit is reached. (#32819) 
Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-05-08 14:45:30 +08:00
wei liu
5038036ece
enhance: Reuse hash locations during access bloom fitler (#32642) 
issue: #32530 

when try to match segment bloom filter with pk, we can reuse the hash
locations. This PR maintain the max hash Func, and compute hash location
once for all segment, reuse hash location can speed up bf access

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-05-07 06:13:47 -07:00
congqixia
40728ce83d
enhance: Add metautil.Channel to convert string compare to int (#32749) 
See also #32748

This PR:

- Add `metautil.Channel` utiltiy which convert virtual name to physical
channel name, collectionID and shard idx
- Add channel mapper interface & implementation to convert limited
physical channel name into int index
- Apply `metautil.Channel` filter in querynode segment manager logic

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-07 19:13:35 +08:00
yihao.dai
9db3aa18bc
enhance: Remove deprecated EnableIndex (#32704) 
/kind improvement

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-05-07 17:11:30 +08:00
chyezh
641f702f64
fix: add request resource timeout for lazy load, refactor context usage in cache (#32709) 
issue: #32663

- Use new param to control request resource timeout for lazy load.

- Remove the timeout parameter of `Do`, remove `DoWait`. use `context`
to control the timeout.

- Use `VersionedNotifier` to avoid notify event lost and broadcast,
remove the redundant goroutine in cache.

related dev pr: #32684

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-05-07 16:33:30 +08:00
congqixia
efa0c73c62
fix: Unify querynode metrics cleanup in collection release (#32805) 
Related to #32803

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-07 15:41:29 +08:00
aoiasd
31dca3249e
enhance: add type info for payload writer error message and add log when querynode find new collection (#32522) 
relate: https://github.com/milvus-io/milvus/issues/32668

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-05-07 14:45:29 +08:00
Jiquan Long
1f58cda957
enhance: add more trace for search & query (#32734) 
issue: https://github.com/milvus-io/milvus/issues/32728

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-05-07 13:03:29 +08:00
congqixia
7102403a6b
fix: Add Wrapper and Keepalive for CTraceContext ids (#32746) 
See also #32742

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-07 10:05:35 +08:00
Bingyi Sun
fecd9c21ba
feat: LRU cache implementation (#32567) 
issue: https://github.com/milvus-io/milvus/issues/32783
This pr is the implementation of lru cache on branch lru-dev.

Signed-off-by: sunby <sunbingyi1992@gmail.com>
Co-authored-by: chyezh <chyezh@outlook.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
 2024-05-06 20:29:30 +08:00
Jiquan Long
c002745902
enhance: retrieve output fields after local reduce (#32346) 
issue: #31822

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-04-25 09:49:26 +08:00
yihao.dai
281a583eda
fix: Correct the negative queryable num entities metric (#32361) 
issue: https://github.com/milvus-io/milvus/issues/32281

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-04-24 15:55:24 +08:00
Cai Yudong
16b8b7b35d
enhance: Add get_vector unittest for float16 & bfloat16 (#32153) 
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-04-23 16:15:23 +08:00
chyezh
e19d17076f
fix: delete may lost when enable lru cache, some field should be reset when ReleaseData (#32012) 
issue: #30361

- Delete may be lost when segment is not data-loaded status in lru
cache. skip filtering to fix it.

- `stats_` and `variable_fields_avg_size_` should be reset when
`ReleaseData`

- Remove repeat load delta log operation in lru.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-04-16 11:17:20 +08:00
wei liu
1a98ce39f5
enhance: Remove useless logic about FromShardLeader (#32029) 
issue: #32047

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-04-10 20:11:19 +08:00
Xiaofan
dbab9c5096
fix: reduce didn't handle offset without limit and reduceStopForBest correctly (#32089) 
fix https://github.com/milvus-io/milvus/issues/32059

this pr fix two issues:

offset is not handled correctly without specify a limit
reduceStopForBest doesn't guarantee to return limit result even if there
are more result when there is small segment

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
 2024-04-10 16:01:18 +08:00
Chun Han
f3f2a5a7e9
fix: evicted segments in the serverlss mode(#31959) (#31961) 
related: #31959
1. reset segment index status after evicting to lazyload=true
2. reset num_rows to null_opt

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-04-10 15:15:19 +08:00
SimFG
90bed1caf9
enhance: add the related data size for the read apis (#31816) 
issue: #30436
origin pr: #30438
related pr: #31772

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-04-10 15:07:17 +08:00
chyezh
c9faa6d936
enhance: add more metrics for cache and search (#31777) 
issue: #30931

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-04-10 10:55:17 +08:00
zhenshan.cao
089c805e0a
enhance:Refactor hybrid search (#32020) 
issue: https://github.com/milvus-io/milvus/issues/25639
https://github.com/milvus-io/milvus/issues/31368

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-04-09 14:21:18 +08:00
chyezh
73adf2a5cc
fix: use stateful lock to avoid load and release on LocalSegment concurrently (#31606) 
issue: #31605

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-04-08 17:09:16 +08:00
chyezh
7b400252ff
fix: add configuration disk capacity config for lru and fix some bug (#31977) 
issue: #30361

- Add configurable disk capacity limit

- fix bitset reset logic

- make insert record reinsert after clear

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-04-08 15:55:16 +08:00
congqixia
0feee53631
enhance: Add back unit test for compactor and fix some TODOs (#31829) 
This PR adds back compactor "Unhandled" data type unit test and fixes
some TODOs behvaior

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-04-02 20:35:14 +08:00
jaime
bd853be8c7
enhance: Add db label for some usual metrics (#30956) 
issue: #31782

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-04-02 14:27:13 +08:00
chyezh
1ad5ccc50f
enhance: add rg and db interface for segment and db/rg metric label (#31715) 
issue: #30931

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-04-01 10:21:21 +08:00
cqy123456
976928ecd1
fix: fix fp16/bf16 some code missing and add more fp16/bf16 test (#31612) 
issue: #31534

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-03-28 14:11:10 +08:00
Bingyi Sun
3d66670619
fix: add lazy load field to mark segment load type (#31591) 
issue: https://github.com/milvus-io/milvus/issues/31673

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-03-28 11:23:10 +08:00