test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-02 20:09:57 +08:00
Code Issues Actions 12 Packages Projects Releases Wiki Activity
19,920 Commits 94 Branches 105 Tags 184 MiB
25d8b74f71
Commit Graph

688 Commits

Author SHA1 Message Date
wei liu
25d8b74f71
enhance: Execute bloom filter apply in parallel to speed up segment predict (#33793) 
issue: #33610
pr: #33792

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-13 14:14:04 +08:00
wei liu
54feef30e7
enhance: Use BatchPkExist to reduce bloom filter func call cost (#33752) 
issue: #33610
pr: #33611

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-12 17:45:58 +08:00
Aldrin
ff9bb17cc5
fix: Masked kafka credentials getting printed in logs (#33747) 
Cherry-pick from master
pr: #33730 
issue: https://github.com/milvus-io/milvus/issues/33727

Signed-off-by: Aldrin <imagesai32@gmail.com>
 2024-06-11 15:59:54 +08:00
SimFG
f664b51ebe
enhance: [2.4] try to speed up the loading of small collections (#33746) 
- issue: #33569
- pr: #33570

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-06-11 15:07:55 +08:00
SimFG
c331aa4ad3
enhance: [2.4] add the includeCurrentMsg param for the Seek method (#33743) 
/kind improvement

- issue: #33325
- pr: #33326

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-06-11 15:01:55 +08:00
Buqian Zheng
39e341e83a
fix: [2.4] update check for sparse hnsw index (#33714) 
issue: #29419
pr: #33713

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-06-07 18:36:07 +08:00
XuanYang-cn
26db471968
enhance: [2.4]Add consts of MsgDispatcher to configs (#33680) 
See also: #33676
pr: #33679

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-07 14:21:58 +08:00
cqy123456
3562ef83b2
enhance:[cherry-pick]Allow hnsw to build sparse (#33653) 
issue:https://github.com/milvus-io/milvus/issues/22837
related:https://github.com/milvus-io/milvus/pull/33675

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-06 19:15:51 +08:00
foxspy
58a7111599
enhance: [cherry-pick] add autoindex mapping for binary/sparse datatype (#33625) 
issue: #22837 
pr: #33624

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
 2024-06-06 10:33:52 +08:00
Chun Han
627b787aed
fix: query iterator lack results(#33137) (#33422) (#33506) 
related: #33137 
pr: https://github.com/milvus-io/milvus/pull/33422

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-06-05 18:51:52 +08:00
aoiasd
aaf6c85095
fix: paramtable cache cause dynamic config non-dynamic (#33473) (#33590) 
relate: https://github.com/milvus-io/milvus/issues/33461
pr: https://github.com/milvus-io/milvus/pull/33473

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-06-05 10:15:51 +08:00
jaime
0ad55c6c44
fix: fix loaded entity num is inaccurate (#33522) 
issue: #33520

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-04 20:09:54 +08:00
Buqian Zheng
d6a994038c
enhance: [2.4] add sparse float vector support to restful v2 (#33555) 
issue: https://github.com/milvus-io/milvus/issues/29419
pr: #33231

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-06-03 19:31:46 +08:00
congqixia
b9b76ee9a4
fix: [2.4]Use localStorage path to check disk cap for indexnode (#33450) (#33505) 
Cherry-pick from master
pr: #33450
See also #30943 #30944

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-03 10:21:47 +08:00
congqixia
8e7f36d928
enhance: Bump milvus version v2.4.4 (#33504) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-31 14:55:44 +08:00
Cai Yudong
68e2d532d8
enhance: Cherry-pick following SparseFloatVector bulk insert PRs to Milvus2.4 (#33391) 
Cherry pick from master
pr: #33064 #33101 #33187 #33259 #33224
#33064 Support readable JSON file import for
Float16/BFloat16/SparseFloat
  #33101 Store SparseFloatVector into parquet as JSON string
  #33187 Fix SparseFloatVector data parse error for parquet
  #33259 Fix SparseFloatVector data parse error for json
  #33224 Optimize bulk insert unittest

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-05-30 10:31:45 +08:00
yihao.dai
7384bfe3f8
fix: use seperate warmup pool and disable warmup by default (#33348) (#33349) 
1. use a small warmup pool to reduce the impact of warmup
2. change the warmup pool to nonblocking mode
3. disable warmup by default
4. remove the maximum size limit of 16 for the load pool

issue: https://github.com/milvus-io/milvus/issues/32772

pr: https://github.com/milvus-io/milvus/pull/33348

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
Co-authored-by: xiaofanluan <xiaofan.luan@zilliz.com>
 2024-05-28 19:27:43 +08:00
congqixia
ef9c1191ae
enhance: Bump milvus version to v2.4.3 (#33408) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-28 10:11:43 +08:00
wei liu
36172492f7
enhance: Decrease bloom filter fp rate to reduce delete impact (#33301) (#33329) 
pr: #33301
when milvus process delete record, it need to find record's corresponded
segment by bloom filter, and higher bloom filter fp rate will cause
delete record forwards to wrong segments.

This PR Decrease bloom filter's default fp to 0.001.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-05-24 10:27:41 +08:00
jaime
8990b8b051
fix: correct error of metrics stats (#33305) 
issue: #32980
cherry pick from master
pr:  #33075 #33255

---------

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-05-24 09:15:41 +08:00
SimFG
37b2f90c90
enhance: [2.4] the panic when db isn't existed in the rate limit interceptor (#33308) 
issue: #33243
pr: #33244

1. fix: the panic when db isn't existed in the rate limit interceptor
#33244
2. enhance: check the auth in some rest v2 api #33256

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-05-23 17:25:41 +08:00
congqixia
d6bc95de55
enhance: [2.4] Add param item to ignore bad message id in checkpoint (#33123) (#33249) 
Cherry-pick from master
pr: #33123 #33158
See also  #33122

This pr add param item `mq.ignoreBadPosition` to control behavior when
mq failed to parse message id from checkpoint

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-23 15:27:41 +08:00
aoiasd
1f23c39700
fix:[Cherry-Pick] can't generate traceID when use noop exporter (#33191) (#33208) 
relate: https://github.com/milvus-io/milvus/issues/33190
pr: https://github.com/milvus-io/milvus/pull/33191

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-05-22 11:13:39 +08:00
SimFG
2a38f5b598
enhance: [2.4] add config to control whether to init public role permissions (#33174) 
issue: #33164
pr: #33165

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-05-21 22:39:46 +08:00
congqixia
b2d83d3354
enhance: Bump milvus version to v2.4.2 (#33048) 
Bumping version to v2.4.2. Also bump milvus-proto version to v2.4.3.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-15 13:57:35 +08:00
jaime
f48a7ff8ff
enhance: use Delete instead of DeletePartialMatch to remove metrics (#33029) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-05-14 18:49:33 +08:00
Cai Yudong
4fc7915c70
enhance: unify data generation test APIs (#32955) 
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2024-05-14 14:33:33 +08:00
jaime
ba625835bc
enhance: Add metrics for segment index files size (#32979) 
issue:#32980

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-05-13 17:59:32 +08:00
foxspy
f6777267e3
enhance: add score compute consistency config for knowhere (#32997) 
issue: https://github.com/milvus-io/milvus/issues/32583
related: #32584

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
 2024-05-13 14:21:31 +08:00
SimFG
4031abd2fa
enhance: change default partition num to 16 when using partition key (#32950) 
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-05-13 14:19:31 +08:00
Bingyi Sun
4724779b3b
enhance: remove fallback keys for config generator (#32946) 
Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-05-13 13:33:31 +08:00
chyezh
1c84a1c9b6
fix: lru related issue fixup patch (#32916) 
issue: #32206, #32801

- search failure with some assertion, segment not loaded and resource
insufficient.

- segment leak when query segments

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-05-10 19:17:30 +08:00
wei liu
e2332bdc17
enhance: Enable channel exclusive balance policy (#32911) 
issue: #32910  
* split replica's node list to channels when create replicas
 * balance nodes among channels when node change happens
 * implement channel level balance, let balance happens in channel level

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-05-10 17:27:31 +08:00
Bingyi Sun
1e1fba0588
enhance: add lazy load retry configurations (#32848) 
Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-05-10 10:55:30 +08:00
wei liu
c35797c399
enhance: expose DescribeDatabase api in proxy (#32732) 
issue: #32707

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-05-09 22:51:30 +08:00
aoiasd
54a51b1236
enhance: Support dynamic config for opentelemetry trace (#32169) 
relate: https://github.com/milvus-io/milvus/issues/31940

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-05-09 17:43:30 +08:00
Bingyi Sun
17a79f4ca9
enhance: The LRU cache evicts items and retries loading if the disk limit is reached. (#32819) 
Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-05-08 14:45:30 +08:00
congqixia
40728ce83d
enhance: Add metautil.Channel to convert string compare to int (#32749) 
See also #32748

This PR:

- Add `metautil.Channel` utiltiy which convert virtual name to physical
channel name, collectionID and shard idx
- Add channel mapper interface & implementation to convert limited
physical channel name into int index
- Apply `metautil.Channel` filter in querynode segment manager logic

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-07 19:13:35 +08:00
chyezh
641f702f64
fix: add request resource timeout for lazy load, refactor context usage in cache (#32709) 
issue: #32663

- Use new param to control request resource timeout for lazy load.

- Remove the timeout parameter of `Do`, remove `DoWait`. use `context`
to control the timeout.

- Use `VersionedNotifier` to avoid notify event lost and broadcast,
remove the redundant goroutine in cache.

related dev pr: #32684

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-05-07 16:33:30 +08:00
yiwangdr
b1eacb2ae8
feat: datacoord/node watch based on rpc (#32036) 
issue: https://github.com/milvus-io/milvus/issues/25309

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
 2024-05-07 15:49:30 +08:00
congqixia
efa0c73c62
fix: Unify querynode metrics cleanup in collection release (#32805) 
Related to #32803

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-07 15:41:29 +08:00
SimFG
0ea08b008a
enhance: add the config to control the way when fail to init plugin (#32680) 
issue: #32679

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-05-07 11:01:31 +08:00
cai.zhang
a5d1135512
fix: Fix scalar auto index config incorrect version (#32795) 
issue: #29309

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-05-07 10:13:00 +08:00
congqixia
02bf47f295
fix: Validate num of rows for insert field data with schema (#32770) 
See also #32769

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-07 10:07:29 +08:00
Bingyi Sun
fecd9c21ba
feat: LRU cache implementation (#32567) 
issue: https://github.com/milvus-io/milvus/issues/32783
This pr is the implementation of lru cache on branch lru-dev.

Signed-off-by: sunby <sunbingyi1992@gmail.com>
Co-authored-by: chyezh <chyezh@outlook.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
 2024-05-06 20:29:30 +08:00
wayblink
42d0412e93
enhance: Add channelCPs in FlushResponse (#32044) 
#32609

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-04-30 09:45:27 +08:00
congqixia
f07e78ec91
enhance: Bump Milvus & proto version to v2.4.1 (#32693) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-04-29 14:37:26 +08:00
SimFG
09cd56d44f
enhance: add the skip auto id and partition key check config (#32592) 
/kind improvement
issue: #32591

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-04-29 10:29:26 +08:00
wei liu
07720f1a95
enhance: expose alter database api in proxy (#32639) 
issue: #30040

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-04-28 18:15:31 +08:00
congqixia
967cfb3133
enhance: Add WithBlock option for etcd client creation (#32641) 
Related to #32598

Use `WithBlock` may fail fast when create etcd client to some invalid
etcd endpoints and make it easier to check problem.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-04-28 14:45:25 +08:00