test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-01 11:29:48 +08:00
Code Issues Actions 11 Packages Projects Releases Wiki Activity
20,122 Commits 94 Branches 105 Tags 184 MiB
ff4bd2c9d2
Commit Graph

291 Commits

Author SHA1 Message Date
Xiaofan
66b5cbe27c
fix: CVE by upgrading some dependencies. (#34693) 
fix #34434 and #34456
upgrade otelgrpc to fix CVE

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
 2024-07-16 14:05:37 +08:00
jaime
60aab15ef4
enhance: support setting properties in create database request (#34511) 
issue: #34493
pr: #34510

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-09 18:16:18 +08:00
wei liu
d3e94f9861
enhance: Use Blocked Bloom Filter instead of basic bloom fitler impl (#34377) 
issue: #32995
pr: #33405
To speed up the construction and querying of Bloom filters, we chose a
blocked Bloom filter instead of a basic Bloom filter implementation.

WARN: This PR is compatible with old version bf impl, but if fall back
to old milvus version, it may causes bloom filter deserialize failed.

In single Bloom filter test cases with a capacity of 1,000,000 and a
false positive rate (FPR) of 0.001, the blocked Bloom filter is 5 times
faster than the basic Bloom filter in both querying and construction, at
the cost of a 30% increase in memory usage.

Block BF construct time {"time": "54.128131ms"}
Block BF size {"size": 3021578}
Block BF Test cost {"time": "55.407352ms"}
Basic BF construct time {"time": "210.262183ms"}
Basic BF size {"size": 2396308}
Basic BF Test cost {"time": "192.596229ms"}
In multi Bloom filter test cases with a capacity of 100,000, an FPR of
0.001, and 100 Bloom filters, we reuse the primary key locations for all
Bloom filters to avoid repeated hash computations. As a result, the
blocked Bloom filter is also 5 times faster than the basic Bloom filter
in querying.

Block BF TestLocation cost {"time": "529.97183ms"}
Basic BF TestLocation cost {"time": "3.197430181s"}

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-05 17:04:10 +08:00
zhenshan.cao
760b3fafd5
enhance: Refine compaction (#33982) (#34363) 
This PR cherry-picks the following commits related to data compaction:
- enhance: Refine compaction.
[#33982](https://github.com/milvus-io/milvus/pull/33982)
- fix l0 compaction may miss some sealed segments.
[#33838](https://github.com/milvus-io/milvus/pull/33980)

issue : https://github.com/milvus-io/milvus/issues/32939
https://github.com/milvus-io/milvus/issues/33955

pr : https://github.com/milvus-io/milvus/pull/33982

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-07-03 09:52:49 +08:00
congqixia
6b348e4e91
enhance: [2.4] Add go-deadlock as unittest only dependency (#33063) (#34322) 
Cherry-pick from master
pr: #33063
See also #33062

This PR:

- Add lock.RWMutex & lock.Mutex alias to switch implementation based on
build flags
- When build flags has test in it, use go-deadlock to detect possible
deadlocks
- Replace all sync.RWMutex & sync.Mutex in datacoord pkg

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-02 18:48:10 +08:00
zhenshan.cao
14a11e379c
enhance: Refactor Compaction to enable persistence(#33265) (#34268) 
pr : #33265 

issue #33586

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-07-01 19:32:07 +08:00
jaime
6423b6c718
enhance: move rocksmq from internal to pkg (#34165) 
pr:  https://github.com/milvus-io/milvus/pull/33881
issue:  https://github.com/milvus-io/milvus/issues/33956

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-26 13:36:05 +08:00
shaoting-huang
b3d425f50a
enhance: Upgrade go version from 1.20 to 1.21 (#33940) 
issue #32982

related pr in master:
pr: #33047 #33150 #33176 #33351 #33202 #33192

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
 2024-06-20 14:52:00 +08:00
congqixia
59144d8246
enhance: Bump milvus & proto version to v2.4.5 (#33852) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-14 22:27:59 +08:00
congqixia
86f3433053
enhance: [2.4]Use fastjson lib for unmarshal delete log (#33787) (#33802) 
Cherry-pick from master
pr: #33878
```
goos: linux
goarch: amd64
GOMAXPROC=1
cpu: Intel(R) Core(TM) i7-8700 CPU @ 3.20GHz
BenchmarkJsonSerdeStd             343872              3568 ns/op            1335 B/op         25 allocs/op
BenchmarkJsonSerdeFastjson       5124177               234.9 ns/op            16 B/op          1 allocs/op
```

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-13 10:27:57 +08:00
congqixia
b2d83d3354
enhance: Bump milvus version to v2.4.2 (#33048) 
Bumping version to v2.4.2. Also bump milvus-proto version to v2.4.3.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-15 13:57:35 +08:00
Xiaofan
36f1ea93a5
enhance: optimize plan parser pool to avoid unnessary recycle (#32869) 
fix #32868
plan parser takes too much cpu on high qps,this pr try to avoid create
lexer and parser too freequent

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
 2024-05-11 10:51:31 +08:00
wei liu
c35797c399
enhance: expose DescribeDatabase api in proxy (#32732) 
issue: #32707

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-05-09 22:51:30 +08:00
wayblink
42d0412e93
enhance: Add channelCPs in FlushResponse (#32044) 
#32609

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-04-30 09:45:27 +08:00
congqixia
f07e78ec91
enhance: Bump Milvus & proto version to v2.4.1 (#32693) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-04-29 14:37:26 +08:00
wei liu
07720f1a95
enhance: expose alter database api in proxy (#32639) 
issue: #30040

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-04-28 18:15:31 +08:00
SimFG
bed6363feb
enhance: update the go-api version for the list api (#32605) 
issue: https://github.com/milvus-io/milvus/issues/32550
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-04-26 17:49:33 +08:00
jaime
2c63f848bf
fix: upgrade nats server to fix security vulnerabilities (#32021) 
issue: #32022

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-04-11 16:07:18 +08:00
SimFG
789e014c74
enhance: add the db id for the describe collection response (#32114) 
/kind improvement
issue: #32110

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-04-10 21:23:20 +08:00
congqixia
9d16aa0bd3
enhance: Bump jose2go for security alerts (#32040) 
See also #31986

- jose2go vulnerable to denial of service via large p2c value

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-04-10 10:05:19 +08:00
congqixia
a860bf66f9
enhance: Bump google.golang.org/grpc to 1.57.1 (#31985) 
See also #31986
See dependency alert for "gRPC-Go HTTP/2 Rapid Reset vulnerability"

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-04-09 10:53:17 +08:00
cqy123456
976928ecd1
fix: fix fp16/bf16 some code missing and add more fp16/bf16 test (#31612) 
issue: #31534

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-03-28 14:11:10 +08:00
SimFG
b1a1cca10b
feat: add more operation detail info for better allocation (#30438) 
issue: #30436

---------

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-03-28 06:33:11 +08:00
yihao.dai
2b7df47aa9
feat: Add RBAC functionality to ImportV2 (#31298) 
issue: https://github.com/milvus-io/milvus/issues/31282,
https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-03-19 20:01:07 +08:00
cai.zhang
72ac2b1e93
fix:Update tencent cloud version (#30847) 
Update tencent cloud version for updating token when expired.
issue: #30846

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-03-01 11:09:00 +08:00
chyezh
e59bc1e526
enhance: use latest milvus proto, add unimplemented server (#30889) 
issue: #30647

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-02-29 10:13:00 +08:00
Buqian Zheng
1632668b64
enhance: update milvus-proto dep to add sparse vector related protos (#30400) 
issue: https://github.com/milvus-io/milvus/issues/29419

this PR solely adds proto definition. sparse float vector support will
be in subsequent PRs.

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-02-02 12:41:04 +08:00
Bingyi Sun
adaf270697
enhance: roll back apimachinery to v0.28.6 (#30398) 
apimachinery use go 1.21 in v0.29.0
reference issue: golang/go#62409

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-02-01 20:31:03 +08:00
cai.zhang
6cf2f09b60
feat: Support tencent cloud object storage for milvus (#30163) 
issue: #30162

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-01-23 11:28:56 +08:00
jaime
ee26561b5d
fix: remove checking if running inside container (#29940) 
issue: https://github.com/milvus-io/milvus/issues/29846

Since checking if Milvus running inside the container is difficult,
which is decided by the system version and cgroup version, we try to get
the memory limit of the container each time instead of checking whether
inside the container first.

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-01-14 10:24:52 +08:00
Bingyi Sun
e1258b8cad
feat: integrate storagev2 into loading segment (#29336) 
issue: #29335

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-01-12 18:10:51 +08:00
jaime
90984a2cab
enhance: Support read hardware metrics for cgroupv2 (#29850) 
issue: https://github.com/milvus-io/milvus/issues/29846

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-01-12 10:30:51 +08:00
zhenshan.cao
2f702ad316
enhance: update milvus-proto to support rbac for alias (#29801) 
issue: https://github.com/milvus-io/milvus/issues/29781
issue: https://github.com/milvus-io/milvus-proto/issues/237
Update the protocol to support the RBAC functionality for aliases

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-01-09 22:02:49 +08:00
wayblink
635a7f777c
feat: add clustering key in create/describe collection (#29506) 
#28410
/kind feature

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-01-07 19:56:48 +08:00
wayblink
05d735c322
enhance: Rename SearchV2 to HybridSearch (#29592) 
related: https://github.com/milvus-io/milvus-proto/pull/233
issue: #29593
/kind enhancement

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-01-04 19:22:46 +08:00
aoiasd
a76e3b2813
Refine delete by expression for forbid proxy dml task scheduler hang (#29340) 
relate: https://github.com/milvus-io/milvus/issues/29146

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-12-26 19:52:48 +08:00
SimFG
dd9c61831d
enhance: Support to get the param value in the runtime (#29297) 
/kind improvement
issue: #29299

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-12-22 18:36:44 +08:00
yah01
a0e1a1eb31
feat: support enable/disable mmap for index (#29005) 
support enable/disable mmap for index, the user could alter the index's
mode by `AlterIndex` method
related: https://github.com/milvus-io/milvus/issues/21866

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-12-21 18:07:24 +08:00
PowderLi
bcd6865b29
enhance: add 3 builtin roles (#28961) 
issue: #28960 [milvus-proto
#212](https://github.com/milvus-io/milvus-proto/issues/212)

add new configuration: builtinRoles
user can define roles in config file: `milvus.yaml`

there is an example:
1. db_ro, only have read privileges, include load
2. db_rw, read and write privileges, include create/drop/rename
collection
3. db_admin, not only read and write privileges, but also user
administration

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-12-18 14:28:41 +08:00
Bingyi Sun
ad866d2889
feat: integrate storagev2 into index build process (#28995) 
issue: https://github.com/milvus-io/milvus/issues/28994

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-12-13 17:24:38 +08:00
Bingyi Sun
4fedff6d47
feat: integrate storage v2 into the write path (#28440) 
#28378

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-11-23 17:26:24 +08:00
yah01
bfccfcd0ca
enhance: refine error messages (#28424) 
- Split the simple reason and full detail
- Refine existing error messages
related: #28422

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-21 17:02:24 +08:00
Bingyi Sun
59355cb3dc
Update arrow version to v12 (#28425) 
issue: https://github.com/milvus-io/milvus/issues/28423

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-11-15 10:36:19 +08:00
yah01
6f59510723
Upgrade go to 1.20 (#24390) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-11-10 07:46:18 +08:00
Jovan
bc85422a04
fix: bitSet panic: runtime error: makeslice: len out of range (#27984) 
Signed-off-by: Jovan <ulovecode@gmail.com>
 2023-10-27 18:02:12 +08:00
congqixia
ee184a709e
Bump Milvus Version & milvus-proto/go-api to v2.3.2 (#27893) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-24 21:02:11 +08:00
SimFG
9b0ecbdca7
Support to replicate the mq message (#27240) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-10-20 14:26:09 +08:00
SimFG
84d05b939b
Support the apikey authentication (#27723) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-10-17 21:00:11 +08:00
Xiaofan
333a0c8a53
Fix RocksMQ seek speed (#27646) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-10-16 10:18:08 +08:00
dependabot[bot]
f0e358fc56
Bump golang.org/x/net from 0.10.0 to 0.17.0 (#27656) 
Bumps [golang.org/x/net](https://github.com/golang/net) from 0.10.0 to 0.17.0.
- [Commits](https://github.com/golang/net/compare/v0.10.0...v0.17.0)

---
updated-dependencies:
- dependency-name: golang.org/x/net
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2023-10-13 17:19:34 +08:00