test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-11-30 02:48:45 +08:00
Code Issues Actions 11 Packages Projects Releases Wiki Activity
20,631 Commits 94 Branches 105 Tags 184 MiB
9d80137698
Commit Graph

866 Commits

Author SHA1 Message Date
congqixia
9d80137698
fix: Check clustering key skip load behavior (#35865) 
feature issue: #35415
See also #35861

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-09-02 11:17:02 +08:00
SimFG
311f860676
enhance: support to drop the role which is related the privilege list (#35727) 
- issue: #35545

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-08-30 15:17:00 +08:00
SimFG
07bb071aaf
enhance: add the unmarshal method for the rbac msg (#35692) 
/kind improvement
- issue: #35391

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-08-30 07:23:00 +08:00
jaime
b0ac04d104
enhance: set database properties to restrict read access (#35745) 
issue: #35744

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-08-29 13:17:01 +08:00
Zhen Ye
99dff06391
enhance: using streaming service in insert/upsert/flush/delete/querynode (#35406) 
issue: #33285

- using streaming service in insert/upsert/flush/delete/querynode
- fixup flusher bugs and refactor the flush operation
- enable streaming service for dml and ddl
- pass the e2e when enabling streaming service
- pass the integration tst when enabling streaming service

---------

Signed-off-by: chyezh <chyezh@outlook.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-29 10:03:08 +08:00
zhagnlu
4d2f96c760
enhance: support bitmap mmap (#35399) 
#32900

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-08-27 16:34:59 +08:00
smellthemoon
098c10922b
enhance: add some check when create collection (#35629) 
not support nullable==true in vector type. check it when create
collection.

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-08-25 16:23:05 +08:00
Xiaofan
50fcfe8ef1
enhance: add nan and inf check (#35683) 
fix #35594
add float check on files

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
 2024-08-25 15:22:57 +08:00
zhagnlu
42f7800b5b
enhance: add bitmap offset cache to speed up retrieve raw data (#35498) 
#35458

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-08-24 01:40:58 +08:00
SimFG
9dc1311f38
enhance: add the param to control whether to include the current msg (#35656) 
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-08-23 15:56:57 +08:00
Gao
e8e3544a11
enhance: add hit segment num metrics for queryHook (#35577) 
issue: #35576

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-08-23 12:49:02 +08:00
smellthemoon
ba6db117e3
enhance: add some integration test in null (#35599) 
#31728

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-08-21 17:44:56 +08:00
wei liu
cf242f9e09
fix: fix dynamic update config doesn't works for some param (#35572) 
issue: #35570
milvus support config cache to spped up config access, but only evict
param's cache when param has been updated. but milvus's param may rely
on other param's value, let's say ParamsA relys on paramsB, when paramsB
updated, it will evict paramB's cache, but the paramA's cache still keep
the old value.

This PR evict all config cache to solve the above issue, cause dynamic
update config won't be much frequetly.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-21 11:02:56 +08:00
SimFG
731d45abbe
enhance: provide more general configuration to control mmap behavior (#35359) 
- issue: #35273

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-08-21 00:22:54 +08:00
Ted Xu
41646c8439
feat: integrate new deltalog format (#35522) 
See #34123

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-08-20 19:06:56 +08:00
congqixia
2fbc628994
feat: Support field partial load collection (#35416) 
Related to #35415

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-20 16:49:02 +08:00
Chun Han
031ee6f155
enhance: support httpv1/v2 throttle and add it for httpV2(#35350) (#35470) 
related: #35350

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-08-20 16:16:55 +08:00
Zhen Ye
4d69898cb2
enhance: support single pchannel level transaction (#35289) 
issue: #33285

- support transaction on single wal.
- last confirmed message id can still be used when enable transaction.
- add fence operation for segment allocation interceptor.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-08-19 21:22:56 +08:00
wei liu
853d2f33e0
enhance: Trim space during parsing string list from string (#35546) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-19 20:48:55 +08:00
congqixia
2736a8b88c
enhance: Update Makefile to generate mockery (#35517) 
Some mockery cmd is out-of-date and fail to work. This PR update these
commands to match current pkg.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-16 18:44:53 +08:00
wei liu
ead37326f0
enhance: Add backup/restore privilege to admin group (#35515) 
issue: #35443

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-16 18:42:54 +08:00
wei liu
a570567644
enhance: Enable ReadOnly/ReadWrite/Admin Privilege Group to simplify RBAC grant progress (#35472) 
issue: #35471

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-16 14:18:54 +08:00
Buqian Zheng
f4a91e135b
enhance: Allow empty sparse row (#34700) 
issue: #29419

* If a sparse vector with 0 non-zero value is inserted, no ANN search on
this sparse vector field will return it as a result. User may retrieve
this row via scalar query or ANN search on another vector field though.
* If the user uses an empty sparse vector as the query vector for a ANN
search, no neighbor will be returned.

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-08-16 14:14:54 +08:00
wei liu
1d49358f82
enhance: Add BackupRBAC/RestoreRBAC API to enable rbac backup (#35444) 
issue: #35443

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-16 10:10:53 +08:00
SimFG
b2cc4b0776
feat: add the rbac msg and send them to the replicate channel (#35392) 
- issue: #35391

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-08-15 12:06:52 +08:00
Ted Xu
c6ae7d4d56
enhance: adding the msgchannel section in generated yaml (#35466) 
See #32168

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-08-14 18:32:53 +08:00
congqixia
f99a761d87
enhance: Support dynamic field in SchemaHelper (#35461) 
Related to #35415

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-14 18:04:59 +08:00
Reid Chan
6c135e5ea9
fix: key of DataCoordTicklePath is wrong (issue: #35259) (#35263) 
issue: [35259](https://github.com/milvus-io/milvus/issues/35259)

Signed-off-by: Reid Chan <reidchan@apache.org>
 2024-08-13 17:40:20 +08:00
chyezh
16b0aee97f
enhance: timetick interceptor optimization (#35287) 
issue: #33285

- remove redundant goroutine by using insepctor.
- remove the coutinous non-message timetick persistence
- periodically push the time tick forward without persistent timetick
message.
- add 'message type filter' deliver filter.

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-08-12 18:58:25 +08:00
smellthemoon
c4a240acb1
enhance: memory value are displayed in MB (#35292) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-08-09 14:14:25 +08:00
wei liu
344dc6a9f8
enhance: enable to set load config in cluster level (#35169) 
issue: #35170
This PR enable to set load configs in cluster level, such as replicas
and resource groups. then when load collections will use the load
config.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-07 12:38:21 +08:00
chyezh
c725416288
enhance: move streaming proto into pkg (#35284) 
issue: #33285

- move streaming related proto into pkg.
- add v2 message type and change flush message into v2 message.

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-08-07 10:34:16 +08:00
Jiquan Long
b5022b0647
enhance: add metric to record disk quota (#35305) 
https://github.com/milvus-io/milvus/issues/35177

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-08-07 10:26:16 +08:00
Xiaofan
885d891106
Change minio use virtual config to not panic (#35327) 
minio use virtual should not panic if empty. User may use other object
storage instead

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
 2024-08-07 09:58:15 +08:00
presburger
a9352a073b
enhance: change gpu default mem pool size (#34159) 
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
 2024-08-06 10:24:15 +08:00
chyezh
14051fed7d
enhance: streaming service client (#34656) 
issue: #33285

- implement streaming service client.
- implement producing and consuming service client by streaming coord
client and streaming node client.

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-08-05 21:38:15 +08:00
cai.zhang
6542c1ab0e
enhance: Add monitoring metrics for task execution time in datacoord (#35139) 
issue: #35138

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-08-05 16:26:17 +08:00
congqixia
a6903701a4
enhance: Remove stackerror imported unexpected (#35256) 
`stackerror` pkg is imported by accident and could be replaced by
cockroachdb errors lib. This PR removes import line and fix this
problem.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-05 14:48:16 +08:00
chyezh
9871966415
enhance: segment alloc interceptor (#34996) 
#33285

- add segment alloc interceptor for streamingnode.
- add add manual alloc segment rpc for datacoord.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-08-04 07:40:15 +08:00
yihao.dai
a4439cc911
enhance: Implement flusher in streamingNode (#34942) 
- Implement flusher to:
  - Manage the pipelines (creation, deletion, etc.)
  - Manage the segment write buffer
  - Manage sync operation (including receive flushMsg and execute flush)
- Add a new `GetChannelRecoveryInfo` RPC in DataCoord.
- Reorganize packages: `flushcommon` and `datanode`.

issue: https://github.com/milvus-io/milvus/issues/33285

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-02 18:30:23 +08:00
jaime
fcec4c21b9
fix: check collection health(queryable) fail for releasing collection (#34947) 
issue: #34946

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-08-02 17:20:15 +08:00
congqixia
db06b86594
enhance: Sync otlp dependency version to fix security issue (#35192) 
Related to #34434

otelgrpc DoS vulnerability due to unbound cardinality metrics

https://github.com/milvus-io/milvus/security/dependabot/91

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-02 16:22:20 +08:00
wei liu
f466129924
enhance: add segment state on data coord bin log size metrics (#35153) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-02 14:48:14 +08:00
aoiasd
3655ab10b2
fix: evict paramtable cache miss (#34771) 
relate: https://github.com/milvus-io/milvus/issues/33461

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-08-02 11:52:14 +08:00
congqixia
c64a078458
enhance: Support proxy/delegator qn client pooling (#35194) 
See also #35196

Add param item for proxy/delegator query node client pooling and
implement pooling logic

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-02 11:24:19 +08:00
wei liu
3b735b4b02
enhance: Refine param init for MmapDirPath (#35181) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-02 11:12:14 +08:00
Jiquan Long
4dbd023402
enhance: add metric to record maxInsertRate and queryNodeMemoryHighWaterLevel (#35185) 
fix: https://github.com/milvus-io/milvus/issues/35177

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-08-02 10:14:13 +08:00
cai.zhang
196a7986b3
enhance: Change the fixed value to a ratio for clustering segment size (#35076) 
issue: #34495

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-08-01 22:04:14 +08:00
yihao.dai
7721a28584
fix: Fix unstable TestDispatchToVchannels ut (#35163) 
issue: https://github.com/milvus-io/milvus/issues/35150

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-01 14:56:13 +08:00
Ted Xu
b4d0f4df0a
enhance: update param documents (#35152) 
See: #35161

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-08-01 12:38:13 +08:00