test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-02 03:48:37 +08:00
Code Issues Actions 12 Packages Projects Releases Wiki Activity
20,447 Commits 94 Branches 105 Tags 184 MiB
b71e058bc5
Commit Graph

830 Commits

Author SHA1 Message Date
cai.zhang
6542c1ab0e
enhance: Add monitoring metrics for task execution time in datacoord (#35139) 
issue: #35138

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-08-05 16:26:17 +08:00
congqixia
a6903701a4
enhance: Remove stackerror imported unexpected (#35256) 
`stackerror` pkg is imported by accident and could be replaced by
cockroachdb errors lib. This PR removes import line and fix this
problem.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-05 14:48:16 +08:00
chyezh
9871966415
enhance: segment alloc interceptor (#34996) 
#33285

- add segment alloc interceptor for streamingnode.
- add add manual alloc segment rpc for datacoord.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-08-04 07:40:15 +08:00
yihao.dai
a4439cc911
enhance: Implement flusher in streamingNode (#34942) 
- Implement flusher to:
  - Manage the pipelines (creation, deletion, etc.)
  - Manage the segment write buffer
  - Manage sync operation (including receive flushMsg and execute flush)
- Add a new `GetChannelRecoveryInfo` RPC in DataCoord.
- Reorganize packages: `flushcommon` and `datanode`.

issue: https://github.com/milvus-io/milvus/issues/33285

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-02 18:30:23 +08:00
jaime
fcec4c21b9
fix: check collection health(queryable) fail for releasing collection (#34947) 
issue: #34946

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-08-02 17:20:15 +08:00
congqixia
db06b86594
enhance: Sync otlp dependency version to fix security issue (#35192) 
Related to #34434

otelgrpc DoS vulnerability due to unbound cardinality metrics

https://github.com/milvus-io/milvus/security/dependabot/91

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-02 16:22:20 +08:00
wei liu
f466129924
enhance: add segment state on data coord bin log size metrics (#35153) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-02 14:48:14 +08:00
aoiasd
3655ab10b2
fix: evict paramtable cache miss (#34771) 
relate: https://github.com/milvus-io/milvus/issues/33461

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-08-02 11:52:14 +08:00
congqixia
c64a078458
enhance: Support proxy/delegator qn client pooling (#35194) 
See also #35196

Add param item for proxy/delegator query node client pooling and
implement pooling logic

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-02 11:24:19 +08:00
wei liu
3b735b4b02
enhance: Refine param init for MmapDirPath (#35181) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-02 11:12:14 +08:00
Jiquan Long
4dbd023402
enhance: add metric to record maxInsertRate and queryNodeMemoryHighWaterLevel (#35185) 
fix: https://github.com/milvus-io/milvus/issues/35177

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-08-02 10:14:13 +08:00
cai.zhang
196a7986b3
enhance: Change the fixed value to a ratio for clustering segment size (#35076) 
issue: #34495

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-08-01 22:04:14 +08:00
yihao.dai
7721a28584
fix: Fix unstable TestDispatchToVchannels ut (#35163) 
issue: https://github.com/milvus-io/milvus/issues/35150

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-01 14:56:13 +08:00
Ted Xu
b4d0f4df0a
enhance: update param documents (#35152) 
See: #35161

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-08-01 12:38:13 +08:00
wei liu
e9d61daa3f
enhance: Reduce delegator memory overloaded factor to 0.1 (#35092) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-01 10:21:50 +08:00
congqixia
dfda9f0478
enhance: Add depguard rules to ban deprecated proto lib (#35140) 
See also #34394 #34252

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-01 10:01:49 +08:00
Gao
6695c6d0a3
enhance: add channel num for queryHook optimization (#35104) 
At most cases, data in each channel is almost evenly distributed, we
could utilize the channel num info to optimize searh param in queryHook

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-07-31 18:59:49 +08:00
wayblink
ce3f836876
fix: compaction task not be cleaned correctly (#34765) 
1.fix compaction task not be cleaned correctly
2.add a new parameter to control compaction gc loop interval
3.remove some useless configs of clustering compaction

bug: #34764

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-30 20:21:56 +08:00
yihao.dai
39d25938c5
enhance: Avoid panic due to nil schema (#35063) 
/kind improvement

issue: https://github.com/milvus-io/milvus/discussions/25620

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-30 20:18:32 +08:00
congqixia
de8a266d8a
enhance: Enable linux code checker (#35084) 
See also #34483

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-30 15:53:51 +08:00
congqixia
972752258a
enhance: Support otlp http exporter (#35053) 
See also #35052

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-29 17:43:49 +08:00
SimFG
0e41f104c5
enhance: use the collection id to group msg pack in the msg dispatcher (#34871) 
- issue: #34870

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-07-29 11:45:49 +08:00
wei liu
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 (#34394) 
issue: #34252

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-29 11:31:51 +08:00
yihao.dai
4a3d98d88c
enhance: Fast close msgdispatcher target (#34803) 
/kind improvement

issue: https://github.com/milvus-io/milvus/issues/34075

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-29 00:25:49 +08:00
PowderLi
5584ae2e14
enhance: add metrics for restful apis (#34969) 
issue: #34968

Signed-off-by: PowderLi <min.li@zilliz.com>
 2024-07-29 00:13:47 +08:00
cai.zhang
2372452fac
enhance: Optimized the GC logic to ensure that memory is released in time (#34949) 
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-28 23:53:47 +08:00
zhagnlu
804ec24c02
fix:fix retrieve raw data from bitmap array index (#34848) 
#34795

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-07-27 01:53:47 +08:00
congqixia
4ee6c69217
enhance: Add Segment Level in milvus segment info APIs (#34763) 
See also #34746

This PR add segment level field in response of
`GetPersistentSegmentInfo` and `GetQuerySegmentInfo`

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-26 10:01:46 +08:00
Ted Xu
63f0154dfb
fix: enable milvus.yaml check (#34567) 
See #32168

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-07-25 18:05:46 +08:00
chyezh
bef06e5acf
enhance: add streaming coord and node grpc service register (#34655) 
issue: #33285

- register streaming coord service into datacoord.
- add new streaming node role.
- add global static switch to enable streaming service or not.

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-07-25 12:19:44 +08:00
chyezh
4f6cbfd520
enhance: specialized immutable and mutable message (#34951) 
issue: #33285

- add specialized mutable and immutable message, make type safe.
- add version based constructor and type.

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-07-25 11:57:45 +08:00
Jiquan Long
b843c91bad
feat: record the number of times milvus turns into force-deny-writing states (#34960) 
/kind improvement
fix: #34961

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-07-25 11:39:46 +08:00
wei liu
166fc902b0
enhance: Limit collection's normal balance speed (#34810) 
issue: #34798

after we remove the task priority on query coord, to avoid load/release
segment blocked by too much balance task, we limit the balance task size
in each round. at same time, we reduce the balance interval to trigger
balance more frequently.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-24 19:11:44 +08:00
David Pichler
a4d7da4973
feat: add ipv6 support (#32960) 
related with #34917
This PR updates the func utils get IP address function to support
getting a IPv6 address if it is available and return it. I've tested
that this works locally when running inside a docker container that has
an IPv6 network address.

This PR should address
https://github.com/milvus-io/milvus/discussions/20217

Thank you for review!

Signed-off-by: David Pichler <david.pichler@grainger.com>
Co-authored-by: David Pichler <david.pichler@grainger.com>
 2024-07-23 15:39:43 +08:00
chyezh
f6c6e98ec5
enhance: recover stack info when non-cgo thread crash (#34865) 
issue: #34864

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-07-23 14:25:44 +08:00
Jiquan Long
a2ac84bd64
feat: record the duration waiting in the proxy queue (#34744) 
fix: https://github.com/milvus-io/milvus/issues/34743

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2024-07-23 14:23:52 +08:00
chyezh
39c7e06bc5
enhance: add message and msgstream msgpack adaptor (#34874) 
issue: #33285

- make message builder and message conversion type safe
- add adaptor type and function to adapt old msgstream msgpack and
message interface

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-07-22 20:59:42 +08:00
wayblink
c2b8b5fe84
enhance: refine clustering compaction configs and logs (#34784) 
#30633

---------

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-21 19:23:40 +08:00
SimFG
16c002b51b
enhance: unified channel name generation and conversion (#34805) 
/kind improvement
- issue: #34804

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-07-20 23:11:39 +08:00
yihao.dai
b22e549844
enhance: Rename config of sealing by growing segmetns size (#34787) 
/kind improvement

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-19 20:27:41 +08:00
chyezh
86eff6e589
enhance: streaming node client implementation (#34653) 
issue: #33285

- add streaming node grpc client wrapper
- add unittest for streaming node grpc client side
- fix binary unsafe bug for message

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-07-19 17:37:40 +08:00
zhagnlu
f1b2f7b640
enhance: refactor bitmap index and internal hybrid index (#34450) 
#32900

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-07-18 10:39:42 +08:00
wayblink
c79d1af390
enhance: Add compaction task slot usage logic (#34581) 
#34544

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-18 10:27:41 +08:00
yihao.dai
4939f82d4f
enhance: Seal by total growing segments size (#34692) 
Seals the largest growing segment if the total size of growing segments
of each shard exceeds the size threshold(default 4GB). Introducing this
policy can help keep the size of growing segments within a suitable
level, alleviating the pressure on the delegator.

issue: https://github.com/milvus-io/milvus/issues/34554

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-17 21:45:41 +08:00
SimFG
203fb554a4
enhance: support to config root user's password (#34752) 
- issue: #33058

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-07-17 20:19:42 +08:00
congqixia
67324eb809
enhance: Add l0 segment entry num quota (#34733) 
See also #34670

This PR add quota configuration for l0 segment entry number per
collection. If l0 compaction cannot keep up the insertion/upsertion
rate, this feature could back press the related rate.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-17 17:35:41 +08:00
chyezh
3306bc2faf
fix: ut failure for grpc upgrade (#34726) 
issue: #34462

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-07-16 21:49:40 +08:00
chyezh
cc8f7aa110
fix: streaming service related fix patch (#34696) 
issue: #33285

- add idAlloc interface
- fix binary unsafe bug for message
- fix service discovery lost when repeated address with different server
id

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-07-16 15:49:38 +08:00
Xiaofan
be7760a9ab
fix: CVE by upgrading some dependencies. (#34462) 
fix #34434 and #34456
upgrade otelgrpc to fix CVE

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
 2024-07-16 11:55:36 +08:00
smellthemoon
b5b5afb63f
fix: [skip e2e]golang lint check fail (#34682) 
#34680

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-07-16 10:47:36 +08:00