678 Commits

Author	SHA1	Message	Date
jaime	12ed40e125	fix: some missing metrics for WebUI restful API (#37747) ... issue: #36621 Signed-off-by: jaime <yun.zhang@zilliz.com>	2024-11-18 14:00:32 +08:00
congqixia	b0bd290a6e	enhance: Use internal json(sonic) to replace std json lib (#37708) ... Related to #35020 Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>	2024-11-18 10:46:31 +08:00
sthuang	ff00a12805	enhance: RBAC custom privilege group ut coverage (#37558) ... issue: https://github.com/milvus-io/milvus/issues/37031 Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>	2024-11-09 20:40:25 +08:00
sthuang	70605cf5b3	enhance: Support custom privilege group for RBAC (#37087) ... issue: #37031 --------- Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>	2024-11-09 08:44:28 +08:00
yihao.dai	ebc3c82761	fix: Fix subscription leak (#37382) ... Close (unsubscribe) the msg stream after completing the `PreCreatedTopic` check to prevent backlog issue. issue: https://github.com/milvus-io/milvus/issues/36021 Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-11-08 16:16:26 +08:00
Zhen Ye	cae9e1c732	fix: drop collection failed if enable streaming service (#37444) ... issue: #36858 - Start channel manager on datacoord, but with empty assign policy in streaming service. - Make collection at dropping state can be recovered by flusher to make sure that milvus consume the dropCollection message. - Add backoff for flusher lifetime. - remove the proxy watcher from timetick at rootcoord in streaming service. Also see the better fixup: #37176 --------- Signed-off-by: chyezh <chyezh@outlook.com>	2024-11-07 10:26:26 +08:00
SimFG	f1dd55e0c0	enhance: improve rootcoord task scheduling policy (#37352) ... - issue: #30301 Signed-off-by: SimFG <bang.fu@zilliz.com>	2024-11-06 14:48:26 +08:00
jaime	9d16b972ea	feat: add tasks page into management WebUI (#37002) ... issue: #36621 1. Add API to access task runtime metrics, including: - build index task - compaction task - import task - balance (including load/release of segments/channels and some leader tasks on querycoord) - sync task 2. Add a debug model to the webpage by using debug=true or debug=false in the URL query parameters to enable or disable debug mode. Signed-off-by: jaime <yun.zhang@zilliz.com>	2024-10-28 10:13:29 +08:00
SimFG	1cc9cb49ad	enhance: allow to delete data when disk quota exhausted (#37134) ... - issue: #37133 Signed-off-by: SimFG <bang.fu@zilliz.com>	2024-10-25 16:47:29 +08:00
jaime	4746f47282	feat: management WebUI homepage (#36822) ... issue: #36784 1. Implement an embedded web server for WebUI access. 2. Complete the homepage development. Home page demo: <img width="2177" alt="iShot_2024-10-10_17 57 34" src="https://github.com/user-attachments/assets/38539917-ce09-4e54-a5b5-7f4f7eaac353"> Signed-off-by: jaime <yun.zhang@zilliz.com>	2024-10-23 11:29:28 +08:00
congqixia	30121a5a0d	fix: Rectify delete buffer row count quota value (#37060) ... Related to #37057 Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>	2024-10-23 10:15:29 +08:00
congqixia	70339820fa	fix: Check whether new collection name is alias (#36981) ... Related to #36963 --------- Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>	2024-10-21 10:25:25 +08:00
XuanYang-cn	0516624ab0	enhance: Log when limit writing by deletion (#36954) ... See also: #36953 Signed-off-by: yangxuan <xuan.yang@zilliz.com>	2024-10-18 17:25:25 +08:00
Bingyi Sun	6851738fd1	fix: fix make generate-mockery panic with go1.22 (#36830) ... https://github.com/milvus-io/milvus/issues/36831 Fix `make generate-mockery` panic. Signed-off-by: sunby <sunbingyi1992@gmail.com>	2024-10-17 12:11:31 +08:00
smellthemoon	1375d690dd	enhance: adjust some err msg (#36840) ... related with #36495 Signed-off-by: lixinguo <xinguo.li@zilliz.com> Co-authored-by: lixinguo <xinguo.li@zilliz.com>	2024-10-17 12:01:25 +08:00
yihao.dai	1bd3228635	enhance: Ignore db not found error in quota center (#36821) ... In quota center, ignore the "DB not found error" to prevent it from affecting the rate limiting of other databases. /kind improvement --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-10-15 15:55:22 +08:00
jaime	5713620825	enhance: skip alter operation when no change are detected (#36785) ... issue: #36784 Signed-off-by: jaime <yun.zhang@zilliz.com>	2024-10-12 11:25:20 +08:00
congqixia	116d7c18cc	fix: Use metrics values instead of hardcode mock values (#36782) ... Resolves #36781 Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>	2024-10-11 20:09:20 +08:00
wei liu	3cd0b26285	enhance: Enable dynamic update loaded collection's replica (#35822) ... issue: #35821 After collection loaded, if we need to increase/decrease collection's replica, we need to release and load it again. milvus offers 4 solution to update loaded collection's replica, this PR aims to dynamic change the replica number without release, and after replica number changed, milvus will execute load replica or release replica in async, and the replica loaded status can be checked by getReplicas API. Notice that if set too much replicas than querynode can afford，the new replica won't be loaded successfully until enough querynode joins. --------- Signed-off-by: Wei Liu <wei.liu@zilliz.com>	2024-09-25 10:13:18 +08:00
aoiasd	da227ff9a1	feat: Support create collection with functions (#35973) ... relate: https://github.com/milvus-io/milvus/issues/35853 Support create collection with functions. Prepare for support bm25 function. --------- Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>	2024-09-12 10:43:06 +08:00
wei liu	32e55a02ea	fix: Fix privilege group hasn't been register for validate (#35937) ... issue: #35471 --------- Signed-off-by: Wei Liu <wei.liu@zilliz.com>	2024-09-05 15:35:04 +08:00
congqixia	8593c4580a	enhance: Add delete buffer related quota logic (#35918) ... See also #35303 --------- Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>	2024-09-05 11:39:03 +08:00
smellthemoon	80a9efd96d	fix: not check nullable and default value in pk field (#35987) ... #35926 Signed-off-by: lixinguo <xinguo.li@zilliz.com> Co-authored-by: lixinguo <xinguo.li@zilliz.com>	2024-09-05 11:27:04 +08:00
jaime	24fb10114b	enhance: remove cooling off in rate limiter for read requests (#35935) ... issue: #35934 Signed-off-by: jaime <yun.zhang@zilliz.com>	2024-09-04 14:39:10 +08:00
SimFG	311f860676	enhance: support to drop the role which is related the privilege list (#35727) ... - issue: #35545 Signed-off-by: SimFG <bang.fu@zilliz.com>	2024-08-30 15:17:00 +08:00
jaime	b0ac04d104	enhance: set database properties to restrict read access (#35745) ... issue: #35744 Signed-off-by: jaime <yun.zhang@zilliz.com>	2024-08-29 13:17:01 +08:00
Zhen Ye	99dff06391	enhance: using streaming service in insert/upsert/flush/delete/querynode (#35406) ... issue: #33285 - using streaming service in insert/upsert/flush/delete/querynode - fixup flusher bugs and refactor the flush operation - enable streaming service for dml and ddl - pass the e2e when enabling streaming service - pass the integration tst when enabling streaming service --------- Signed-off-by: chyezh <chyezh@outlook.com> Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-08-29 10:03:08 +08:00
yihao.dai	56b3490a98	fix: Fix rate wasn't limited to the expected value (#35699) ... Each time the rate is reset, the token bucket is fully refilled, causing the rate wasn't limited to the expected value. This PR addresses the issue by preventing the token reset. issue: https://github.com/milvus-io/milvus/issues/35675, https://github.com/milvus-io/milvus/issues/35702 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-08-28 14:13:00 +08:00
jaime	d8aa01bc1a	fix: inconsistent meta view causes rate limit invalid (#35665) ... issue: #35663 Signed-off-by: jaime <yun.zhang@zilliz.com>	2024-08-28 11:21:04 +08:00
jaime	7d3c0d748c	fix: the incorrect number of partitions in rootcoord meta (#35600) ... issue: #35698 Signed-off-by: jaime <yun.zhang@zilliz.com>	2024-08-26 15:20:58 +08:00
smellthemoon	098c10922b	enhance: add some check when create collection (#35629) ... not support nullable==true in vector type. check it when create collection. Signed-off-by: lixinguo <xinguo.li@zilliz.com> Co-authored-by: lixinguo <xinguo.li@zilliz.com>	2024-08-25 16:23:05 +08:00
wei liu	5c245d51c4	enhance: Refresh proxy cache after restore rbac meta (#35635) ... issue: #35443 --------- Signed-off-by: Wei Liu <wei.liu@zilliz.com>	2024-08-22 19:09:01 +08:00
zhagnlu	3107701fe8	enhance: optimize retrieve on dynamic field (#35580) ... #35514 Signed-off-by: luzhang <luzhang@zilliz.com> Co-authored-by: luzhang <luzhang@zilliz.com> Co-authored-by: zhenshan.cao <zhenshan.cao@zilliz.com>	2024-08-22 14:24:56 +08:00
Zhen Ye	4d69898cb2	enhance: support single pchannel level transaction (#35289) ... issue: #33285 - support transaction on single wal. - last confirmed message id can still be used when enable transaction. - add fence operation for segment allocation interceptor. --------- Signed-off-by: chyezh <chyezh@outlook.com>	2024-08-19 21:22:56 +08:00
wei liu	1d49358f82	enhance: Add BackupRBAC/RestoreRBAC API to enable rbac backup (#35444) ... issue: #35443 --------- Signed-off-by: Wei Liu <wei.liu@zilliz.com>	2024-08-16 10:10:53 +08:00
yihao.dai	e58d656e65	fix: Fix compile error (#35442) ... issue: https://github.com/milvus-io/milvus/issues/33285 Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-08-13 15:46:19 +08:00
yihao.dai	efadf22802	enhance: Append create partition msg to wal (#35398) ... issue: https://github.com/milvus-io/milvus/issues/33285 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-08-13 14:28:20 +08:00
chyezh	16b0aee97f	enhance: timetick interceptor optimization (#35287) ... issue: #33285 - remove redundant goroutine by using insepctor. - remove the coutinous non-message timetick persistence - periodically push the time tick forward without persistent timetick message. - add 'message type filter' deliver filter. Signed-off-by: chyezh <chyezh@outlook.com>	2024-08-12 18:58:25 +08:00
yihao.dai	72a175478f	enhance: Append drop partition msg to wal (#35326) ... issue: https://github.com/milvus-io/milvus/issues/33285 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-08-07 17:28:16 +08:00
Jiquan Long	b5022b0647	enhance: add metric to record disk quota (#35305) ... https://github.com/milvus-io/milvus/issues/35177 Signed-off-by: longjiquan <jiquan.long@zilliz.com>	2024-08-07 10:26:16 +08:00
Jiquan Long	4dbd023402	enhance: add metric to record maxInsertRate and queryNodeMemoryHighWaterLevel (#35185) ... fix: https://github.com/milvus-io/milvus/issues/35177 --------- Signed-off-by: longjiquan <jiquan.long@zilliz.com>	2024-08-02 10:14:13 +08:00
congqixia	de8a266d8a	enhance: Enable linux code checker (#35084) ... See also #34483 --------- Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>	2024-07-30 15:53:51 +08:00
wei liu	c45f38aa61	enhance: Update protobuf-go to protobuf-go v2 (#34394) ... issue: #34252 Signed-off-by: Wei Liu <wei.liu@zilliz.com>	2024-07-29 11:31:51 +08:00
chyezh	d93c51d5f6	enhance: change GetVChannels rpc into GetPChannelInfo (#34940) ... issue: #33285 Signed-off-by: chyezh <chyezh@outlook.com>	2024-07-25 13:29:45 +08:00
Jiquan Long	b843c91bad	feat: record the number of times milvus turns into force-deny-writing states (#34960) ... /kind improvement fix: #34961 --------- Signed-off-by: longjiquan <jiquan.long@zilliz.com>	2024-07-25 11:39:46 +08:00
chyezh	39c7e06bc5	enhance: add message and msgstream msgpack adaptor (#34874) ... issue: #33285 - make message builder and message conversion type safe - add adaptor type and function to adapt old msgstream msgpack and message interface --------- Signed-off-by: chyezh <chyezh@outlook.com>	2024-07-22 20:59:42 +08:00
yihao.dai	8aab6cbfac	enhance: Organize the common modules of streamingNode and dataNode (#34773) ... 1. Move the common modules of streamingNode and dataNode to flushcommon 2. Add new GetVChannels interface for rootcoord issue: https://github.com/milvus-io/milvus/issues/33285 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-07-22 11:33:51 +08:00
SimFG	16c002b51b	enhance: unified channel name generation and conversion (#34805) ... /kind improvement - issue: #34804 Signed-off-by: SimFG <bang.fu@zilliz.com>	2024-07-20 23:11:39 +08:00
SimFG	203fb554a4	enhance: support to config root user's password (#34752) ... - issue: #33058 Signed-off-by: SimFG <bang.fu@zilliz.com>	2024-07-17 20:19:42 +08:00
congqixia	67324eb809	enhance: Add l0 segment entry num quota (#34733) ... See also #34670 This PR add quota configuration for l0 segment entry number per collection. If l0 compaction cannot keep up the insertion/upsertion rate, this feature could back press the related rate. --------- Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>	2024-07-17 17:35:41 +08:00