wei liu
14ec3dc357
enhance: Enable ReadOnly/ReadWrite/Admin Privilege Group to simplify RBAC grant progress ( #35472 ) ( #35543 )
...
issue: #35471
pr: #35472 #35515
---------
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-19 16:24:54 +08:00
wei liu
248a6ea401
enhance: Add BackupRBAC/RestoreRBAC API to enable rbac backup ( #35444 ) ( #35513 )
...
issue: #35443
pr: #35444
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-18 13:42:55 +08:00
congqixia
4e098f1b23
enhance: Bump milvus & proto to v2.4.9 ( #35516 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-16 16:24:54 +08:00
congqixia
c8926aee23
enhance: [2.4] Support dynamic field in SchemaHelper ( #35461 ) ( #35469 )
...
Cherry-pick from master
pr: #35461
Related to #35415
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-15 10:02:54 +08:00
Ted Xu
57d4bcbf15
enhance: adding the msgchannel section in generated yaml ( #35466 )
...
See #32168
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-08-14 19:03:11 +08:00
congqixia
e44c63883b
enhance: Bump milvus & proto to v2.4.8 ( #35460 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-14 12:52:52 +08:00
Ted Xu
ce53e79f12
fix: enable milvus.yaml check ( #34567 ) ( #35446 )
...
See #32168
pr: #34567 #35152
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-08-13 19:00:23 +08:00
aoiasd
a20cb727eb
enhance:[Cherry-pick] Check by proxy rate limiter when delete get data by query. ( #30891 ) ( #35262 )
...
relate: https://github.com/milvus-io/milvus/issues/30927
pr: https://github.com/milvus-io/milvus/pull/30891
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-08-13 14:32:21 +08:00
congqixia
bafc5a831a
enhance: [2.4] Remove stackerror imported unexpected ( #35256 ) ( #35330 )
...
Cherry-pick from master
pr: #35256
`stackerror` pkg is imported by accident and could be replaced by
cockroachdb errors lib. This PR removes import line and fix this
problem.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-12 11:12:20 +08:00
zhagnlu
db9e4b898a
enhance: support bitmap index ( #35336 )
...
pr: #32902
cherry-pick bitmap index from master
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-07 20:12:18 +08:00
wei liu
0201e00a2f
enhance: enable to set load config in cluster level ( #35293 )
...
issue: #35170
pr: #35169
This PR enable to set load configs in cluster level, such as replicas
and resource groups. then when load collections will use the load
config.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-07 12:38:21 +08:00
Jiquan Long
d14d00b0cc
enhance: add metric to record disk quota ( #35305 ) ( #35306 )
...
https://github.com/milvus-io/milvus/issues/35177
pr: https://github.com/milvus-io/milvus/pull/35305
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-07 10:24:27 +08:00
wei liu
2ac1bf7532
enhance: Enable setting the replica number and resource group during collection creation ( #34403 ) ( #34561 )
...
issue: #30040
pr: #34403
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-06 15:06:17 +08:00
congqixia
eebd9804d6
enhance: Bump milvus & proto version to v2.4.7 ( #35034 )
...
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-06 12:40:16 +08:00
cai.zhang
2534b30e39
enhance: [cherry-pick] Add monitoring metrics for task execution time in datacoord ( #35141 )
...
issue: #35138
master pr: #35139
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-02 19:46:16 +08:00
wei liu
2609e95cb5
enhance: add segment state on data coord bin log size metrics ( #35153 ) ( #35235 )
...
pr: #35153
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-02 18:28:14 +08:00
wei liu
d767f8977a
enhance: Refine param init for MmapDirPath ( #35181 ) ( #35214 )
...
pr: #35181
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-02 16:30:15 +08:00
congqixia
f8444b900f
enhance: [2.4] Support proxy/delegator qn client pooling ( #35195 )
...
Cherry pick from master
pr: #35194
See also #35196
Add param item for proxy/delegator query node client pooling and
implement pooling logic
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-02 11:24:19 +08:00
Jiquan Long
ae3657b23e
enhance: add metric to record maxInsertRate and queryNodeMemoryHighWaterLevel ( #35185 ) ( #35188 )
...
pr: https://github.com/milvus-io/milvus/pull/35185
fix: https://github.com/milvus-io/milvus/issues/35177
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-02 10:16:15 +08:00
wei liu
5f601fcc50
enhance: Reduce delegator memory overloaded factor to 0.1 ( #35092 ) ( #35164 )
...
pr: #35092
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-01 14:20:13 +08:00
Gao
be0123863f
enhance: add channel num for queryHook optimization ( #35105 )
...
pr: #35104
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-07-31 18:23:51 +08:00
cai.zhang
c340f387cf
enhance: [cherry-pick] Change the fixed value to a ratio for clustering segment size ( #35075 )
...
issue: #34495
master pr: #35076
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-31 10:32:00 +08:00
congqixia
935a117396
enhance: [2.4] Support otlp http exporter ( #35053 ) ( #35073 )
...
Cherry-pick from master
pr: #35053
See also #35052
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-30 21:00:08 +08:00
Jiquan Long
86edca8c1b
fix: support auto index for array ( #35095 )
...
/kind branch-feature
pr: #34450
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
Co-authored-by: Zhagnlu <lu.zhang@zilliz.com>
2024-07-30 17:57:50 +08:00
PowderLi
fa9d606aa8
enhance: add metrics for restful apis ( #35083 )
...
issue: #34968
pr: #34969
Signed-off-by: PowderLi <min.li@zilliz.com>
2024-07-30 14:13:49 +08:00
yihao.dai
cc188fb322
enhance: Avoid panic due to nil schema ( #35063 ) ( #35064 )
...
/kind improvement
issue: https://github.com/milvus-io/milvus/discussions/25620
pr: https://github.com/milvus-io/milvus/pull/35063
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-29 20:07:49 +08:00
congqixia
d16320705e
enhance: [2.4] Add Segment Level in milvus segment info APIs ( #34763 ) ( #35023 )
...
Cherry-pick from master
pr: #34763
See also #34746
This PR add segment level field in response of
`GetPersistentSegmentInfo` and `GetQuerySegmentInfo`
---------
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-29 10:11:52 +08:00
wei liu
b3bc7f3985
enhance: Limit collection's normal balance speed ( #34810 ) ( #34987 )
...
issue: #34798
pr: #34810
after we remove the task priority on query coord, to avoid load/release
segment blocked by too much balance task, we limit the balance task size
in each round. at same time, we reduce the balance interval to trigger
balance more frequently.
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-26 10:13:46 +08:00
Jiquan Long
2822d8729c
feat: record the number of times milvus turns into force-deny-writing states ( #34960 ) ( #34989 )
...
/kind improvement
fix : #34961
pr: #34960
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-07-25 15:43:09 +08:00
jaime
77ae127a62
fix: check collection health(queryable) fail for releasing collection ( #34948 )
...
issue: #34946
pr: #34947
---------
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-25 10:25:57 +08:00
cai.zhang
12a24c3957
enhance: [cherry-pick] Refine code for GetRecoveryInfo ( #34974 )
...
issue: #34495
master pr: #34973
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-24 22:01:44 +08:00
cai.zhang
74adedf750
enhance: Optimized the GC logic to ensure that memory is released in time ( #34950 )
...
issue: #34703
master pr: #34949
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-24 14:07:43 +08:00
chyezh
0f1bb9c697
enhance: recover stack info when non-cgo thread crash ( #34866 )
...
issue: #34864
pr: #34865
Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-23 14:21:46 +08:00
aoiasd
48063252b1
fix: [Cheryy-Pick] evict paramtable cache failed ( #34772 )
...
relate: https://github.com/milvus-io/milvus/issues/33461
pr: https://github.com/milvus-io/milvus/pull/34771
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-07-23 12:25:44 +08:00
congqixia
7540714c1b
enhance: [2.4] Add l0 segment entry num quota ( #34733 ) ( #34837 )
...
Cherry-pick from master
pr: #34733
See also #34670
This PR add quota configuration for l0 segment entry number per
collection. If l0 compaction cannot keep up the insertion/upsertion
rate, this feature could back press the related rate.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-22 12:05:43 +08:00
wayblink
c0c3c5f528
enhance: [cherry-pick] refine clustering compaction configs and logs ( #34818 )
...
issue: #30633
pr: #34784
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-21 19:27:41 +08:00
yihao.dai
07bc1b6717
enhance: Seal by total growing segments size ( #34692 ) ( #34779 )
...
Seals the largest growing segment if the total size of growing segments
of each shard exceeds the size threshold(default 4GB). Introducing this
policy can help keep the size of growing segments within a suitable
level, alleviating the pressure on the delegator.
issue: https://github.com/milvus-io/milvus/issues/34554
pr: https://github.com/milvus-io/milvus/pull/34692
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-19 18:25:50 +08:00
Jiquan Long
34873a7f76
feat: record the duration waiting in the proxy queue ( #34744 ) ( #34788 )
...
fix: https://github.com/milvus-io/milvus/issues/34743
pr: #34744
---------
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-07-19 15:37:44 +08:00
SimFG
0e226502e4
enhance: [2.4] pick default root password and log level pr ( #34777 )
...
default root password
- issue: #33058
- pr: #34752
set log level
- issue: #34756
- pr: #34757
---------
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-18 13:45:43 +08:00
wayblink
a26e965e6a
enhance:[cherry-pick] Add compaction task slot usage logic ( #34625 )
...
issue: #34544
pr: #34581
---------
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-18 09:55:43 +08:00
wayblink
83fc26c31a
fix: [cherry-pick] compaction task not be cleaned correctly ( #34766 )
...
1.fix compaction task not be cleaned correctly
2.add a new parameter to control compaction gc loop interval
3.remove some useless configs of clustering compaction
bug: #34764
pr: #34765
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-17 22:17:42 +08:00
Patrick Weizhi Xu
3cf2e1831a
feat: [cherry-pick][2.4] support partition key isolation ( #34336 ) ( #34613 )
...
issue: #34332
pr: #34336
---------
Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit 104d0966b7
2024-07-17 14:03:43 +08:00
Xiaofan
66b5cbe27c
fix: CVE by upgrading some dependencies. ( #34693 )
...
fix #34434 and #34456
upgrade otelgrpc to fix CVE
Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-07-16 14:05:37 +08:00
wei liu
cf701a9bf0
enhance: Preserve fixed-size memory in delegator node for growing segment ( #34600 )
...
issue: #34595
pr: #34596
When consuming insert data on the delegator node, QueryCoord will move
out some sealed segments to manage its memory usage. After the growing
segment gets flushed, some sealed segments from other workers will be
moved back to the delegator node. To avoid the frequent movement of
segments, we estimate the maximum growing row count and preserve a
fixed-size memory in the delegator node.
---------
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-15 20:51:46 +08:00
congqixia
10c04f33c7
enhance: [2.4] Add param item for segmentFlushInterval ( #34629 ) ( #34663 )
...
Cherry-pick from master
pr: #34629
See also #28817
Add paramitem for segment flush interval
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-15 14:31:40 +08:00
wayblink
9dc98ac76d
enhance: [cherry-pick] Refine clustering compaction latency metrics ( #34539 )
...
issue: #34527
pr: #34528
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-11 17:43:43 +08:00
edward.zeng
3d29ebfffb
enhance: Refine default histogram metric bucket ( #34592 )
...
/kind improvement
pr: https://github.com/milvus-io/milvus/pull/34580
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
2024-07-11 16:51:35 +08:00
jaime
073a501fc7
enhance: skip orphan channel cp meta when checking cp lag ( #34556 )
...
issue: #34545
pr: #34555
Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-11 09:36:56 +08:00
yihao.dai
cbba5481f8
fix: Prevent dispatcher merging if curTs is 0 ( #34562 ) ( #34563 )
...
When the main dispatcher has not yet consumed data, curTs is 0. During
this time, merging dispatchers should not be allowed; otherwise, the
data of the solo dispatcher will be skipped.
issue: https://github.com/milvus-io/milvus/issues/34255
pr: https://github.com/milvus-io/milvus/pull/34562
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-10 22:46:51 +08:00
SimFG
15adb2feac
enhance: [2.4] add the seal segment when dispatch delete msgs ( #34566 )
...
/kind improvement
- pr: #34565
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-10 19:18:51 +08:00
