Commit Graph

20677 Commits

Author SHA1 Message Date
Bingyi Sun
3641ae6611
fix: Fix index memory estimation (#35225)
issue: https://github.com/milvus-io/milvus/issues/35229

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-08-02 16:24:15 +08:00
congqixia
db06b86594
enhance: Sync otlp dependency version to fix security issue (#35192)
Related to #34434

otelgrpc DoS vulnerability due to unbound cardinality metrics

https://github.com/milvus-io/milvus/security/dependabot/91

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-02 16:22:20 +08:00
wayblink
81773bfadf
enhance: add commit time in partitionStats proto (#35125)
fix: #35110

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-02 16:16:14 +08:00
smellthemoon
475c333fa2
enhance: add valid_data in span (#35030)
#31728

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-02 15:40:14 +08:00
wei liu
f466129924
enhance: add segment state on data coord bin log size metrics (#35153)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-02 14:48:14 +08:00
wei liu
3dd3749f0b
enhance: Avoid unnecesary syncTargetVersion func call after querycoord recover (#34954)
before querycoord stop gracefully, we will save the current target to
meta store and recover it after querycoord start up, to speed the
querycoord's recovery time. but the target version hasn't been recovered
as expected, and it use latest timestamp as current target's version,
which has no effect to querycoord but an unnecessary syncTargetVersion
func call.

This PR recover the correct target version to avoid unnecessary
syncTargetVersion func call

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-02 14:46:13 +08:00
congqixia
095d77269b
enhance: Set WatchProgress for channel operation progress response (#35183)
The watch progress is always zero in CheckChannelOperationProgress
response, which is meaningless and confusing. This PR set progress value
in rpc response to fix this problem.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-02 12:06:14 +08:00
aoiasd
3655ab10b2
fix: evict paramtable cache miss (#34771)
relate: https://github.com/milvus-io/milvus/issues/33461

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-08-02 11:52:14 +08:00
congqixia
c64a078458
enhance: Support proxy/delegator qn client pooling (#35194)
See also #35196

Add param item for proxy/delegator query node client pooling and
implement pooling logic

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-02 11:24:19 +08:00
wei liu
3b735b4b02
enhance: Refine param init for MmapDirPath (#35181)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-02 11:12:14 +08:00
wayblink
95462668ca
enhance: unify time in clustering compaction task to unix (#35167)
#34495

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-02 10:30:19 +08:00
wei liu
27b6d58981
fix: Set legacy level to l0 segment after qc restart (#35197)
issue: #35087
after qc restarts, and target is not ready yet, if dist_handler try to
update segment dist, it will set legacy level to l0 segment, which may
cause l0 segment be moved to other node, cause search/query failed.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-02 10:18:13 +08:00
Jiquan Long
4dbd023402
enhance: add metric to record maxInsertRate and queryNodeMemoryHighWaterLevel (#35185)
fix: https://github.com/milvus-io/milvus/issues/35177

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-02 10:14:13 +08:00
cai.zhang
196a7986b3
enhance: Change the fixed value to a ratio for clustering segment size (#35076)
issue: #34495

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-01 22:04:14 +08:00
congqixia
a642a26ed4
enhance: Resolve ChunkFileWriter lint issue (#35166)
See also #34483

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-01 16:52:13 +08:00
yihao.dai
7721a28584
fix: Fix unstable TestDispatchToVchannels ut (#35163)
issue: https://github.com/milvus-io/milvus/issues/35150

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-01 14:56:13 +08:00
wayblink
5bbb1c201c
enhance:support l2 single compaction (#34935)
#34928

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-01 14:36:13 +08:00
zhagnlu
f8c1b138a8
fix:fix get array error for int type (#35154)
#35055

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-01 14:30:12 +08:00
Ted Xu
b4d0f4df0a
enhance: update param documents (#35152)
See: #35161

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-08-01 12:38:13 +08:00
zhenshan.cao
aa247f192d
enhance: remove unused code for StorageV2 (#35132)
issue: https://github.com/milvus-io/milvus/issues/34168

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-08-01 12:08:13 +08:00
cai.zhang
9412002d7d
fix: Fix data race for clustering buffer writer (#35145)
issue: #34495

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-01 11:20:13 +08:00
cai.zhang
089fb2b5fb
enhance: Skip updating the level that are already at the specified level (#35094)
issue: #34495

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-01 11:18:12 +08:00
Bingyi Sun
f229f244d2
enhance: add chunk basic impl (#34634)
https://github.com/milvus-io/milvus/issues/35112
This pr would not affect milvus functionality by now.
It implments a Chunk memory layout that looks like 
```
VariableColumn
|offset|offset|offset|
|data|data|data|

```
We maybe move offsets to the beginning and add null bitmaps later but
not in this PR.
And mmap test will also be added in another PR.

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-08-01 10:29:51 +08:00
wei liu
e9d61daa3f
enhance: Reduce delegator memory overloaded factor to 0.1 (#35092)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-01 10:21:50 +08:00
congqixia
dfda9f0478
enhance: Add depguard rules to ban deprecated proto lib (#35140)
See also #34394 #34252

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-01 10:01:49 +08:00
sre-ci-robot
c15d074b21
[automated] Update Knowhere Commit (#35157)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-08-01 01:17:49 +08:00
Gao
6695c6d0a3
enhance: add channel num for queryHook optimization (#35104)
At most cases, data in each channel is almost evenly distributed, we
could utilize the channel num info to optimize searh param in queryHook

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-07-31 18:59:49 +08:00
wei liu
03912a8788
enhance: Avoid balance stuck after segment list become stable (#34728)
issue: #34715
if collection's segment list doesn't changes anymore, then the next
target will be empty at most time, and balance segment will check
whether segment exist in both current and next target, so the balance
cloud be blocked due to next target is empty.

This PR permit segment to be moved if next target is empty, to avoid
balance stuck.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-31 18:09:48 +08:00
Patrick Weizhi Xu
a9744d62b1
enhance: improve error msg when expr invalid during isolation (#35127)
issue: #34336 

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit b1d7b24972e9414ab9488c7611aca9975e674cd4)
2024-07-31 17:23:51 +08:00
smellthemoon
6106a48acb
fix: upsert result use the previous pk (#34672)
#34668

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-07-31 15:25:51 +08:00
Patrick Weizhi Xu
acaa78db58
fix: disable sparse and binary in MV (#35126)
issue: #35120 

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit 0d340ebe7412cd85963825bc8abd3c874e339679)
2024-07-31 15:02:01 +08:00
congqixia
f7f9a729c9
enhance: Pre-allocate space for reduce data structure (#35118)
Grow slice & map.growWork may cause a lot when segment number is large
for big K query. This PR pre-allocate space for reduce methods to avoid
this cost.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-31 10:35:49 +08:00
foxspy
48b14c56f6
fix: add prometheus dependency for milvus_monitor (#35078)
add prometheus dependency for monitor module. Or else Some compilers may
report a compilation failure.
issue: #35077

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-07-31 10:15:48 +08:00
wayblink
f919b6e1b8
enhance: refine error handle in clustering compaction task (#34956)
related: #30633 #34792

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-30 20:37:57 +08:00
wayblink
ce3f836876
fix: compaction task not be cleaned correctly (#34765)
1.fix compaction task not be cleaned correctly
2.add a new parameter to control compaction gc loop interval
3.remove some useless configs of clustering compaction

bug: #34764

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-30 20:21:56 +08:00
yihao.dai
39d25938c5
enhance: Avoid panic due to nil schema (#35063)
/kind improvement

issue: https://github.com/milvus-io/milvus/discussions/25620

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-30 20:18:32 +08:00
zhagnlu
dd0c26cf58
enhance: redefine variable column block size (#35040)
#35013

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-30 19:23:50 +08:00
yellow-shine
273e9203f4
feat: [skip e2e] effective way to use cache (#34774) (#35109)
See also #34483

Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-07-30 18:41:51 +08:00
congqixia
b787bb4783
enhance: [skip e2e] Skip Amazon&Rockylinux instead of commenting job (#35108)
Related to #35084

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-30 16:23:49 +08:00
congqixia
de8a266d8a
enhance: Enable linux code checker (#35084)
See also #34483

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-30 15:53:51 +08:00
wei liu
4bb969ef8f
enhance: Skip manual stopped component during health check (#34953)
after manual stop component by management restful api, `healthz` may
return unhealthy state. k8s may restart the pod to save the unhealthy
sate, and the manual stop operation will got unexpected result.

to solve this, we make `healthz` API skip the manual stopped component.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-30 14:37:51 +08:00
congqixia
cabb200498
enhance: [GoSDK] Expose indexing progress in DescribeIndex result (#35066)
See also #31293

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-30 10:33:49 +08:00
zhagnlu
a8a4779749
fix: fix parse plan proto failed for search type (#34944)
#25848

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-29 21:19:49 +08:00
congqixia
d36cfe71e5
enhance: Refine protobuf dependency installation in Makefile (#35072)
Related to #34394

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-29 21:15:48 +08:00
zhagnlu
86322e0468
fix: fix query count(*) concurrently (#35007)
#34778
#34849
fix two problems:
1. count(*) incorrect, if growing insert duplicated (pk, timestamp)
pairs that pk and timestamp all same, need to keep just one pair.
2. count(*) may core dump, if get_real_count interface get snapshot and
do mvcc at not consistency status, mainly happens under concurrency.

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-29 19:53:50 +08:00
congqixia
972752258a
enhance: Support otlp http exporter (#35053)
See also #35052

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-29 17:43:49 +08:00
chyezh
a495754034
enhance: the datacoord gc should fast quitable (#35050)
issue: #35049

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-29 17:03:48 +08:00
yellow-shine
e62f6b2f4e
enhance: [skip e2e] enlarge timeout of UT for go up to 90 mins (#35035)
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-07-29 15:51:49 +08:00
Patrick Weizhi Xu
3c7f73137e
fix: disallow expr when partition key isolation is enabled (#35031)
issue: #34336 

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit 2889481ce9a14230e9c7f1c8f9c3c1decde77e03)
2024-07-29 14:21:50 +08:00
SimFG
0e41f104c5
enhance: use the collection id to group msg pack in the msg dispatcher (#34871)
- issue: #34870

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-29 11:45:49 +08:00