Commit Graph

19257 Commits

Author SHA1 Message Date
aoiasd
4ef6217fc0
enhance: add pool for scheduler dmQueue task (#30103)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-01-19 14:20:54 +08:00
congqixia
15716ec283
fix: Move compress binlog after check result GetCompactionResult error (#30127)
See also: #28873

When datanode returns error or go offline during GetCompactionResult
call, the compress binlog logic will panic since it was using a nil
result

This PR move it after the CheckRPCCall error to prevent this case.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-19 14:12:55 +08:00
yah01
f542bdbf3c
enhance: calc the accurate mem size of segment (#30093)
this stats the real memory size of segment, also reduces the memory
usage in mmap mode
resolve #30095

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-19 12:32:53 +08:00
Bingyi Sun
dad27cbb40
fix: fix wrong log file name when running as mixture type (#29979)
See also: #25323, #29969
many users reported log file name is incorrect when starting in mixture
type.

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-01-19 11:48:54 +08:00
XuanYang-cn
86f48861c1
fix: Add more throughput in related metrics (#30038)
This PR also fixes bugs in l0 compactor where
l0 results would never be removed from datanode

See also: #30099

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-19 11:34:54 +08:00
SimFG
ddccccbcab
enhance: add the bytes data type for merge data and format some code (#30105)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-01-18 22:18:55 +08:00
smellthemoon
e52ce370b6
enhance:don't store logPath in meta to reduce memory (#28873)
don't store logPath in meta to reduce memory, when service get
segmentinfo, generate logpath from logid.
#28885

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-18 22:06:31 +08:00
wei liu
fc56ed5c27
fix: Proxy restful api doesn't register (#30072)
issue: #30074
This PR fix that management restful api in proxy doesn't register to
http service

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-18 20:28:52 +08:00
zhuwenxing
c91254f762
test: update pyarrow version (#29992)
fix vulnerability alerts

Signed-off-by: zhuwenxing <wenxing.zhu@zilliz.com>
2024-01-18 17:56:54 +08:00
PowderLi
631ebb1f07
fix: CollectionSchema.autoID is deprecated (#30002)
issue: #30000
related to: [milvus-proto
#202](https://github.com/milvus-io/milvus-proto/pull/202)

1. replace collSchema.AutoID with primaryField.AutoID
2. show `enableDynamic` & `enableDynamicField` at the same time
3. avoid data race about the access to metacache

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-01-18 16:16:52 +08:00
congqixia
9f8eb0e527
enhance: make integration test case timeout configurable (#30073)
currently integration test may timeout if any case run time is above 3
minutes. This duration was hard coded.

This PR change this duration into a customized parameter and could be
passed via test running commands.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-18 12:22:54 +08:00
yah01
0d4e781f69
fix: the system rejects all queries and never recovers if enabled read rate limit (#30061)
fix #30060

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-17 23:30:55 +08:00
xige-16
91aa81b4d7
fix: Add more checks to rank params (#29950)
issue: #29840 #29867
/kind bug

Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-01-17 20:28:58 +08:00
xige-16
fa7cf587b0
enhance: Opt metric type does not match error message (#29927)
issue: #29791 
/kind improvement
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2024-01-17 20:25:03 +08:00
sre-ci-robot
6fd432593c Update all contributors
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2024-01-17 12:00:50 +00:00
wei liu
f8695aef9d
fix: Trigger leader checker too frequency (#29991)
issue: #29841
This PR fix leader checker use wrong check interval, which causes leader
checker trigger too frequency

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-17 19:40:53 +08:00
congqixia
4c93912135
enhance: Shuffle candidates before channel assignment (#30066)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-17 19:34:53 +08:00
aoiasd
04e21564b6
enhance: delete should inc ProxyFunctionCall metric failed label when run failed instead abandon (#29621)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-01-17 19:16:55 +08:00
wei liu
57bd3e2181
fix: Leader checker canot submit load task (#30067)
issue: #29841
if segment loaded, submit load segment task for it isn't permitted, to
avoid load segment twice. but this logic blocks the leader checker to
correct leader view by `LoadSegment`

This PR remove the segment loaded check, to fix that leader checker
cann't submit load task

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-17 19:12:54 +08:00
zhenshan.cao
31740445fa
fix: duplicate dynamic field data by mistake (#30042)
issue : #30000

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-01-17 19:02:56 +08:00
congqixia
80465a01b5
enhance: change some important request log level to Info (#30062)
Some important request log level shall be at least Info level

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-17 17:48:53 +08:00
yah01
1185e4dcd5
fix: written file size is over the int32 range and raises error (#30057)
we sum the total data size in int32, which could lead to an overflow
error
related #30056

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-17 16:42:54 +08:00
yah01
1c8ce33eea
fix: report error if the altering index doesn't support mmap (#29832)
this also checks the param value
fix #29909

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2024-01-17 16:40:54 +08:00
sammy.huang
e7dc82d8cc
enhance: [skip e2e]to workaround the error "HOME not set" (#30047)
issue: #30046

to workaround the error "HOME not set"  happened in self hosted runner

solution from
https://alexanderhose.com/aws-codecommit-with-instance-role-and-user-data/

Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-01-17 16:12:54 +08:00
sammy.huang
1152838a54
enhance: [skip e2e] when issue is required and when is not (#30063)
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-01-17 15:16:55 +08:00
congqixia
10acdbbe8e
enhance: free CString in InitTraceConfig (#30055)
`C.CString` result needs to be freed after usage

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-17 15:15:03 +08:00
yah01
9a3837212c
enhance: add index after load succeeded (#30015)
this avoids a corner case: after load index failed, this index can be
never loaded as it has been added into the segment's index map

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-17 15:06:53 +08:00
yah01
8f083e45cb
enhance: enable converting segcore error to merr (#29914)
this converts the segcore error to merr if possible

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-17 11:28:53 +08:00
wei liu
9abc868d15
fix: Remove heartbeat lag logic during get shard leaders (#29999)
issue: #29677 #29838
during get shard leaders, if qeurynode doesn't ack the heartbeat than
10s, querycoord will treat it as unavailable, and won't return shard
leader on it. but when querynode has a full cpu usage, it's easily to
stuck for more than 10s without ack the heartbeat, which cause no shard
leader to search/query.

This PR remove heartbeat lag logic during get shard leaders

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-17 11:22:52 +08:00
XuanYang-cn
ad7a0b4091
fix: Change finish log level to info (#30031)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-17 10:12:55 +08:00
XuanYang-cn
7f059b1025
fix: record apply pk latency metric to ms (#29987)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-17 10:11:03 +08:00
congqixia
c0f0548702
fix: use SafeChan preventing close channel multiple times (#30022)
See also #29935

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-16 17:34:54 +08:00
congqixia
7cb6bebd96
enhance: replace magic number with ParamItem for dist handler (#30020)
See also #28817

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-16 17:33:03 +08:00
MrPresent-Han
2a0eb1d2e6
feat: support general capacity restrict for cloud-side resoure contro… (#29845)
related: #29844

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-01-16 16:32:53 +08:00
sammy.huang
750166fd97
enhance: [skip e2e] failed to run all-contributors workflow (#29974)
issue: #30025

Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-01-16 16:24:52 +08:00
yah01
c68c128e47
fix: level 0 segments not loaded (#29908)
the recent changes move the level 0 segments list to a new proto field,
which leads to the QueryCoord can't see the level 0 segments, handle the
new changes
fix #29907

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-16 14:40:53 +08:00
congqixia
2dacca7b84
enhance: [skip e2e] use checkout action to get ".git" for codecov (#30014)
Ignoring .git may result "unusable report" in codecov. Use checkout
action to get .git information in upload codecov step

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-16 13:56:53 +08:00
yah01
cda44c09f9
enhance: adjust the GPU pool size (#29937)
according to benchmark, the GPU pool size with 6 performs best

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-16 11:54:52 +08:00
Bingyi Sun
8030b90891
fix: correct file name when loading index (#29985)
issue: #29973

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-01-16 10:24:52 +08:00
MrPresent-Han
c31e68446e
enhance: refine groupby-performance (#29933)
related: #29844

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-01-15 14:12:52 +08:00
XuanYang-cn
f9508a4851
fix: Remove logging data when logging skip msg (#29707)
See also: #29696

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-15 11:40:51 +08:00
wei liu
fb4fbcf48c
fix: Dynamic update rate limit config with wrong value (#29901)
when apply dynamic config changes, we should format the value to proper
unit
This PR fix update rate limit config with wrong value.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-15 09:52:51 +08:00
yihao.dai
8febbc79bb
fix: Fix unstable msg dispatcher ut (#29925)
This fix will not augment the execution time of unit tests, but solely
enhances tolerance for waiting for failure.

issue: https://github.com/milvus-io/milvus/issues/29921

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-14 10:40:52 +08:00
smellthemoon
073de855b3
enhance:remove some unused code (#29827)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-14 10:38:51 +08:00
chyezh
df5445f18d
fix: unhealthy datacoord started with unhealthy channel manager (#29848)
issue: #29818

Signed-off-by: chyezh <ye.zhen@zilliz.com>
2024-01-14 10:36:52 +08:00
chyezh
def717af55
fix: SealedIndexingEntry in SealedIndexingRecord may leak without smart pointer protect. (#29932)
may related issue: #29828

Signed-off-by: chyezh <ye.zhen@zilliz.com>
2024-01-14 10:28:51 +08:00
SimFG
d9edd50f97
fix: the delete msg disorder issue (#29915)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-01-14 10:26:52 +08:00
jaime
ee26561b5d
fix: remove checking if running inside container (#29940)
issue: https://github.com/milvus-io/milvus/issues/29846

Since checking if Milvus running inside the container is difficult,
which is decided by the system version and cgroup version, we try to get
the memory limit of the container each time instead of checking whether
inside the container first.

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-01-14 10:24:52 +08:00
chyezh
d300bc7bcb
fix: querynode num entity metric is broken by illegal label (#29948)
issue: #29766
also see pr: #29825

Signed-off-by: chyezh <ye.zhen@zilliz.com>
2024-01-14 10:23:00 +08:00
smellthemoon
595ec2559c
enhance: change some frequent log level (#29953)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-14 10:19:16 +08:00