Commit Graph

18926 Commits

Author SHA1 Message Date
congqixia
c0f0548702
fix: use SafeChan preventing close channel multiple times (#30022)
See also #29935

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-16 17:34:54 +08:00
congqixia
7cb6bebd96
enhance: replace magic number with ParamItem for dist handler (#30020)
See also #28817

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-16 17:33:03 +08:00
MrPresent-Han
2a0eb1d2e6
feat: support general capacity restrict for cloud-side resoure contro… (#29845)
related: #29844

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-01-16 16:32:53 +08:00
sammy.huang
750166fd97
enhance: [skip e2e] failed to run all-contributors workflow (#29974)
issue: #30025

Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-01-16 16:24:52 +08:00
yah01
c68c128e47
fix: level 0 segments not loaded (#29908)
the recent changes move the level 0 segments list to a new proto field,
which leads to the QueryCoord can't see the level 0 segments, handle the
new changes
fix #29907

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-16 14:40:53 +08:00
congqixia
2dacca7b84
enhance: [skip e2e] use checkout action to get ".git" for codecov (#30014)
Ignoring .git may result "unusable report" in codecov. Use checkout
action to get .git information in upload codecov step

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-16 13:56:53 +08:00
yah01
cda44c09f9
enhance: adjust the GPU pool size (#29937)
according to benchmark, the GPU pool size with 6 performs best

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-16 11:54:52 +08:00
Bingyi Sun
8030b90891
fix: correct file name when loading index (#29985)
issue: #29973

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-01-16 10:24:52 +08:00
MrPresent-Han
c31e68446e
enhance: refine groupby-performance (#29933)
related: #29844

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-01-15 14:12:52 +08:00
XuanYang-cn
f9508a4851
fix: Remove logging data when logging skip msg (#29707)
See also: #29696

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-01-15 11:40:51 +08:00
wei liu
fb4fbcf48c
fix: Dynamic update rate limit config with wrong value (#29901)
when apply dynamic config changes, we should format the value to proper
unit
This PR fix update rate limit config with wrong value.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-15 09:52:51 +08:00
yihao.dai
8febbc79bb
fix: Fix unstable msg dispatcher ut (#29925)
This fix will not augment the execution time of unit tests, but solely
enhances tolerance for waiting for failure.

issue: https://github.com/milvus-io/milvus/issues/29921

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-14 10:40:52 +08:00
smellthemoon
073de855b3
enhance:remove some unused code (#29827)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-14 10:38:51 +08:00
chyezh
df5445f18d
fix: unhealthy datacoord started with unhealthy channel manager (#29848)
issue: #29818

Signed-off-by: chyezh <ye.zhen@zilliz.com>
2024-01-14 10:36:52 +08:00
chyezh
def717af55
fix: SealedIndexingEntry in SealedIndexingRecord may leak without smart pointer protect. (#29932)
may related issue: #29828

Signed-off-by: chyezh <ye.zhen@zilliz.com>
2024-01-14 10:28:51 +08:00
SimFG
d9edd50f97
fix: the delete msg disorder issue (#29915)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-01-14 10:26:52 +08:00
jaime
ee26561b5d
fix: remove checking if running inside container (#29940)
issue: https://github.com/milvus-io/milvus/issues/29846

Since checking if Milvus running inside the container is difficult,
which is decided by the system version and cgroup version, we try to get
the memory limit of the container each time instead of checking whether
inside the container first.

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-01-14 10:24:52 +08:00
chyezh
d300bc7bcb
fix: querynode num entity metric is broken by illegal label (#29948)
issue: #29766
also see pr: #29825

Signed-off-by: chyezh <ye.zhen@zilliz.com>
2024-01-14 10:23:00 +08:00
smellthemoon
595ec2559c
enhance: change some frequent log level (#29953)
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-14 10:19:16 +08:00
congqixia
082ee1a709
enhance: Use newer checkpoint when packing LoadSegmentRequest (#29922)
See also: #29650

Either segment dml position & channel checkpoint could be newer in some
cases. This PR make PackLoadSegments use the newer one improving load
performance during cases where there are lots of upsert.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-13 10:46:53 +08:00
congqixia
ed89c6a2ee
enhance: make compactor use actual buffer size to decide when to sync (#29945)
See also: #29657

Datanode Compactor use estimated row number from schema to decide when
to sync the batch of data when executing compaction. This est value
could go way from actual size when the schema contains variable field(
say VarChar, JSON, etc.)

This PR make compactor able to check the actual buffer data size and
make it possible to sync when buffer is actually beyong max binglog
size.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-13 01:32:52 +08:00
yanliang567
c1b0562d21
enhance: Add a group by case (#29939)
Related issue: #29883 
xfail for now.

Signed-off-by: yanliang567 <yanliang.qiao@zilliz.com>
2024-01-13 01:06:51 +08:00
cai.zhang
8c89ad694e
fix: Fix error message for indexing (#29898)
issue: #29897

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-01-12 19:44:51 +08:00
wei liu
565fc3a019
enhance: Skip generate load segment task (#29724)
issue: #29814
if channel is not subscribed yet, the generated load segment task will
be remove from task scheduler due to the load segment task need to be
transfer to worker node by shard leader.

This PR skip generate load segment task when channel is not subscribed
yet.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-12 18:56:58 +08:00
Bingyi Sun
e1258b8cad
feat: integrate storagev2 into loading segment (#29336)
issue: #29335

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-01-12 18:10:51 +08:00
yah01
f2e36db488
enhance: optimize the loading index performance (#29894)
this utilizes concurrent loading

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-12 17:44:51 +08:00
zhenshan.cao
a442194abc
feat: Add RBAC functionality to alias (#29885)
issue: https://github.com/milvus-io/milvus/issues/29781
issue: https://github.com/milvus-io/milvus-proto/issues/237

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-01-12 15:26:50 +08:00
sammy.huang
d3dd2a2032
enhance: correct nightly e2e (#29919)
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-01-12 14:42:52 +08:00
congqixia
10622698df
enhance: reduce delete detail log to delete range (#29916)
Delete detail log will be large and hard to read when log level is
debug. This PR change the log to stringer and print only pk range,
number.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-12 14:40:59 +08:00
yah01
6c477ce3a7
enhance: optimize the loading strategy (#29910)
as we have the pool size limit so we don't need to limit the concurrency
manually

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-12 14:26:50 +08:00
yah01
aba2656e68
fix: missing field data after appending scalar index to loaded segment (#29912)
related #29843

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-12 14:04:54 +08:00
congqixia
adf0c8885c
enhance: add trace span for wait tsafe (#29911)
Add tracing span for search/query operation waiting tsafe duration

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-01-12 14:03:04 +08:00
sre-ci-robot
4d11525f55
[automated] Update Knowhere Commit (#29904)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-01-12 14:00:50 +08:00
sammy.huang
61acda4948
enhance: [skip e2e]debug why no configuration file when docker-compose up (#29890)
#29889

Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-01-12 10:38:51 +08:00
jaime
90984a2cab
enhance: Support read hardware metrics for cgroupv2 (#29850)
issue: https://github.com/milvus-io/milvus/issues/29846

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-01-12 10:30:51 +08:00
sre-ci-robot
0d395913b8
[automated] Update Pytest image changes (#29895)
Update Pytest image changes
See changes:
930735c46c
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-01-12 09:46:58 +08:00
elstic
315d49261a
test: remove xfail from case (#29853)
pr: https://github.com/milvus-io/milvus/pull/29634 

After verifying that the problem is fixed, it is necessary to run this
case every night for regression.

Signed-off-by: elstic <hao.wang@zilliz.com>
2024-01-12 09:45:05 +08:00
nico
930735c46c
enhance: update pymilvus version and test cases (#29882)
Signed-off-by: nico <cheng.yuan@zilliz.com>
2024-01-11 19:20:57 +08:00
wayblink
1df3f90696
feat: Implement DescribeAlias and ListAliases interfaces (#29641)
#22882
/kind feature

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-01-11 19:12:51 +08:00
sammy.huang
0b3b753ef6
enhance: clean .github/workflow/main.yaml (#29524)
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-01-11 18:20:52 +08:00
sammy.huang
27992e12dc
enhance: add standalone mode for nightly ci (#29884)
Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2024-01-11 18:06:50 +08:00
wei liu
797847904c
enhance: Change some frequency log to rated level (#29720)
This PR change some frequency log to rated level

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-11 16:30:50 +08:00
zhagnlu
673f980216
fix: Add multi logical exprs case (#29871)
https://github.com/milvus-io/milvus/issues/29759

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-01-11 16:22:49 +08:00
Xu Tong
e429965f32
Add float16 approve for multi-type part (#28427)
issue:https://github.com/milvus-io/milvus/issues/22837

Add bfloat16 vector, add the index part of float16 vector.

Signed-off-by: Writer-X <1256866856@qq.com>
2024-01-11 15:48:51 +08:00
yah01
26e900180e
fix: the insert count is zero after set the pointer to nil (#29870)
this leads to the EntitiesNum metric would be never reduced

fix: #29766

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-11 14:56:50 +08:00
yihao.dai
a4f15f4f43
enhance: Simplify cross cluster routing integration test (#29873)
This pull request simplifies the integration test for cross-cluster
routing by reusing `integration.MiniClusterSuite`, instead of defining
custom Milvus clients, servers, and etcd client.

issue: https://github.com/milvus-io/milvus/issues/29874

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-01-11 14:00:52 +08:00
wei liu
1f759837c4
fix: remove Unnecessary lock in config manager (#29836)
issue: #29709 #291712
to avoid concurrent recursive RLock and Lock cause deadlock, This PR
remove the unnecessary lock in config manager

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-11 13:48:49 +08:00
wei liu
9fc5f1176c
fix: Drop segment meta info with prefix (#29856)
If segment has more than 128 log fils, drop segment will exceed etcd txn
ops limit, which will failed the drop segment request
This PR drop segment meta info with prefix, to avoid drop segment meta
failed

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-01-11 12:22:51 +08:00
yah01
44fe06f198
enhance: skip loading duplicated index (#29715)
this protect the loading index from failure, and speed up the loading
progress

Signed-off-by: yah01 <yang.cen@zilliz.com>
2024-01-11 11:52:49 +08:00
Jiquan Long
67ab5be15a
enhance: optimize search performance of inverted index (#29794)
issue: #29793 
Use `DocSetCollector` instead of `TopDocsCollector`, which will avoid
scoring and sorting.

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-01-11 11:12:49 +08:00