Commit Graph

18559 Commits

Author SHA1 Message Date
congqixia
e79c3eaa90
enhance: [skip e2e] Update install_deps.sh cmake version (#28946)
cmake version needs to be greater or equal than 3.26.4, sync
`install_deps.sh` scripts version to 3.26.5

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-04 18:26:34 +08:00
aoiasd
b4af6f8c40
fix: sync action load segment with lack collection index info list (#28788)
relate: https://github.com/milvus-io/milvus/issues/28779
https://github.com/milvus-io/milvus/issues/28637

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-04 18:14:34 +08:00
Bingyi Sun
45e6801ce4
feat: Add checker activation service interfaces (#28850)
issue: #28610

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-12-04 17:38:37 +08:00
sre-ci-robot
9b6cbe956a
[automated] Update Knowhere Commit (#28917)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-12-04 15:42:34 +08:00
yihao.dai
9723e16058
enhance: Update pymilvus (#28922)
issue: https://github.com/milvus-io/milvus/issues/17614

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-12-04 15:36:35 +08:00
congqixia
c8b1a4618a
enhance: Resolve libunwind requirement conflict using 1.7.2 (#28929)
Try to resolve libunwind dependency requirement conflict between glog &
folly

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-04 14:06:40 +08:00
nico
cbd3b3c836
test: modify test cases for error msg update (#28846)
#23075

Signed-off-by: nico <cheng.yuan@zilliz.com>
2023-12-04 11:22:37 +08:00
nico
8740adbe28
test: update pymilvus version (#28832)
#17614

Signed-off-by: nico <cheng.yuan@zilliz.com>
2023-12-04 11:20:47 +08:00
wei liu
d081fd5481
enhance: Change some frequency log to rated level (#28897)
This pr change some frequency log's level to rated.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-04 10:38:35 +08:00
Jiquan Long
fccec125b2
enhance: [skip-e2e] add rust to builder image (#28838)
https://github.com/milvus-io/milvus/issues/27704

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-12-04 10:10:35 +08:00
wei liu
6718986a5b
enhance: Remove too frequency read ddl log from rootcoord (#28891)
This PR removed too frequency log for such
`DescribeCollection/ShowPartition` operation from root coord

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-04 09:42:33 +08:00
PowderLi
20fc90c591
enhance: find collection schema from cache (#28782)
issue: #28781 #28329

1. There is no need to call `DescribeCollection`, if the collection's
schema is found in the globalMetaCache
2. did `GetProperties` to check the access to Azure Blob Service while
construct the ChunkManager

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-12-03 19:22:33 +08:00
yah01
342635ed61
enhance: enable assert method to format arguments (#28812)
for now the assert method in segcore could accept a string information,
too many codes don't print the value they assert.

make it happy
related #28811

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-12-01 18:04:33 +08:00
aoiasd
24c565e37b
fix: accesslog method status not return failed when error in response (#28824)
relate: https://github.com/milvus-io/milvus/issues/28086

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-12-01 15:30:38 +08:00
sammy.huang
1174a7527a
enhance: [skip e2e]delete unsed arch for gpu builder (#28887)
1. disable arm64 arch 
2. only use docker cli instead of docker buildx cli

---------

Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2023-12-01 14:48:32 +08:00
yihao.dai
f5856812a2
fix: Fix get binary vector from chunk cache (#28866)
The way of getting binary vector size is wrong. This PR will fix it.

issue: https://github.com/milvus-io/milvus/issues/28865

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-12-01 14:40:32 +08:00
Gao
7206795e91
fix: update folly to resolve simd issue (#28878)
related #27552 , after this, milvus could run successfully on sse4.2
only machine

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-12-01 13:50:32 +08:00
congqixia
038eebba4d
enhance: Move datacoord broker into separate package (#28876)
See also #28861

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-12-01 10:22:34 +08:00
Bingyi Sun
8036ee13fa
feat: avoid dereferencing nullptr (#28862)
issue: #28793

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-12-01 10:20:32 +08:00
wei liu
043ac87be0
fix: Balance channel may cause channel not availble error (#28829)
issue: #28831
release old delegator before new delegator update it's distribution may
cause `channel not availble` error

This PR will block release old delgator before new delegator finish
`syncDistribution`

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2023-12-01 10:08:34 +08:00
sre-ci-robot
af98d1cb64
[automated] Update Builder image changes (#28874)
Update Builder image changes
See changes:
cac802ef7f
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-12-01 10:04:32 +08:00
sre-ci-robot
ecc3ca374c
[automated] Update Knowhere Commit (#28882)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-12-01 02:28:31 +08:00
XuanYang-cn
e62edb991a
enhance: Add FlowgraphManager interface (#28852)
- Change flowgraphManager to fgManagerImpl
- Change close to stop
- change execute to controlMemWaterLevel
- Change method name of fgManager for readability
- Add mockery for fgmanager

Issue: #28853

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-30 18:42:32 +08:00
yah01
bf633bb5d7
enhance: refine the retry error (#28573)
return the last error but not combining all errors, to improve
readability and erorr handling

resolve: #28572

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-30 18:34:32 +08:00
yihao.dai
b4353ca4ce
enhance: Remove vector chunk manager (#28569)
We have implemented the chunkcache (in cpp) to retrieve vectors, hence
rendering the vectorchunkcache (in golang) obsolete.

issue: https://github.com/milvus-io/milvus/issues/28568

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-11-30 18:00:33 +08:00
smellthemoon
adce6aab4f
fix:handle err when ManualCompaction (#28804)
handle err when ManualCompaction
#28644

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2023-11-30 17:06:31 +08:00
XuanYang-cn
5d0a9f9344
fix: Forget to set EntriesNum for deltalogs (#28858)
See also: #28520

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-30 16:52:31 +08:00
PowderLi
cac802ef7f
enhance: use already installed vcpkg (#28703)
issue #28686
1. Update Builder gpu image changes, see changes #28505
2. update azure-identity-cpp from beta to release

Signed-off-by: PowderLi <min.li@zilliz.com>
2023-11-30 15:58:32 +08:00
XuanYang-cn
aae7e62729
feat: Add levelzero compaction in DN (#28470)
See also: #27606

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-30 14:30:28 +08:00
yah01
d69440524b
fix: bypass growing index if no index meta (#28791)
we shouldn't panic if no index meta, just skip building it
fix #28022

Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-11-30 14:10:27 +08:00
Sammy Huang
43680043e3 auto close Update Knowhere Commit PR as long as ci-passed
Assign the 'lgtm' and 'approved' labels following the successful testing of the 'Update Knowhere Commit'
then prow/tide bot will auto merge the PR to master branch.

Signed-off-by: Sammy Huang <sammy.huang@zilliz.com>
2023-11-30 12:03:55 +08:00
shaoyue
8b2b0d412c
enhance: storeageType default value change to remote (#28792)
/kind enhancement
/cc @PowderLi

Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2023-11-30 11:34:27 +08:00
wayblink
b592f08db3
enhance: Use smaller lock in datacoord.meta.UpdateChannelCheckpoint (#28753)
/kind enhancement

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-11-30 11:00:28 +08:00
yihao.dai
4b8bc2798e
enhance: Print nq (#28507)
Log nq in search path.

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-11-30 10:42:27 +08:00
congqixia
2cd8daaf0b
fix: compacted segment still buffers delta data (#28816)
Related to #28628
Compacted segment syncing counter is not set correctly in sync task and
the bf write buffer shall not use compacted segment as candidate when
buffering delta data

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-30 10:20:28 +08:00
congqixia
f9bb8e9648
enhance: Change const magic number in querycoord to param (#28819)
See also #28817

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-30 09:06:28 +08:00
congqixia
1dc086496f
fix: schema->size() check logic with system field (#28802)
Now segcore load system field info as well, the growing segment
assertion shall not pass with "+ 2" value
This will cause all growing segments load failure
Fix #28801
Related to #28478
See also #28524

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-29 22:40:28 +08:00
cai.zhang
f5f4f0872e
enhance: Support importing data with parquet file (#28608)
issue: #28272

Numpy does not support array type import. 
Array type data is imported through parquet.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-29 20:52:27 +08:00
sre-ci-robot
381a45012a Update all contributors
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2023-11-29 12:25:57 +00:00
congqixia
3f9da2b8a0
fix: Check whether compaction plan exist before update (#28826)
Fix: #28825

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-29 18:54:27 +08:00
cqy123456
3b1b14dd78
fix: update binlog index memory uasge before loading segments (#28528)
issue: #27678 
when interimIndex = true, memory predict should be update with the
memory usage of binlog index build process.

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2023-11-29 16:42:27 +08:00
sre-ci-robot
d10c5af7aa
[automated] Update Pytest image changes (#28751)
Update Pytest image changes
See changes:
8fe2fb3eb9
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2023-11-29 12:56:27 +08:00
yah01
c0f6eccb7a
fix: No LevelZero segment in target (#28803)
the incorrect filter causes all LevelZero segment filtered, so the
deleted entities may be still visible
related: #27349

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-29 11:48:27 +08:00
XuanYang-cn
321c5c32e3
fix: Separate schedule and check results loop (#28692)
This PR:

- Separates compaction scheduler and check results loop So that slow in
check-loop doesn't influence execution.

- Cleans compaction tasks when drop a vchannel so dropped-channel's
compaction tasks won't be checked over and over again.

  - Skips meta change when meta's already changed, avoid panic
  - Remove not inuse injectDone(bool) parameter

See also: #28628, #28209

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-29 10:50:29 +08:00
nico
845851ea1c
test: add test cases (#28541)
1. modify test cases about output a non-existed field. issue: #28465
2. add cases about creating collection using different language
3. add test case about inserting a string value to a pk(int64) field
4. add test case about search with expression using double quotes.
issue: #28365

Signed-off-by: nico <cheng.yuan@zilliz.com>
2023-11-29 10:48:26 +08:00
nico
92b48c5384
test: add range search iterator back (#28659)
Signed-off-by: nico <cheng.yuan@zilliz.com>
2023-11-29 10:30:28 +08:00
nico
4d53980456
test: modify test cases (#28800)
1. Drop a non-existing partition will not raise error now. #28722
2. Search will not raise error now when some row has less elements than
expression index. #28293

Signed-off-by: nico <cheng.yuan@zilliz.com>
2023-11-29 10:26:26 +08:00
Gao
5bd73d5503
enhance: add efficient distance computations in Go (#28657)
Related #28656
Add more efficient calc_distance at Go side.

---------

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2023-11-28 18:20:26 +08:00
XuanYang-cn
606ec77b66
enhance: Unify levelzero segment config in DN (#28720)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-11-28 18:04:26 +08:00
congqixia
881a166b5a
fix: data race on rootcoord.proxyClientManager.proxyClient (#28787)
Fix #28786

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-11-28 15:52:26 +08:00