test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-02 20:09:57 +08:00
Code Issues Actions 12 Packages Projects Releases Wiki Activity
19,546 Commits 94 Branches 105 Tags 184 MiB
c012e6786f
Commit Graph

443 Commits

Author SHA1 Message Date
Bingyi Sun
406bf14e84
enhance: Add growing row count weight (#30271) 
Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-01-29 14:05:02 +08:00
xige-16
033eae9e73
enhance: Set segment.maxSize param to 1024M (#30139) 
issue: #25639 
/kind improvement

When the number of vector columns increases, the number of rows per
segment will decrease. In order to reduce the impact on vector indexing
performance, it is necessary to increase the segment max limit.

If a collection has multiple vector fields with memory and disk indices
on different vector fields, the size limit after segment compaction is
the minimum of segment.maxSize and segment.diskSegmentMaxSize.

Signed-off-by: xige-16 <xi.ge@zilliz.com>

---------

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2024-01-29 10:17:02 +08:00
congqixia
7ced0af197
enhance: Enlarge default datanode sync parallel to 256 (#30270) 
See also #27675

After supporting control sync parallel in datanode globally, the shall
change default value to a more suitable value for most use cases.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-26 11:35:00 +08:00
yihao.dai
c02fb64ad6
enhance: Allows proactive warming up of chunk cache (#30182) 
Allows proactive warming up of chunk cache. Original vector data will be
asynchronously loaded into the chunk cache during the load process. It
has the potential to significantly reduce query/search latency for a
certain duration after the load, albeit with a concurrent increase in
disk usage.

issue: https://github.com/milvus-io/milvus/issues/30181

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-01-25 19:55:39 +08:00
SimFG
aa7014a360
enhance: move the cgo code in the pkg dir to interal dir (#30261) 
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-01-25 15:15:01 +08:00
yah01
51fe4743f1
enhance: hint the invalid metric type in error message (#30263) 
this hints the user passed invalid metric type in error message, so
users could know what's wrong

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-25 14:15:00 +08:00
PowderLi
08ca0a2ca5
feat: support etcd authentication (#30226) 
issue: #28895
add 3 configuration for ETCD config

Signed-off-by: PowderLi <min.li@zilliz.com>
 2024-01-24 11:35:00 +08:00
Patrick Weizhi Xu
0907d76253
enhance: pass partition key scalar info if enabled when build vector index (#29931) 
issue: #29892 

Pass optional scalar IVF offsets to Cardinal

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-01-24 00:04:55 +08:00
congqixia
c9860e3d2a
fix: Init paramtable after milvus role setup (#30211) 
See also #30176

Move paramtable.Init after env setup in roles.Run. Also introduced a
flag for mixture run to set role correctly for mixture mode.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-23 19:10:56 +08:00
SimFG
463765922e
enhance: support related privilege for grant api (#30153) 
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-01-23 15:56:54 +08:00
SimFG
ddccccbcab
enhance: add the bytes data type for merge data and format some code (#30105) 
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-01-18 22:18:55 +08:00
yah01
1c8ce33eea
fix: report error if the altering index doesn't support mmap (#29832) 
this also checks the param value
fix #29909

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2024-01-17 16:40:54 +08:00
yah01
8f083e45cb
enhance: enable converting segcore error to merr (#29914) 
this converts the segcore error to merr if possible

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-17 11:28:53 +08:00
congqixia
7cb6bebd96
enhance: replace magic number with ParamItem for dist handler (#30020) 
See also #28817

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-16 17:33:03 +08:00
MrPresent-Han
2a0eb1d2e6
feat: support general capacity restrict for cloud-side resoure contro… (#29845) 
related: #29844

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-01-16 16:32:53 +08:00
yah01
cda44c09f9
enhance: adjust the GPU pool size (#29937) 
according to benchmark, the GPU pool size with 6 performs best

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-16 11:54:52 +08:00
jaime
ee26561b5d
fix: remove checking if running inside container (#29940) 
issue: https://github.com/milvus-io/milvus/issues/29846

Since checking if Milvus running inside the container is difficult,
which is decided by the system version and cgroup version, we try to get
the memory limit of the container each time instead of checking whether
inside the container first.

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-01-14 10:24:52 +08:00
Bingyi Sun
e1258b8cad
feat: integrate storagev2 into loading segment (#29336) 
issue: #29335

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2024-01-12 18:10:51 +08:00
zhenshan.cao
a442194abc
feat: Add RBAC functionality to alias (#29885) 
issue: https://github.com/milvus-io/milvus/issues/29781
issue: https://github.com/milvus-io/milvus-proto/issues/237

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-01-12 15:26:50 +08:00
jaime
90984a2cab
enhance: Support read hardware metrics for cgroupv2 (#29850) 
issue: https://github.com/milvus-io/milvus/issues/29846

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-01-12 10:30:51 +08:00
wayblink
1df3f90696
feat: Implement DescribeAlias and ListAliases interfaces (#29641) 
#22882
/kind feature

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-01-11 19:12:51 +08:00
Xu Tong
e429965f32
Add float16 approve for multi-type part (#28427) 
issue：https://github.com/milvus-io/milvus/issues/22837

Add bfloat16 vector, add the index part of float16 vector.

Signed-off-by: Writer-X <1256866856@qq.com>
 2024-01-11 15:48:51 +08:00
wei liu
1f759837c4
fix: remove Unnecessary lock in config manager (#29836) 
issue: #29709 #291712
to avoid concurrent recursive RLock and Lock cause deadlock, This PR
remove the unnecessary lock in config manager

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-01-11 13:48:49 +08:00
congqixia
8a6e1a4b27
enhance: pre-allocate result FieldData space to reduce copy & growslice (#29726) 
See also: #29113

Add a new utitliy function in `pkg/util/typetuil` to pre-allocate field
data slice capacity acoording to search limit. This shall avoid copying
the data during `AppendFieldData` when previous slice is out of space.
And shall also save CPU time during high paylog.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-09 15:48:55 +08:00
xige-16
9702cef2b5
feat: Support multiple vector search (#29433) 
issue #25639 

Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2024-01-08 15:34:48 +08:00
wayblink
635a7f777c
feat: add clustering key in create/describe collection (#29506) 
#28410
/kind feature

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-01-07 19:56:48 +08:00
yihao.dai
23183ffb0f
feat: Add import reader for json (#29252) 
This PR implements a new json reader for import.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-01-05 18:12:48 +08:00
smellthemoon
1c1f2a1371
enhance:change some logs (#29579) 
related #29588

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-01-05 16:12:48 +08:00
wei liu
07057fcf7c
fix: Unexpected rpc msg size limit (#29682) 
due to `clientMaxSendSize` and `serverMaxRecvSize` will limit the rpc
request size limit, they should use same config value, and
`serverMaxSendSize` and `clientMaxRecvSize` will limit the rpc response
size limit, they should use same config value too.

This PR fix unexpected rpc msg limit which caused by the wrong usage of
misunderstanding rpc config items

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-01-05 15:56:47 +08:00
MrPresent-Han
9e2e7157e9
feat: support search_group_by for milvus(#25324) (#28983) 
related: #25324

Search GroupBy function, used to aggregate result entities based on a
specific scalar column.
several points to mention:

1. Temporarliy, the whole groupby is implemented separated from
iterative expr framework **for the first period**
2. In the long term, the groupBy operation will be incorporated into the
iterative expr framework:https://github.com/milvus-io/milvus/pull/28166
3. This pr includes some unrelated mocked interface regarding alterIndex
due to some unworth-to-mention reasons. All these un-associated content
will be removed before the final pr is merged. This version of pr is
only for review
4. All other related details were commented in the files comparison

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-01-05 15:50:47 +08:00
cqy123456
22bb84fa9d
feat:add new gpu index:GPU_BRUTE_FORCE and limit gpu index metric type (#29590) 
issue: https://github.com/milvus-io/milvus/issues/29230
this pr do these things:
1. add gpu brute force;
2. limit gpu index only support l2 / ip;

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-01-05 15:24:48 +08:00
yihao.dai
3561586edf
feat: Add import reader for binlog (#28910) 
This PR defines the new import reader interfaces and implement a binlog
reader for import.

issue: https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-01-05 11:48:47 +08:00
yah01
9e0163e12f
enhance: use GPU pool for gpu tasks (#29678) 
- this much improve the performance for GPU index

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2024-01-04 17:50:46 +08:00
congqixia
da7c3cbd88
enhance: make delegator delete buffer holding all delete from cp (#29626) 
See also #29625

This PR:
- Add a new implemention of `DeleteBuffer`: listDeleteBuffer
  - holds cacheBlock slice
  - `Put` method append new delete data into last block
  - when a block is full, append a new block into the list
- Add `TryDiscard` method for `DeleteBuffer` interface
  - For doubleCacheBuffer, do nothing
- For listDeleteBuffer, try to evict "old" blocks, which are blocks
before the first block whose start ts is behind provided ts
- Add checkpoint field for `UpdateVersion` sync action, which shall be
used to discard old cache delete block

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-01-04 17:02:46 +08:00
SimFG
d23f87a393
enhance: Add concurrency for datacoord segment GC (#29561) 
issue: https://github.com/milvus-io/milvus/issues/29553
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-01-03 13:16:57 +08:00
MrPresent-Han
ed644983e2
enhance: add param for bloomfilter(#29388) (#29490) 
related: #29388

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-12-28 18:10:46 +08:00
xige-16
0a70e8b601
enhance: Remove multiple vector field limit (#27827) 
issue: https://github.com/milvus-io/milvus/issues/25639

/kind improvement
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-12-28 16:40:46 +08:00
Gao
8a4c0d4a3f
enhance: compile go assembly with build constraints for different arch (#29515) 
issue #28657 
follow https://pkg.go.dev/cmd/go#hdr-Build_constraints to compile go
assembly with different cpu arch

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-12-28 15:40:47 +08:00
Gao
8a630f733a
enhance: add new optimize param for queryhook (#29495) 
add a flag to indicate if we use search param optimizations, default is
on

---------

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-12-28 10:04:46 +08:00
wei liu
839a72129e
fix: Auto balance param can't be updated by dynamic (#29501) 
This PR fixed that auto balance param can't be updated by dynamic

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-12-27 14:30:53 +08:00
MrPresent-Han
7c7003bff6
enhance:refine the range of chunk size config value(#29388) (#29389) 
related: #29388

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-12-26 17:36:46 +08:00
cqy123456
4c979538a4
enhance: update cagra index params in config and add params check (#29045) 
issue:https://github.com/milvus-io/milvus/issues/29230
this pr do two things about cagra index:
 a.milvus yaml config support gpu memory settings

 b.add cagra-params check

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
Co-authored-by: yusheng.ma <yusheng.ma@zilliz.com>
 2023-12-26 11:04:47 +08:00
aoiasd
ac23e67f07
enhance: add default file path for access log (#29460) 
relate:https://github.com/milvus-io/milvus/issues/29459

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-12-26 10:38:45 +08:00
MrPresent-Han
bd3bde82f0
fix iterator lose data for duplicted result(#29406) (#29451) 
related: #29406

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-12-25 23:28:47 +08:00
PowderLi
13988cbde7
fix: Partition related privileges should belong to Collection (#29428) 
issue: #29427

it will try to find privilege_name from the object_type, while grant a
privilege to a role

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-12-24 20:32:45 +08:00
SimFG
dd9c61831d
enhance: Support to get the param value in the runtime (#29297) 
/kind improvement
issue: #29299

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-12-22 18:36:44 +08:00
yah01
a0e1a1eb31
feat: support enable/disable mmap for index (#29005) 
support enable/disable mmap for index, the user could alter the index's
mode by `AlterIndex` method
related: https://github.com/milvus-io/milvus/issues/21866

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-12-21 18:07:24 +08:00
wei liu
fcba1c0d9e
fix: Rename invalid parameterutil package path (#29334) 
This PR renames the invalid parameterutil package path

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-12-21 11:40:42 +08:00
yihao.dai
f457b9f7c9
fix: Return time tick delay error and refine quota error messages (#29289) 
This pr:
1. Handles the time tick delay error when converting old error codes to
milvus errors.
2. Enhances quota error messages by eliminating "force deny" and
substituting it with "quota exceeded."

issue: https://github.com/milvus-io/milvus/issues/29288

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-12-20 22:52:44 +08:00
aoiasd
46a937ac43
fix: msgstream memory leak caused by config event don't deregister (#29266) 
relate: https://github.com/milvus-io/milvus/issues/28620

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-12-20 20:32:44 +08:00
XuanYang-cn
7d32861d53
enhance: Add channel check interval params (#29311) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-12-20 15:06:42 +08:00
PowderLi
bcd6865b29
enhance: add 3 builtin roles (#28961) 
issue: #28960 [milvus-proto
#212](https://github.com/milvus-io/milvus-proto/issues/212)

add new configuration: builtinRoles
user can define roles in config file: `milvus.yaml`

there is an example:
1. db_ro, only have read privileges, include load
2. db_rw, read and write privileges, include create/drop/rename
collection
3. db_admin, not only read and write privileges, but also user
administration

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-12-18 14:28:41 +08:00
zhagnlu
a602171d06
enhance: Refactor runtime and expr framework (#28166) 
#28165

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-12-18 12:04:42 +08:00
wei liu
820ee692fc
enhance: Add config for querycoord auto balance channel (#29231) 
issue: #23726
This PR add control config to querycoord's background auto balance
channel operation

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-12-18 10:00:40 +08:00
congqixia
88b4b8b77c
enhance: make segments SQPool & LoadPool resizable (#29239) 
See also #29223

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-15 20:22:39 +08:00
congqixia
4731c1b0d5
enhance: make SyncManager pool size refreshable (#29224) 
See also #29223

This PR make `conc.Pool` resizable by adding `Resize` method for it. 
Also make newly added datanode `MaxParallelSyncMgrTasks` config
refreshable

---------

Signed-off-by: Congqi.Xia <congqi.xia@zilliz.com>
 2023-12-15 09:58:43 +08:00
congqixia
25a4525297
enhance: Change sync manager parallel config item (#29216) 
Since the sync manager is global in datanode now, the old
`maxParallelSyncTaskNum` does not fit into current implementation
anymore.

This PR add a new param item for sync mgr parallel control and enlarge
default value

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-14 20:46:41 +08:00
congqixia
9407be6eb2
enhance: Add a config item for partition name as regexp feature and disable it by default (#29154) 
See also #29177
Add a config item for partition name as regexp feature and disable it by
default

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-14 10:04:38 +08:00
aoiasd
b5ee563914
fix: accesslog can not print search expression (#28899) 
relate: https://github.com/milvus-io/milvus/issues/28893

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-12-13 18:50:42 +08:00
Jiquan Long
520c07ec9d
fix: autoindex panic with flat index type (#29071) 
issue: https://github.com/milvus-io/milvus/issues/29048

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-12-12 22:08:46 +08:00
Jiquan Long
e5f155612a
fix: txn control not applied to snapshot.Remove (#29124) 
issue: #29123

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
 2023-12-12 14:48:38 +08:00
wayblink
51f870da7e
feat: Introduce channelCheckpointUpdater to reduce goroutine use in ttNode (#28570) 
/kind enhancement

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-12-12 13:48:42 +08:00
wei liu
69a71e11d1
enhance: Add config for auto upgrade segment index (#29112) 
This PR add config for auto upgrade segment index to index engine's
version.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-12-12 11:56:37 +08:00
Enwei Jiao
0e65e90338
enhance: Support otlp with insecure (#29115) 
issue: https://github.com/milvus-io/milvus/issues/28914

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-12-12 11:14:37 +08:00
yah01
0a87724f18
enhance: remove merger for load segments (#29062) 
remove merger as now QueryNode could load segments concurrently
fix #29063

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-12-12 10:48:37 +08:00
cai.zhang
2b05460ef9
enhance: Make import-related error message clearer (#28978) 
issue: #28976

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-12-08 10:12:38 +08:00
wayblink
6736f65345
feat: skip some empty ttMsg in Datanode flowgraph (#28756) 
/kind feature

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-12-07 01:00:37 +08:00
shaoyue
4a067a4c8c
enhance: Add proxy.ginLogSkipPaths (#28945) 
Fix #28944

/cc @xiaofan-luan

Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2023-12-06 10:26:35 +08:00
groot
ad1daebc8e
fix: Change ranges of HNSW parameters according to Knowhere (#28875) 
fix: https://github.com/milvus-io/milvus/issues/28860

Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2023-12-05 16:20:42 +08:00
yah01
bf633bb5d7
enhance: refine the retry error (#28573) 
return the last error but not combining all errors, to improve
readability and erorr handling

resolve: #28572

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-30 18:34:32 +08:00
shaoyue
8b2b0d412c
enhance: storeageType default value change to remote (#28792) 
/kind enhancement
/cc @PowderLi

Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2023-11-30 11:34:27 +08:00
congqixia
f9bb8e9648
enhance: Change const magic number in querycoord to param (#28819) 
See also #28817

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-30 09:06:28 +08:00
cqy123456
3b1b14dd78
fix: update binlog index memory uasge before loading segments (#28528) 
issue: #27678 
when interimIndex = true, memory predict should be update with the
memory usage of binlog index build process.

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2023-11-29 16:42:27 +08:00
Gao
5bd73d5503
enhance: add efficient distance computations in Go (#28657) 
Related #28656
Add more efficient calc_distance at Go side.

---------

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-11-28 18:20:26 +08:00
XuanYang-cn
606ec77b66
enhance: Unify levelzero segment config in DN (#28720) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-28 18:04:26 +08:00
aoiasd
89d8ce2f73
enhance: refine access log to support format access log by yaml and print name info. (#28319) 
relate: https://github.com/milvus-io/milvus/issues/28086

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-11-28 15:32:31 +08:00
cai.zhang
1b7a503f89
enhance: Revert import support csv format (#28760) 
Revert import support csv format.
issue: #28778

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-11-28 14:32:27 +08:00
SimFG
9c46788d87
enhance: Support to trace restful request and request error (#28685) 
issue: #28348

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-11-27 20:14:26 +08:00
cai.zhang
c29b60e18e
enhance: Support Array DataType for bulk_insert (#28341) 
issue: #28272 
Support array DataType for bulk_insert with json, binlog files.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-11-27 13:50:27 +08:00
Bingyi Sun
2843664a20
fix: channel name can not contains '-partition-' (#28689) 
issue: https://github.com/milvus-io/milvus/issues/28675

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-11-24 10:14:24 +08:00
wei liu
4b8d3eaa41
enhance: Increase heartbeat timeout (#28648) 
it's easy to trigger heartbeat timeout after 100ms when standalone cpu
usage reach 100%.
This PR increase the heartbeat timeout param to 2000ms

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-23 19:50:24 +08:00
congqixia
a2fe9dad49
enhance: Make etcd kv request timeout configurable (#28661) 
See also #28660
This pr add request timeout config item for etcd kv request timeout
 Sync the default timeout value to same value for etcdKV & tikv config

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-23 19:34:23 +08:00
zhagnlu
bc43c5cf34
fix: Panic if minio port is empty (#28461) 
#28460

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-11-23 19:24:25 +08:00
Bingyi Sun
4fedff6d47
feat: integrate storage v2 into the write path (#28440) 
#28378

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
 2023-11-23 17:26:24 +08:00
yah01
c96d07682e
enhance: enhance the accuracy of memory usage (#28554) 
before this, Milvus use container/system's memory info to get the memory
usage, which could be inaccurate.

we allocates the memory by private anon mmap,
then `rss - shared` would be the accurate memory usage

resolve #28553

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-23 15:12:23 +08:00
yah01
bfccfcd0ca
enhance: refine error messages (#28424) 
- Split the simple reason and full detail
- Refine existing error messages
related: #28422

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-21 17:02:24 +08:00
XuanYang-cn
e88bbaac24
feat: Add universal levelzero segment switch (#28483) 
See also: #27349

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-20 19:04:22 +08:00
PowderLi
7029797452
Improve HTTP server include metrics and RESTful API (#28045) 
/kind improvement
issue: #27653

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-11-19 17:14:20 +08:00
yah01
e9ff7ed13d
enhance: remove legacy error related code (#28385) 
/kind improvement

Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-11-14 15:02:18 +08:00
SimFG
cfb6edea61
Support to trace the grpc request (#28349) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-11-13 20:16:18 +08:00
XuanYang-cn
f8aa46419a
Add LevelZeroCompaction configs (#28190) 
See also: #27606

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-13 11:18:19 +08:00
SimFG
7dda2e8814
Change some log level in the pkg package (#28181) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-11-08 23:34:22 +08:00
zhenshan.cao
2057fdce43
Change default storage type to minio (#28236) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-11-07 21:15:31 +08:00
wei liu
5b45a138b1
disable auto balance when old node exists (#28191) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-07 14:02:20 +08:00
Xiaofan
da19e49daf
Support purge old session for standalone (#28184) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-11-06 21:21:42 +08:00
yah01
90e2c63d9e
Fix getting incorrect CPU num (#28146) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-11-06 06:02:16 +08:00
congqixia
bf2f62c1e7
Add WriteBuffer to provide abstraction for delta policy (#27874) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-04 12:10:17 +08:00
SimFG
ef68680639
Fix the unstable resource manager testcase (#28033) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-11-03 11:00:17 +08:00
aoiasd
d2727cc0ff
Add write cache for access log (#27792) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-11-03 10:28:17 +08:00
wei liu
3b8fbbd91d
avoid sleep to ctx done during retry (#27992) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-11-01 18:52:24 +08:00
PowderLi
0c0f012e03
add a custom http header: Accept-Type-Allow-Int64 (#27901) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-11-01 11:42:16 +08:00
Enwei Jiao
8ae9c947ae
Use OpenDAL to access object store (#25642) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-11-01 09:00:14 +08:00
cqy123456
4fbe3c9142
replace loaded binlog with binlog index for search performance (#27673) 
Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2023-11-01 02:20:15 +08:00
yah01
9658367a3c
Refine chunk manager errors (#27590) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-31 12:18:15 +08:00
yah01
2af46d7333
Increase the ChunkManager request timeout (#28015) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-31 09:06:13 +08:00
MrPresent-Han
ccaec44930
enable parameter for dataCoord broker timeout(#26972) (#27825) 
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-10-28 06:14:11 +08:00
yihao.dai
f9c630247d
Construct plan directly when search with vector output (#27928) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-10-26 19:30:10 +08:00
congqixia
8ab01e5a14
Set ElementType for empty array FieldData (#27924) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-25 20:08:10 +08:00
Gao
c8415453a1
Support prepare config for index (#27920) 
Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-10-25 19:54:12 +08:00
aoiasd
53246b1b38
Set accesslog default to close and use stdout (#27891) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-10-25 10:30:10 +08:00
wei liu
40723a292e
reduce compact parallel task num (#27899) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-10-25 09:40:12 +08:00
jaime
4640928280
Fix initialization and backward compatibility issue for GRPC compression (#27894) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-10-24 22:14:14 +08:00
congqixia
93a877f55e
Make qcv2 target&leader observer execute in parallel (#27844) 
- Add `taskDispatcher` to submit and run task async safely
- Change `LeaderObeserver` and `TargetObserver` schedule and manual check action to submitting task into dispatcher
- Fix logic problem in collection observer when manual check return false

See also #27494

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-24 10:14:11 +08:00
Xiaofan
2ea7579dbb
Reduce rpc size for GetRecoveryInfoV2 (#27483) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-10-23 21:44:09 +08:00
zhagnlu
6060dd7ea8
Add chunk manager request timeout (#27692) 
Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2023-10-23 20:08:08 +08:00
congqixia
4a6790b63a
Make KeyLock generic (#27854) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-23 17:40:11 +08:00
SimFG
9b0ecbdca7
Support to replicate the mq message (#27240) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-10-20 14:26:09 +08:00
zhenshan.cao
020ad9a6bc
Rectify wrong exception messages associated with Array datatype (#27769) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-10-19 17:24:07 +08:00
smellthemoon
4b0ec156b3
Set channel work pool size in datanode (#27728) 
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2023-10-19 08:28:08 +08:00
yah01
9467de79c3
Use pool to block exceeded tasks (#27767) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-19 03:30:06 +08:00
yihao.dai
49b3a12804
Return newly defined merr instead of grpc unimplemented err (#27751) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-10-18 15:32:11 +08:00
congqixia
2f201c25e2
Remove deprecated io/ioutil usage (#27747) 
`io/ioutil` package is deprecated, use `io`,`os` package replacement
also added golangci-lint rule to block future reference

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: guoguangwu <guoguangwu@magic-shield.com>
 2023-10-17 20:32:09 +08:00
Futao Wei
599012a340
Implement SearchByPks path for Search (#25882) 
Signed-off-by: unfode <forrest.futao.wei@gmail.com>
 2023-10-16 15:34:08 +08:00
wayblink
e3f2122618
Expose metrics of stanby coordinators (#27698) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-10-16 15:04:09 +08:00
wei liu
43f2452f4b
speed up shard leader cache update frequency (#27640) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-10-16 10:52:10 +08:00
yah01
be980fbc38
Refine state check (#27541) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-11 21:01:35 +08:00
foxspy
2b46bd1f08
fix diskann searchCache param compatibility (#27557) 
Signed-off-by: xianliang <xianliang.li@zilliz.com>
 2023-10-11 16:57:34 +08:00
yah01
2df9908c6a
Map old error code to new error (#27616) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-11 12:43:33 +08:00
yah01
3759857bc5
Refine Proxy errors (#27499) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-09 10:09:33 +08:00
XuanYang-cn
56c94cdfa7
Add channel manager in DataNode (#27308) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-10-08 21:37:33 +08:00
yah01
41495ed266
Improve the error message for getting all indexes of collection (#27389) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-08 21:23:32 +08:00
PowderLi
8d3069b1db
update openssl to 3.1.2 (#27399) 
deal with root path's normalization

Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-10-08 19:17:31 +08:00
congqixia
5d558623fe
Add revive sub-lints and fix existing problems (#27495) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-07 20:53:38 +08:00
congqixia
80eb5434d3
Fix var-naming caused by old PRs (#27501) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-10-07 14:43:32 +08:00
yah01
8394b3a1ec
Block creating new error from status reason (#27426) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-10-07 11:29:32 +08:00
yah01
63ac43a3b8
Refine errors for import (#27379) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-30 10:31:28 +08:00
yihao.dai
106c17f304
Make read ahead policy in ChunkCache configurable (#27291) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2023-09-28 15:47:27 +08:00
congqixia
258e1ccd66
Refine querynode scheduler lifetime (#26915) 
This PR refines scheduler lifetime control:
- Move private tri-state into lifetime package
- Make scheduler block incoming "Add" task
- Make scheduler Stop wait until all previously accepted task done

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-28 10:21:26 +08:00
MrPresent-Han
505e8509b7
refine the default values for segcore threadpool size (#27406) 
/kind improvement

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2023-09-27 20:43:28 +08:00
wei liu
4071132f6a
reload loading collection when qc recover (#27300) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-27 11:43:28 +08:00
SimFG
9fb4c27a90
Fix rootcoord failure to start when reaching the limit of role num (#27361) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-27 11:03:26 +08:00
congqixia
a3dd2756cf
Add predicates for TxnKV operations (#27365) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-27 10:21:26 +08:00
yah01
6539a5ae2c
Refine DataCoord status (#27262) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-26 17:15:27 +08:00
cai.zhang
30a6c6072e
Fix bug for loading array data panic (#27343) 
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2023-09-26 12:23:26 +08:00
wayblink
7dd0be1b2c
Enable bulkinsert binlog data with partitionkey (#27241) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2023-09-26 10:25:25 +08:00
wei liu
27046e242b
add timeout for graceful stop (#27326) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2023-09-26 10:15:25 +08:00
foxspy
5db4a0489e
dynamic index version control (#27335) 
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-25 21:39:27 +08:00
foxspy
370b6fde58
milvus support multi index engine (#27178) 
Co-authored-by: longjiquan <jiquan.long@zilliz.com>
 2023-09-22 09:59:26 +08:00
congqixia
67a471e09c
Fix gofumpt issue in scann_checker_test.go (#27305) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-22 00:39:26 +08:00
Gao
ab2d8dd0a7
Add ScaNN index param checker (#27268) 
Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2023-09-21 14:51:25 +08:00