test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-02 11:59:00 +08:00
Code Issues Actions 12 Packages Projects Releases Wiki Activity
20,272 Commits 94 Branches 105 Tags 184 MiB
1a248f2668
Commit Graph

777 Commits

Author SHA1 Message Date
congqixia
1a248f2668
enhance: Add param item for segmentFlushInterval (#34629) 
See also #28817

Add paramitem for segment flush interval

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-12 18:59:35 +08:00
SimFG
d8e68cbb97
enhance: use the insert rate limit for the upsert request (#34615) 
- issue: #34614

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-07-12 16:17:35 +08:00
Patrick Weizhi Xu
104d0966b7
feat: support partition key isolation (#34336) 
issue: #34332

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-07-11 19:01:35 +08:00
wayblink
fd3da90089
enhance: Refine clustering compaction latency metrics (#34528) 
#34527

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-11 17:43:43 +08:00
chyezh
1bc3c0b925
enhance: implement balancer at streaming coord (#34435) 
issue: #33285

- add balancer implementation
- add channel count fair balance policy
- add channel assignment discover grpc service

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-07-11 09:58:48 +08:00
jaime
c332f69dec
enhance: skip orphan channel cp meta when checking cp lag (#34555) 
issue: # #34545

Print warn log instead of check health fail if orphan channel cp meta is
found in health check request.

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-11 09:36:56 +08:00
yihao.dai
aef7664fc6
fix: Prevent dispatcher merging if curTs is 0 (#34562) 
When the main dispatcher has not yet consumed data, curTs is 0. During
this time, merging dispatchers should not be allowed; otherwise, the
data of the solo dispatcher will be skipped.

issue: https://github.com/milvus-io/milvus/issues/34255

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-11 00:02:50 +08:00
SimFG
b58a5617ef
enhance: add the seal segment when dispatch delete msgs (#34565) 
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-07-10 19:14:51 +08:00
wei liu
9b37d3f517
enhance: Enable setting the replica number and resource group during collection creation (#34403) 
issue: #30040

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-10 10:20:13 +08:00
aoiasd
c1e04534c3
enhance:change access log write cache default config (#34354) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-07-09 18:18:12 +08:00
yihao.dai
391d2aa5b5
fix: Fix unstable ut of msgdispatcher (#34485) 
issue: https://github.com/milvus-io/milvus/issues/34484

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-09 10:08:11 +08:00
congqixia
3333160b8d
enhance: Fix lint issues from recent PRs (#34482) 
See also #34483
Some lint issues are introduced due to lack of static check run. This PR
fixes these problems.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-09 10:06:24 +08:00
Patrick Weizhi Xu
df5ba3fae3
enhance: update milvus proto (#34490) 
issue: #34336

Revert CollectionSchema changes

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-07-09 10:04:11 +08:00
chyezh
dfe0416a70
enhance: implement streaming node server service (#34166) 
issue: #33285

- implement producing and consuming server of message
- implement management operation for streaming node server

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-07-09 07:58:14 +08:00
congqixia
80b620ebcf
fix: Check err is ErrKeyNotFound when CASCachedValue (#34488) 
See also #33785

When config item is not present in paramtable, CAS fails due to
GetConfig returns error.

This PR make this returned err instance of ErrKeyNotFound and check
error type in \`CASCachedValue\` methods.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-08 22:00:16 +08:00
Chun Han
8af187f673
fix: lose partitionIDs when scalar pruning and refine segment prune ratio metrics(#30376) (#34477) 
related: #30376
fix: paritionIDs lost when no setting paritions
enhance: refine metrics for segment prune

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
 2024-07-08 19:54:15 +08:00
Gao
ae6d6f91e6
enhance: change autoindex default metric type (#34261) 
issue: #34304 
cosine is more widely used in float vectors, and cosine and hamming
distance are 'metrics' which have good geometric properties

Signed-off-by: chasingegg <chao.gao@zilliz.com>
 2024-07-08 19:52:24 +08:00
congqixia
145f4f919b
fix: Add nbits parameter check for IVF_PQ (#34451) 
See also #34426

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-08 11:16:10 +08:00
yihao.dai
43fd8d19c2
enhance: Check segment existence when FlushSegments and add some key logs (#34438) 
Check if the segment exists during FlushSegments and add some key logs
in write path.

issue: https://github.com/milvus-io/milvus/issues/34255

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-06 08:50:11 +08:00
jaime
60be454db0
enhance: add disk quota and max collections into db properties (#34368) 
issue: #34385

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-05 18:22:17 +08:00
zhagnlu
18f2458385
fix: fix bitmap supported type (#34350) 
#34314

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
 2024-07-05 15:50:10 +08:00
jaime
21fc5f5d46
enhance: Remove datanode reporting TT based on MQ implementation (#34421) 
issue: #34420

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-05 15:48:09 +08:00
SimFG
0817802db8
enhance: use the key lock and concurrent map in the msg dispatcher client (#34278) 
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-07-05 15:11:31 +08:00
chyezh
ba04981a43
enhance: implement wal managerment on streaming node (#34153) 
issue: #33285

- add lifetime control for wal.
- implement distributed-safe wal manager on streaming node.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-07-05 14:29:42 +08:00
chyezh
7611128e57
enhance: wal adaptor implementation (#34122) 
issue: #33285

- add adaptor to implement walimpls into wal interface.
- implement timetick sorted and filtering scanner.
- add test for wal.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-07-04 15:23:08 +08:00
SimFG
c6e2900cfc
enhance: add the tick log for the tt msgstream seek method (#34397) 
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-07-04 10:18:09 +08:00
XuanYang-cn
8a2be8a457
fix: DataNode might OOM by estimating based on MemorySize (#34201) 
See also: #34136

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-07-04 10:16:17 +08:00
wei liu
9db897985b
enhance: refine misleading param name for bloom filter parallel factor (#34334) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-04 10:06:10 +08:00
congqixia
0fd0fcfe1d
enhance: Fix lint issues & sdk testcase (#34399) 
Some lint issue is not detect due to recent static check pipeline issue.
This PR fixes these problem and Go milvusclient testcases.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-03 19:42:10 +08:00
congqixia
94fb5805d5
fix: Use raw parameter value to perform CAS (#34343) 
See also #34342

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-03 16:00:08 +08:00
cai.zhang
8165961223
enhance: Optimeize clustering compaction (#34313) 
issue: #30633

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-03 15:14:08 +08:00
Patrick Weizhi Xu
32c943f1bc
enhance: [skip e2e] update the version of MV (#34378) 
issue: #29892

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-07-03 14:28:16 +08:00
Chun Han
34bec2ea5e
enhance: add metrics for segment prune latnecy(#30376) (#34094) 
related: #30376

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-07-03 10:04:07 +08:00
Buqian Zheng
fa8d641ce6
fix: update check for sparse hnsw index (#33713) 
issue: #29419

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
 2024-07-02 21:56:09 +08:00
chyezh
3563136c2a
enhance: timetick interceptor implementation (#34238) 
issue: #33285

- optimize the message package
- add interceptor package to achieve append operation intercepting.
- add timetick interceptor to attach timetick properties for message.
- add timetick background task to send timetick message.

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-07-02 14:42:08 +08:00
cai.zhang
a5be322ab2
enhance: Don't set the timeout for SyncSegments (#34226) 
issue: #32809

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-02 14:28:07 +08:00
yihao.dai
ff51c7e628
fix: Fix panic due to failed to seek (#34229) 
Converting the same msgposition's vchannel to a pchannel multiple times
would result in an invalid pchannel, leading to seek failure and panic.
This PR:
1. Make a copy of msgposition in msgdispatcher.
2. Check if channel is already a pchannel, no further channel conversion
is performed.

issue: https://github.com/milvus-io/milvus/issues/34221

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-01 16:08:12 +08:00
aoiasd
186757e622
enhance: support mark error as user error (#33498) 
relate: https://github.com/milvus-io/milvus/issues/33492

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2024-07-01 14:56:12 +08:00
jaime
d6afb31b94
enhance: make subfunctions of datanode component modular (#33992) 
issue: #33994

also remove deprecated channel manager based on the etcd implementation

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-01 14:46:07 +08:00
jaime
0426390f06
enhance: improve check health (#33800) 
issue: #34264

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-01 10:16:06 +08:00
wayblink
73ffc1b424
Refine clustering_compaction_task retry mechanism (#34194) 
#32939

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-06-30 20:22:09 +08:00
XuanYang-cn
dda70aa81b
fix: LegacyVersionWithoutRPCWatch default value to 2.4.1 (#34184) 
See also: #31933

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-27 19:36:06 +08:00
chyezh
d2bc4a53be
enhance: implement rmq and pulsar as wal (#34046) 
issue: #33285

- use reader but not consumer for pulsar
- advanced test framework
- move some streaming related package into pkg

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-06-27 15:11:05 +08:00
cqy123456
51ebe95be5
enhance: remove CheckVecIndexWithDataTypeExist function in pkg and remove some cgo call (#34102) 
issue: https://github.com/milvus-io/milvus/issues/22837
related pr: https://github.com/milvus-io/milvus/pull/34104

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-26 21:13:44 +08:00
cqy123456
381d43717d
enhance: update new mmap config parmeters version (#34143) 
issue: https://github.com/milvus-io/milvus/issues/32984
related pr: https://github.com/milvus-io/milvus/pull/34145

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-25 21:20:05 +08:00
jaime
9630974fbb
enhance: move rocksmq from internal to pkg module (#33881) 
issue: #33956

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-25 21:18:15 +08:00
wayblink
f9a0f7bb25
Add an option to enable/disable vector field clustering key (#34097) 
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-06-25 18:52:04 +08:00
jaime
d08cb885ca
enhance: enable flush rate limiter of collection level (#33837) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-24 10:52:03 +08:00
Patrick Weizhi Xu
b961767005
enhance: support integral type for MV and skip MV if there is only one category (#33161) 
issue: #29892

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
 2024-06-24 10:20:01 +08:00
Chun Han
ca7ef26e4b
fix: sync part stats task cannot be finished(#30376) (#34027) 
related: #30376
also: refine log output for query_coord task by rephrasing action string

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-06-24 10:16:02 +08:00