Commit Graph

8606 Commits

Author SHA1 Message Date
Cai Yudong
16b8b7b35d
enhance: Add get_vector unittest for float16 & bfloat16 (#32153)
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-23 16:15:23 +08:00
Ted Xu
fef7812254
enhance: trigger compaction by size (#32326)
See #32328

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-04-23 12:11:24 +08:00
congqixia
bfebdecf3e
enhance: Make LeaderView Manager filter use map index (#32505)
See also #32165

Change `LeaderViewFilter` to interface to provided map key to avoid
iterating all key-values in LeaderViewManager

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-23 11:07:24 +08:00
chyezh
21a9de5c8e
fix: resource group ut fixup (#32509)
issue: #30647

Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-23 10:01:23 +08:00
sre-ci-robot
15558112c5
[automated] Update Knowhere Commit (#32516)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-04-23 01:33:23 +08:00
Gao
0fab265eed
enhance: update knowhere and some header changes (#32468)
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-04-22 15:47:26 +08:00
aoiasd
d0ffb71a17
enhance: forbid delete with always true expression (#32472)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-04-22 11:09:23 +08:00
congqixia
d7ff1bbe5c
enhance: Make querycoordv2 collection observer task driven (#32441)
See also #32440

- Add loadTask in collection observer
- For load collection/partitions, load task shall timeout as a whole
- Change related constructor to load jobs

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-22 10:39:22 +08:00
Cai Yudong
5fc439c600
feat: Bulk insert support fp16/bf16 (#32157)
Issue: #22837

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-22 10:05:22 +08:00
yiwangdr
037de8e4d3
enhance: speed up minor functions calls in datacoord (#32389)
Related to https://github.com/milvus-io/milvus/issues/32165

1. nodeid based channel store access should use map access instead of
iteration.

2. The join-ish functions calls are slow when # collections/segments
increases (e.g. 10k).
e.g.
getNumRowsOfCollectionUnsafe is O(num_segments); GetAllCollectionNumRows
is of O(num_collections*num_segments).

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-04-20 07:55:21 +08:00
zhagnlu
93f0c262c1
fix: add bitset sve support macro protect (#32463)
#32137

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-04-19 18:05:21 +08:00
XuanYang-cn
7e830f6185
fix: [skip-e2e] TestAvgReassignPolicy ut (#32450)
See also: #32448

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-04-19 15:11:20 +08:00
congqixia
01c16fe6e3
enhance: Manual release pool after save targets (#32358)
See also #31632

Release conc.Pool after usage to clean worker and stop background purge
and ticktock.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-19 13:51:21 +08:00
SimFG
8594b55ad5
enhance: add max insert request size and must use partition key configs (#32433)
issue: https://github.com/milvus-io/milvus/issues/30577
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-19 10:31:20 +08:00
SimFG
31a29a2451
enhance: avoid the repeated metric info in the proxy (#32380)
issue: #30577

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-19 10:21:20 +08:00
PowderLi
a0a23d0666
fix: [restful]use default search parameter nq: 0 (#32355)
issue: #32225 #31978 #32360

1. v1 can only accept one vector, but v2 accept list of vectors #32225
2. cannot get dbName from AliasReq #31978
3. enhance: [restful v2]support config DB-Name in the http header #32360

---------

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-04-18 14:41:20 +08:00
zhagnlu
e2c38750c7
fix: modify retry error (#32351)
#32322

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-04-18 14:25:14 +08:00
presburger
e2541f04eb
fix: crash on arm64 (#31951) (#31980)
issue: #31953

Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
2024-04-17 18:03:20 +08:00
zhenshan.cao
02f17b842a
fix: fix incomplete hybrid search result when nq > 1 (#32177)
issue: https://github.com/milvus-io/milvus/issues/25639

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-04-17 17:09:32 +08:00
Jiquan Long
7add7248e2
fix: remove useless log of quota center (#32307)
/kind improvement
issue: #31272

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-04-17 16:17:19 +08:00
chyezh
a8c8a6bb0f
fix: parameter check of TransferReplica and TransferNode (#32297)
issue: #30647 

- Same dst and src resource group should not be allowed in
`TransferReplica` and `TransferNode`.

-  Remove redundant parameter check.

Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-17 15:27:19 +08:00
yiwangdr
7deda4d5e9
enhance: speed up GetByCollectionAndNode (#32232)
Related to https://github.com/milvus-io/milvus/issues/32165

Avoid iterating through all replicas/collections if possible. Iteration
is expensive when there are large number of replicas/collections.

Signed-off-by: yiwangdr <yiwangdr@gmail.com>
2024-04-17 10:23:25 +08:00
zhagnlu
4586bcef9f
fix: correct AssignSegmentID return and add retry for loadCollectionF… (#32335)
#32322
#31942

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-04-16 10:20:10 -07:00
congqixia
72c172a7d7
enhance: Remove duplicated collectionID label for task latency (#32308)
`CollectionID` already exists in channel name, so remove it to save
metrics traffic.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-16 18:55:19 +08:00
SimFG
84f05ba66e
enhance: make the auth error message more suitable (#32253)
/issue: #32252
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-16 16:47:19 +08:00
congqixia
83da08c388
enhance: Use map instead of slice to maintain channel info (#32273)
See also #32165

`ChannelManager.Match` is a frequent operation for datacoord. When the
collection number is large, iteration over all channels will cost lots
of CPU time and time consuming.

This PR change the data structure storing datanode-channel info to map
avoiding this iteration when checking channel existence.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-16 15:57:19 +08:00
zhagnlu
9248a6a149
fix: remove sve flags (#32270)
#32129

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-04-16 15:21:19 +08:00
yihao.dai
558feed5ed
fix: Use pk from binlog during import (#32118)
During binlog import, even if the primary key's autoID is set to true,
the primary key from the binlog should be used instead of being
reassigned.

issue: https://github.com/milvus-io/milvus/discussions/31943,
https://github.com/milvus-io/milvus/issues/28521

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-04-16 14:51:20 +08:00
chyezh
e19d17076f
fix: delete may lost when enable lru cache, some field should be reset when ReleaseData (#32012)
issue: #30361

- Delete may be lost when segment is not data-loaded status in lru
cache. skip filtering to fix it.

- `stats_` and `variable_fields_avg_size_` should be reset when
`ReleaseData`

- Remove repeat load delta log operation in lru.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-16 11:17:20 +08:00
Gao
55d894bd5e
enhance: support disable search optimization (#32141)
Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-04-16 10:51:20 +08:00
SimFG
1af084ea6b
enhance: Make datanode exit and case TestProxy faster (#32218)
/kind improvement
issue: #32219

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-04-16 10:49:20 +08:00
chyezh
70e3d5b495
fix: wrong node id in TestCheckNodesInReplica (#32268)
issue: #31930

Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-15 17:38:17 +08:00
wei liu
0d849a6c0a
fix: fix collectionInfo leak in datacoord (#32175)
issue: #32029

lack of logic to clean collection info in datacoord's meta, This PR
clean collection info after drop channel, to avoid collection info leak
in datacoord

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-15 16:33:19 +08:00
wei liu
4822b109bd
fix: Skip to load l0 segment on old version query node (#32124)
issue: #32107

during rolling upgrade progress, skip to load l0 segment on old version
query node

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-15 11:23:23 +08:00
congqixia
dc11cbd123
enhance: Maintain collection-patitions mapping in qc meta (#32227)
Related to #32165

Add collection to partitionIDs mapping to avoid interation on all
partitions loaded when trying to get all partitions with collection id

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-15 10:05:19 +08:00
chyezh
48fe977a9d
enhance: declarative resource group api (#31930)
issue: #30647

- Add declarative resource group api

- Add config for resource group management

- Resource group recovery enhancement

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-04-15 08:13:19 +08:00
Chun Han
337cc0756d
fix: lack good results for insufficient ef(#29883) (#32080)
related: #29883

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-04-13 22:13:23 +08:00
yihao.dai
aa96843d31
fix: Fix import hanging and improve logging output (#32166)
Fix import hanging when the previous import task failed, and improve
parquet import logging outout.

issue: https://github.com/milvus-io/milvus/issues/31834

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-04-13 22:03:23 +08:00
Jiquan Long
4fb85be525
fix: put inverted index into local storage (#32209)
issue: https://github.com/milvus-io/milvus/issues/32154

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-04-13 21:57:19 +08:00
PowderLi
610a65af14
fix: restful v2 (#32144)
issue: #31176

1. cannot get dbName correctly while describe alias #31978
2. return a valid json string even if the user doesn't have the whole
privileges to describe collection #31635
3. rename IndexParam.IndexConfig to IndexParam.Params
4. FieldSchema.ElementTypeParams, IndexParam.Params can not only accept
string

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-04-13 21:55:29 +08:00
congqixia
674991d07c
fix: Wait StandBy server ready for testcase (#32216)
See also #32069

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-12 23:49:18 +08:00
Cai Yudong
06e0c8baac
fix: fix estimate float16 field data size wrong (#32193)
Issue: #32192

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-04-12 16:29:26 +08:00
zhenshan.cao
88c6828d6c
fix: failed to raise metric_type not match error (#32202)
issue: https://github.com/milvus-io/milvus/issues/32176

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-04-12 16:19:18 +08:00
SimFG
c012e6786f
feat: support rate limiter based on db and partition levels (#31070)
issue: https://github.com/milvus-io/milvus/issues/30577
co-author: @jaime0815

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
Signed-off-by: SimFG <bang.fu@zilliz.com>
Co-authored-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-04-12 16:01:19 +08:00
aoiasd
fb376fd1e6
enhance: Refine varchar length exceeds max length error message (#31521)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-04-12 15:01:19 +08:00
XuanYang-cn
4617d22482
enhance: Use channel manager interface in server_test (#31621)
Tidy the following test codes

    - Remove channel in newTestServer
    - Remove newTestServerWithMeta
    - Remove newTestServer2
    - Remove testDataCoordBase
    - Use the same func for handleTTmsg and handleRPCTTmsg

See also: #31620

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-04-12 14:59:20 +08:00
congqixia
c0fa169d9a
enhance: Make write buffer memory check do until safe (#32172)
See also #27675 #26177

Make memory check evict memory buffer until memory water level is safe.
Also make `EvictBuffer` wait until sync task done.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-04-12 10:55:18 +08:00
wei liu
68dec7dcd4
fix: Use correct ts to avoid exclude segment list leak (#31991)
issue: #31990

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-04-12 10:39:19 +08:00
aoiasd
43a9be2cce
enhance: support access log dynamic config and prepare for support resutful interface access log (#29638)
relate: https://github.com/milvus-io/milvus/issues/29637
https://github.com/milvus-io/milvus/issues/31823

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-04-12 10:37:27 +08:00
sre-ci-robot
454984aa4e
[automated] Update Knowhere Commit (#32181)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-04-12 01:25:19 +08:00