Commit Graph

20265 Commits

Author SHA1 Message Date
XuanYang-cn
d7966f46ad
fix: Compaction task l0 state transfer wrong (#34597)
This bug caused failed L0 compaction tasks never end.

See also: #34460

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-11 21:39:36 +08:00
Patrick Weizhi Xu
104d0966b7
feat: support partition key isolation (#34336)
issue: #34332

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-07-11 19:01:35 +08:00
XuanYang-cn
d7a3697fb5
enhance: Add back compactionTaskNum metrics (#34583)
Fix L0 compaction task recover unable to set segment not isCompacting

See also: #34460

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-11 17:45:37 +08:00
wayblink
fd3da90089
enhance: Refine clustering compaction latency metrics (#34528)
#34527

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-11 17:43:43 +08:00
smellthemoon
07b94b4615
enhance: support upsert autoid==true (#30342)
related with: #29258

---------

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-07-11 16:53:35 +08:00
ThreadDao
e4115485b6
test: add cases for gosdk v2 partitions (#34593)
issue: #33419 #34426

Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2024-07-11 16:49:36 +08:00
XuanYang-cn
e0b39d8bf4
fix: Milvus panic when compaction disabled and dropping a collection (#34103)
See also: #31059

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-11 14:44:52 +08:00
congqixia
be92147723
enhance: Print reason when seal segments (#34575)
See also #34574

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-11 11:40:51 +08:00
jaime
3b62138c5c
fix: unstable UT for level0 deletion (#34524)
issue: #34533

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-11 10:02:56 +08:00
chyezh
1bc3c0b925
enhance: implement balancer at streaming coord (#34435)
issue: #33285

- add balancer implementation
- add channel count fair balance policy
- add channel assignment discover grpc service

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-11 09:58:48 +08:00
jaime
c332f69dec
enhance: skip orphan channel cp meta when checking cp lag (#34555)
issue: # #34545

Print warn log instead of check health fail if orphan channel cp meta is
found in health check request.

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-11 09:36:56 +08:00
yihao.dai
aef7664fc6
fix: Prevent dispatcher merging if curTs is 0 (#34562)
When the main dispatcher has not yet consumed data, curTs is 0. During
this time, merging dispatchers should not be allowed; otherwise, the
data of the solo dispatcher will be skipped.

issue: https://github.com/milvus-io/milvus/issues/34255

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-11 00:02:50 +08:00
SimFG
b58a5617ef
enhance: add the seal segment when dispatch delete msgs (#34565)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-10 19:14:51 +08:00
wei liu
ebc68d2774
fix: Indexnode stuck at stopping progress cause by wrong lifetime control (#34558)
issue: #34557

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-10 15:12:13 +08:00
congqixia
4850336ca3
fix: Write padding at end of mmap file not chunk (#34529)
Related to #34508

The padding bytes shall be written only at the end of the mmap file not
the chunk of each field data file.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-10 11:12:14 +08:00
wei liu
9b37d3f517
enhance: Enable setting the replica number and resource group during collection creation (#34403)
issue: #30040

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-10 10:20:13 +08:00
congqixia
d60e628aed
enhance: Avoid use concrete segment type in segments interfaces (#34521)
See also #34519

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-10 10:18:12 +08:00
wei liu
3c7046353a
fix: RootCoord panic during check health (#34541)
issue: #34540

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-10 09:51:44 +08:00
sre-ci-robot
2f70bbed88 Update all contributors
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2024-07-10 00:01:19 +00:00
sre-ci-robot
0b3f317b14
[automated] Update Knowhere Commit (#34549)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-07-10 01:26:14 +08:00
XuanYang-cn
314f4d995b
enhance: Tidy dc channel manager (#34515)
See also: #34518

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-09 18:26:12 +08:00
aoiasd
c1e04534c3
enhance:change access log write cache default config (#34354)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-07-09 18:18:12 +08:00
jaime
4365308241
enhance: support setting properties in create database request (#34510)
issue: #34493

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-09 18:16:18 +08:00
SimFG
5016038781
enhance: release the record in delete codec and add some log for compaction (#34454)
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-09 15:40:17 +08:00
Ted Xu
eae4dfca7b
fix: reuse deserialize result to help improve memory management (#34507)
Fixed #33268
The original reuse is broken by #33359

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-07-09 14:12:10 +08:00
wei liu
eeb03a0d6a
fix: Query may return deleted records (#34501)
issue: #34500
cause the sort in `GetLevel0Deletions` will broken the corresponed order
between pks and tss, then the pks and tss will be sorted in
segment.Delete() interface.

This PR remove this uncessary and incorrect sort progress to avoid query
may return deleted records.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-09 10:46:11 +08:00
wayblink
0c4d82c6d3
fix: fix compaction task startTime is 0 (#34496)
fix bug: #34497 #34498

---------

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-09 10:22:13 +08:00
yihao.dai
391d2aa5b5
fix: Fix unstable ut of msgdispatcher (#34485)
issue: https://github.com/milvus-io/milvus/issues/34484

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-09 10:08:11 +08:00
congqixia
3333160b8d
enhance: Fix lint issues from recent PRs (#34482)
See also #34483
Some lint issues are introduced due to lack of static check run. This PR
fixes these problems.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-09 10:06:24 +08:00
Patrick Weizhi Xu
df5ba3fae3
enhance: update milvus proto (#34490)
issue: #34336

Revert CollectionSchema changes

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-07-09 10:04:11 +08:00
chyezh
dfe0416a70
enhance: implement streaming node server service (#34166)
issue: #33285

- implement producing and consuming server of message
- implement management operation for streaming node server

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-09 07:58:14 +08:00
congqixia
80b620ebcf
fix: Check err is ErrKeyNotFound when CASCachedValue (#34488)
See also #33785

When config item is not present in paramtable, CAS fails due to
GetConfig returns error.

This PR make this returned err instance of ErrKeyNotFound and check
error type in \`CASCachedValue\` methods.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-08 22:00:16 +08:00
yihao.dai
e15ac2b472
fix: Fix incorrect segment num rows (#34441)
Repeated calls to UpdateStatistics, this PR correct it.

issue: https://github.com/milvus-io/milvus/issues/34440

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-08 20:00:14 +08:00
Chun Han
8af187f673
fix: lose partitionIDs when scalar pruning and refine segment prune ratio metrics(#30376) (#34477)
related: #30376
fix: paritionIDs lost when no setting paritions
enhance: refine metrics for segment prune

Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2024-07-08 19:54:15 +08:00
Gao
ae6d6f91e6
enhance: change autoindex default metric type (#34261)
issue: #34304 
cosine is more widely used in float vectors, and cosine and hamming
distance are 'metrics' which have good geometric properties

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-07-08 19:52:24 +08:00
Aldrin
686a212d8b
fix: Upgraded Azidentity Package to v1.6.0 (#34464)
issue : https://github.com/milvus-io/milvus/issues/34456

Signed-off-by: Ald392 <imagesai32@gmail.com>
2024-07-08 17:51:32 +08:00
XuanYang-cn
467113deab
fix: Accidently exit the check loop (#34481)
See also: #34460

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-08 16:46:13 +08:00
elstic
f7898a3ad6
test: update test case (#34108)
Signed-off-by: elstic <hao.wang@zilliz.com>
2024-07-08 11:46:10 +08:00
congqixia
145f4f919b
fix: Add nbits parameter check for IVF_PQ (#34451)
See also #34426

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-08 11:16:10 +08:00
wayblink
efdaed4ac6
fix: fix clustering compaction can't enqueue when compaction queue is full (#34445)
#30633

---------

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-08 10:02:13 +08:00
yihao.dai
4e5f1d5f75
enhance: Pre-allocate ids for import (#33958)
The import is dependent on syncTask, which in turn relies on the
allocator. This PR pre-allocate the necessary IDs for import syncTask.

issue: https://github.com/milvus-io/milvus/issues/33957

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-07 21:26:14 +08:00
shaoting-huang
f4dd7c7efb
enhance: add delta log stream new format reader and writer (#34116)
issue: #34123

Benchmark case: The benchmark run the go benchmark function
`BenchmarkDeltalogFormat` which is put in the Files changed. It tests
the performance of serializing and deserializing from two different data
formats under a 10 million delete log dataset.

Metrics: The benchmarks measure the average time taken per operation
(ns/op), memory allocated per operation (MB/op), and the number of
memory allocations per operation (allocs/op).
| Test Name | Avg Time (ns/op) | Time Comparison | Memory Allocation
(MB/op) | Memory Comparison | Allocation Count (allocs/op) | Allocation
Comparison |

|---------------------------------|------------------|-----------------|---------------------------|-------------------|------------------------------|------------------------|
| one_string_format_reader | 2,781,990,000 | Baseline | 2,422 | Baseline
| 20,336,539 | Baseline |
| pk_ts_separate_format_reader | 480,682,639 | -82.72% | 1,765 | -27.14%
| 20,396,958 | +0.30% |
| one_string_format_writer | 5,483,436,041 | Baseline | 13,900 |
Baseline | 70,057,473 | Baseline |
| pk_and_ts_separate_format_writer| 798,591,584 | -85.43% | 2,178 |
-84.34% | 30,270,488 | -56.78% |

Both read and write operations show significant improvements in both
speed and memory allocation.

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-07-06 09:08:09 +08:00
yihao.dai
43fd8d19c2
enhance: Check segment existence when FlushSegments and add some key logs (#34438)
Check if the segment exists during FlushSegments and add some key logs
in write path.

issue: https://github.com/milvus-io/milvus/issues/34255

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-06 08:50:11 +08:00
charleskakumanu
05df70973c
enhance: retry etcd server start with 3 default retries (#34208)
Added retry method and unit test cases for retrying etcd server start.
New to open source and Go language. Please reject if this is not the
requirement/ specify the modifications needed in code.
issue : #17569

Signed-off-by: Charles Kakumanu <charles_kakumanu@apple.com>
Co-authored-by: Charles Kakumanu <charles_kakumanu@apple.com>
2024-07-05 20:24:11 +08:00
jaime
60be454db0
enhance: add disk quota and max collections into db properties (#34368)
issue: #34385

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-05 18:22:17 +08:00
congqixia
a1c65dec89
fix: [GoSDK] Avoid creating schema for non-fast CreateCollection (#34448)
See also #31293

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-05 18:02:09 +08:00
congqixia
6b4d977a10
fix: Write padding into mmap file in case of SIGBUS (#34443)
See also #34442

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-05 17:44:09 +08:00
aoiasd
233b8486ec
fix: generate gin logs failed cause access logs panic (#34142)
relate: https://github.com/milvus-io/milvus/issues/34141

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-07-05 16:30:10 +08:00
zhagnlu
18f2458385
fix: fix bitmap supported type (#34350)
#34314

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-05 15:50:10 +08:00
jaime
21fc5f5d46
enhance: Remove datanode reporting TT based on MQ implementation (#34421)
issue: #34420

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-05 15:48:09 +08:00