Commit Graph

20507 Commits

Author SHA1 Message Date
wayblink
c79d1af390
enhance: Add compaction task slot usage logic (#34581)
#34544

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-18 10:27:41 +08:00
cai.zhang
0c0ca4cf0e
fix: Fix bug where binlogs already flushed with new segment during pack (#34762)
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-18 09:15:48 +08:00
yihao.dai
4939f82d4f
enhance: Seal by total growing segments size (#34692)
Seals the largest growing segment if the total size of growing segments
of each shard exceeds the size threshold(default 4GB). Introducing this
policy can help keep the size of growing segments within a suitable
level, alleviating the pressure on the delegator.

issue: https://github.com/milvus-io/milvus/issues/34554

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-17 21:45:41 +08:00
SimFG
e56ab76d99
fix: make the log level can be dynamically changed at runtime (#34757)
- issue: #34756

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-17 20:31:42 +08:00
SimFG
203fb554a4
enhance: support to config root user's password (#34752)
- issue: #33058

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-17 20:19:42 +08:00
sre-ci-robot
1fd477d145 Update all contributors
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2024-07-17 12:00:48 +00:00
pingliu
21aa23a23e
doc: [skip e2e] fix the build deb (#34751)
Signed-off-by: ping.liu <ping.liu@zilliz.com>
2024-07-17 19:39:42 +08:00
shaoting-huang
88b373b024
enhance: binlog primary key turn off dict encoding (#34358)
issue: #34357 

Go Parquet uses dictionary encoding by default, and it will fall back to
plain encoding if the dictionary size exceeds the dictionary size page
limit. Users can specify custom fallback encoding by using
`parquet.WithEncoding(ENCODING_METHOD)` in writer properties. However,
Go Parquet [fallbacks to plain
encoding](e65c1e295d/go/parquet/file/column_writer_types.gen.go.tmpl (L238))
rather than custom encoding method users provide. Therefore, this patch
only turns off dictionary encoding for the primary key.

With a 5 million auto ID primary key benchmark, the parquet file size
improves from 13.93 MB to 8.36 MB when dictionary encoding is turned
off, reducing primary key storage space by 40%.

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-07-17 17:47:44 +08:00
congqixia
67324eb809
enhance: Add l0 segment entry num quota (#34733)
See also #34670

This PR add quota configuration for l0 segment entry number per
collection. If l0 compaction cannot keep up the insertion/upsertion
rate, this feature could back press the related rate.

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-17 17:35:41 +08:00
Buqian Zheng
aa5418a5a9
fix: correctly set search params when using knowhere iterator (#34731)
issue: #34730

Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-07-17 15:13:41 +08:00
yihao.dai
ca758c36cc
enhance: Pre-allocate ids for compaction (#34187)
This PR removes the dependency of compaction on the ID allocator by
pre-allocating the logID and segmentID.

issue: https://github.com/milvus-io/milvus/issues/33957

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-17 13:23:42 +08:00
wayblink
90e765db87
fix: fix bug if datacoord restart when l0 compaction saveSegmentMeta (#34679)
#34681

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-17 13:03:41 +08:00
cai.zhang
17c96e1bde
fix: Fix the bug that caused small segment flush frequently (#34725)
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-17 12:15:40 +08:00
chyezh
3306bc2faf
fix: ut failure for grpc upgrade (#34726)
issue: #34462

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-16 21:49:40 +08:00
sre-ci-robot
9000c29dcd Update all contributors
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2024-07-16 12:02:21 +00:00
XuanYang-cn
fd7221f4be
fix: Fix accidentlly exit MixCompaction task loop (#34688)
See also: #33431, #34460

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-16 15:57:42 +08:00
chyezh
cc8f7aa110
fix: streaming service related fix patch (#34696)
issue: #33285

- add idAlloc interface
- fix binary unsafe bug for message
- fix service discovery lost when repeated address with different server
id

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-16 15:49:38 +08:00
cai.zhang
fbc10ac695
fix:Reset flushed row num after pack segment for clustering compaction (#34702)
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-16 15:37:38 +08:00
zhagnlu
804dd5409a
enhance: mark duplicated pk as deleted (#34586)
fix #34247

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-16 14:25:39 +08:00
Xiaofan
be7760a9ab
fix: CVE by upgrading some dependencies. (#34462)
fix #34434 and #34456
upgrade otelgrpc to fix CVE

Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>
2024-07-16 11:55:36 +08:00
smellthemoon
b5b5afb63f
fix: [skip e2e]golang lint check fail (#34682)
#34680

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-07-16 10:47:36 +08:00
congqixia
ceb138da6d
enhance: [GoSDK] Add release methods & GPU indexes (#34690)
Related to #31293

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-16 10:27:38 +08:00
balloon1995
7306d2d115
fix: fix metaCache cleanup issue when listPolicy failed (#34449)
issue: #34667

---------

Signed-off-by: balloon1995 <hszoe1995@outlook.com>
Co-authored-by: congqixia <congqi.xia@zilliz.com>
2024-07-16 10:03:38 +08:00
congqixia
b5225df21c
enhance: Use unified requery flag when set metrics (#34632)
See also #34601

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-15 23:03:38 +08:00
congqixia
eb4bfa3281
fix: Revert reuse deserialize result to fix data overwritten (#34683)
See also #34637

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-15 22:31:38 +08:00
wei liu
acb33bba4d
enhance: Preserve fixed-size memory in delegator node for growing segment. (#34596)
issue: #34595
When consuming insert data on the delegator node, QueryCoord will move
out some sealed segments to manage its memory usage. After the growing
segment gets flushed, some sealed segments from other workers will be
moved back to the delegator node. To avoid the frequent movement of
segments, we estimate the maximum growing row count and preserve a
fixed-size memory in the delegator node.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-15 20:51:46 +08:00
chyezh
fda720b880
enhance: streaming service grpc utilities (#34436)
issue: #33285

- add two grpc resolver (by session and by streaming coord assignment
service)
- add one grpc balancer (by serverID and roundrobin)
- add lazy conn to avoid block by first service discovery
- add some utility function for streaming service

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-15 20:49:38 +08:00
XuanYang-cn
eb472b7f08
enhance: [skip e2e]Enable compaction it test (#34526)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-15 20:45:39 +08:00
yihao.dai
1495b7a2c9
fix: Fix no L0 compaction tasks are executed (#34537)
The nodeID for compaction task initialization is 0. This PR adjusts the
task reassignment conditions to allow new compaction tasks to be
reassigned and executed.

issue: https://github.com/milvus-io/milvus/issues/34460

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-15 17:01:37 +08:00
congqixia
8b5754f7fe
enhance: Add segment seal proportion jitter (#34636)
See also #34574

Add jitter for segment seal proportion to avoid seal operation burst in
short period of time.

This PR also fix license header in paramtable pkg.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-15 14:47:39 +08:00
wayblink
48898cefea
fix: fix meta revert bug when clustering compaction task fail (#34604)
#34603

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-14 22:29:38 +08:00
zhagnlu
bd9727a1f7
fix: fix bug that set incorrect info to columnbase (#34428)
#34427

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-14 22:27:46 +08:00
zhagnlu
18c83c6466
fix: fix auto merge error (#34661)
#33704

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-13 13:07:37 +08:00
elstic
2c462d387b
test: Fix index creation mismatch for sparse vector columns in multivector case (#34618)
If `is_all_data_type` is true, the case will add float32, f16, bf16, and
sparse vectors, but the created indexes are all `flat` indexes by
default. The sparse type cannot create a flat index. Fix the test code
to create a `SPARSE_INVERTED_INDEX` index for the sparse vector when
is_all_data_type is true

Signed-off-by: elstic <hao.wang@zilliz.com>
2024-07-13 10:45:37 +08:00
cai.zhang
93a500676b
fix: Remove flushed segment in segment manager generated through import (#34649)
issue: #34648

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-12 23:15:22 +08:00
congqixia
1a248f2668
enhance: Add param item for segmentFlushInterval (#34629)
See also #28817

Add paramitem for segment flush interval

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-12 18:59:35 +08:00
SimFG
d8e68cbb97
enhance: use the insert rate limit for the upsert request (#34615)
- issue: #34614

Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-12 16:17:35 +08:00
congqixia
531092c031
enhance: Add lint rule to forbid gogo protobuf (#34594)
github.com/gogo/protobuf is deprecated and could be error prune after
upgrade protobuf message to v2.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-12 10:19:35 +08:00
Chun Han
f00c529aea
feat: support group_size for search_group_by(#33544) (#33720)
related: #33544

mainly changes in three aspects:

1. enable setting group_size for group by function
2. separate normal reduce and group by reduce
3. eleminate uncessary padding in search result for reducing

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-12 10:17:36 +08:00
aoiasd
5bb0d21e32
fix: Restful API use deprecate error code cause access log panic. (#34576)
relate: https://github.com/milvus-io/milvus/issues/34578

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-07-12 10:13:35 +08:00
wayblink
358e9a10d2
enhance: Alter compactTo segments before compactFrom to avoid data loss if crash (#34513)
#34512

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-12 00:55:34 +08:00
jaime
a08a0c831f
fix: encountering orphan channel-cp meta after DataCoord GC (#34612)
issue: #34545

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-11 23:01:35 +08:00
XuanYang-cn
d7966f46ad
fix: Compaction task l0 state transfer wrong (#34597)
This bug caused failed L0 compaction tasks never end.

See also: #34460

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-11 21:39:36 +08:00
Patrick Weizhi Xu
104d0966b7
feat: support partition key isolation (#34336)
issue: #34332

---------

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
2024-07-11 19:01:35 +08:00
XuanYang-cn
d7a3697fb5
enhance: Add back compactionTaskNum metrics (#34583)
Fix L0 compaction task recover unable to set segment not isCompacting

See also: #34460

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-11 17:45:37 +08:00
wayblink
fd3da90089
enhance: Refine clustering compaction latency metrics (#34528)
#34527

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-11 17:43:43 +08:00
smellthemoon
07b94b4615
enhance: support upsert autoid==true (#30342)
related with: #29258

---------

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-07-11 16:53:35 +08:00
ThreadDao
e4115485b6
test: add cases for gosdk v2 partitions (#34593)
issue: #33419 #34426

Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2024-07-11 16:49:36 +08:00
XuanYang-cn
e0b39d8bf4
fix: Milvus panic when compaction disabled and dropping a collection (#34103)
See also: #31059

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-11 14:44:52 +08:00
congqixia
be92147723
enhance: Print reason when seal segments (#34575)
See also #34574

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-11 11:40:51 +08:00