Commit Graph

9170 Commits

Author SHA1 Message Date
congqixia
a642a26ed4
enhance: Resolve ChunkFileWriter lint issue (#35166)
See also #34483

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-01 16:52:13 +08:00
wayblink
5bbb1c201c
enhance:support l2 single compaction (#34935)
#34928

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-01 14:36:13 +08:00
zhagnlu
f8c1b138a8
fix:fix get array error for int type (#35154)
#35055

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-01 14:30:12 +08:00
zhenshan.cao
aa247f192d
enhance: remove unused code for StorageV2 (#35132)
issue: https://github.com/milvus-io/milvus/issues/34168

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-08-01 12:08:13 +08:00
cai.zhang
9412002d7d
fix: Fix data race for clustering buffer writer (#35145)
issue: #34495

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-01 11:20:13 +08:00
cai.zhang
089fb2b5fb
enhance: Skip updating the level that are already at the specified level (#35094)
issue: #34495

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-01 11:18:12 +08:00
Bingyi Sun
f229f244d2
enhance: add chunk basic impl (#34634)
https://github.com/milvus-io/milvus/issues/35112
This pr would not affect milvus functionality by now.
It implments a Chunk memory layout that looks like 
```
VariableColumn
|offset|offset|offset|
|data|data|data|

```
We maybe move offsets to the beginning and add null bitmaps later but
not in this PR.
And mmap test will also be added in another PR.

---------

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-08-01 10:29:51 +08:00
wei liu
e9d61daa3f
enhance: Reduce delegator memory overloaded factor to 0.1 (#35092)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-01 10:21:50 +08:00
sre-ci-robot
c15d074b21
[automated] Update Knowhere Commit (#35157)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-08-01 01:17:49 +08:00
Gao
6695c6d0a3
enhance: add channel num for queryHook optimization (#35104)
At most cases, data in each channel is almost evenly distributed, we
could utilize the channel num info to optimize searh param in queryHook

Signed-off-by: chasingegg <chao.gao@zilliz.com>
2024-07-31 18:59:49 +08:00
wei liu
03912a8788
enhance: Avoid balance stuck after segment list become stable (#34728)
issue: #34715
if collection's segment list doesn't changes anymore, then the next
target will be empty at most time, and balance segment will check
whether segment exist in both current and next target, so the balance
cloud be blocked due to next target is empty.

This PR permit segment to be moved if next target is empty, to avoid
balance stuck.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-31 18:09:48 +08:00
Patrick Weizhi Xu
a9744d62b1
enhance: improve error msg when expr invalid during isolation (#35127)
issue: #34336 

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit b1d7b24972e9414ab9488c7611aca9975e674cd4)
2024-07-31 17:23:51 +08:00
smellthemoon
6106a48acb
fix: upsert result use the previous pk (#34672)
#34668

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-07-31 15:25:51 +08:00
Patrick Weizhi Xu
acaa78db58
fix: disable sparse and binary in MV (#35126)
issue: #35120 

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit 0d340ebe7412cd85963825bc8abd3c874e339679)
2024-07-31 15:02:01 +08:00
congqixia
f7f9a729c9
enhance: Pre-allocate space for reduce data structure (#35118)
Grow slice & map.growWork may cause a lot when segment number is large
for big K query. This PR pre-allocate space for reduce methods to avoid
this cost.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-31 10:35:49 +08:00
foxspy
48b14c56f6
fix: add prometheus dependency for milvus_monitor (#35078)
add prometheus dependency for monitor module. Or else Some compilers may
report a compilation failure.
issue: #35077

Signed-off-by: xianliang.li <xianliang.li@zilliz.com>
2024-07-31 10:15:48 +08:00
wayblink
f919b6e1b8
enhance: refine error handle in clustering compaction task (#34956)
related: #30633 #34792

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-30 20:37:57 +08:00
wayblink
ce3f836876
fix: compaction task not be cleaned correctly (#34765)
1.fix compaction task not be cleaned correctly
2.add a new parameter to control compaction gc loop interval
3.remove some useless configs of clustering compaction

bug: #34764

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-30 20:21:56 +08:00
zhagnlu
dd0c26cf58
enhance: redefine variable column block size (#35040)
#35013

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-30 19:23:50 +08:00
congqixia
de8a266d8a
enhance: Enable linux code checker (#35084)
See also #34483

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-30 15:53:51 +08:00
wei liu
4bb969ef8f
enhance: Skip manual stopped component during health check (#34953)
after manual stop component by management restful api, `healthz` may
return unhealthy state. k8s may restart the pod to save the unhealthy
sate, and the manual stop operation will got unexpected result.

to solve this, we make `healthz` API skip the manual stopped component.

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-30 14:37:51 +08:00
zhagnlu
a8a4779749
fix: fix parse plan proto failed for search type (#34944)
#25848

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-29 21:19:49 +08:00
zhagnlu
86322e0468
fix: fix query count(*) concurrently (#35007)
#34778
#34849
fix two problems:
1. count(*) incorrect, if growing insert duplicated (pk, timestamp)
pairs that pk and timestamp all same, need to keep just one pair.
2. count(*) may core dump, if get_real_count interface get snapshot and
do mvcc at not consistency status, mainly happens under concurrency.

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-29 19:53:50 +08:00
congqixia
972752258a
enhance: Support otlp http exporter (#35053)
See also #35052

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-29 17:43:49 +08:00
chyezh
a495754034
enhance: the datacoord gc should fast quitable (#35050)
issue: #35049

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-29 17:03:48 +08:00
Patrick Weizhi Xu
3c7f73137e
fix: disallow expr when partition key isolation is enabled (#35031)
issue: #34336 

Signed-off-by: Patrick Weizhi Xu <weizhi.xu@zilliz.com>
(cherry picked from commit 2889481ce9a14230e9c7f1c8f9c3c1decde77e03)
2024-07-29 14:21:50 +08:00
wei liu
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 (#34394)
issue: #34252

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-29 11:31:51 +08:00
Chun Han
e2e38e98df
fix: nil part stats without l2 compaction(#34923) (#34992)
related: #34923

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-29 11:07:48 +08:00
XuanYang-cn
9a07790215
enhance: Add ut for l0CompactionTask processExecuting (#34800)
See also: #34796

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-07-29 10:53:48 +08:00
zhagnlu
f77f5364b2
fix: disable use_index when some array expr (#34894)
#34797

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-29 00:17:46 +08:00
PowderLi
5584ae2e14
enhance: add metrics for restful apis (#34969)
issue: #34968

Signed-off-by: PowderLi <min.li@zilliz.com>
2024-07-29 00:13:47 +08:00
cai.zhang
497afcb897
enhance: Refine code for GetRecoveryInfo (#34973)
issue: #34495

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-29 00:01:46 +08:00
cai.zhang
2372452fac
enhance: Optimized the GC logic to ensure that memory is released in time (#34949)
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-28 23:53:47 +08:00
jaime
08fa51d4f4
fix: memory leak while parsing query plan (#34931)
issue: #34930

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-07-28 21:50:20 +08:00
Aldrin
9463eeef2b
fix: Avoided dereferencing NULL pointer (#34836)
issue : https://github.com/milvus-io/milvus/issues/34835

Signed-off-by: Ald392 <imagesai32@gmail.com>
2024-07-27 17:27:52 +08:00
zhagnlu
804ec24c02
fix:fix retrieve raw data from bitmap array index (#34848)
#34795

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-07-27 01:53:47 +08:00
sre-ci-robot
e93687cdd3
[automated] Update Knowhere Commit (#35039)
Update Knowhere Commit
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-07-27 00:59:46 +08:00
congqixia
783f9d9c33
fix: Unify hook singleton implementation in proxy (#34887)
Related to #34885

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-26 18:07:53 +08:00
wei liu
6e9fbd1630
enhance: Solve channel unbalance on datanode (#34984)
issue: #33583
the old policy permit datanode has at most 2 more channels than other
datanode. so if milvus has 2 datanode and 2 channels, both 2 channels
will be assign to 1 datanode, left another datanode empty.

This PR refine the balance policy to solve channel unbalance on datanode

---------

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-26 17:35:47 +08:00
congqixia
94d8f48b85
enhance: Use auto detection proto interface to fetch grpc size (#35024)
Related to #34763

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-26 14:23:47 +08:00
cai.zhang
ea13b97b6a
fix: Fix bug for block clustering compaction (#35019)
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-26 11:33:58 +08:00
chyezh
1cff55381d
enhance: add manual alloc segment rpc for datacoord (#35002)
issue: #33285

- segment allocation will move to streamingnode, so a manual alloc
segment rpc is required

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-26 10:15:46 +08:00
congqixia
4ee6c69217
enhance: Add Segment Level in milvus segment info APIs (#34763)
See also #34746

This PR add segment level field in response of
`GetPersistentSegmentInfo` and `GetQuerySegmentInfo`

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-26 10:01:46 +08:00
congqixia
e8038a76e1
fix: Remove timeout in datanode watch ctx (#35011)
See also #35008

Use tickle timeout logic instead of hardcode context timeout

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-25 22:17:45 +08:00
Chun Han
c46c401112
fix: refine handling type for segment pruner(#34923) (#34925)
related: #34923

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-07-25 13:57:45 +08:00
chyezh
d93c51d5f6
enhance: change GetVChannels rpc into GetPChannelInfo (#34940)
issue: #33285

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-25 13:29:45 +08:00
chyezh
bef06e5acf
enhance: add streaming coord and node grpc service register (#34655)
issue: #33285

- register streaming coord service into datacoord.
- add new streaming node role.
- add global static switch to enable streaming service or not.

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-25 12:19:44 +08:00
chyezh
4f6cbfd520
enhance: specialized immutable and mutable message (#34951)
issue: #33285

- add specialized mutable and immutable message, make type safe.
- add version based constructor and type.

Signed-off-by: chyezh <chyezh@outlook.com>
2024-07-25 11:57:45 +08:00
Jiquan Long
b843c91bad
feat: record the number of times milvus turns into force-deny-writing states (#34960)
/kind improvement
fix: #34961

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-07-25 11:39:46 +08:00
congqixia
e2f40fc2a8
fix: Check legacy guarantee ts when skipping alloc ts (#34981)
See also #34980

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-25 10:17:45 +08:00