Commit Graph

20507 Commits

Author SHA1 Message Date
sre-ci-robot
e937197270
[automated] Update gpu Builder image changes (#35300)
Update gpu Builder image changes
See changes:
d8668fe405
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-08-06 17:38:16 +08:00
Cai Yudong
3c9a47c8db
feat: Encode traceID and spanID as hex string (#34807)
Issue: https://github.com/zilliztech/knowhere/pull/714

Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
2024-08-06 15:20:16 +08:00
Jiquan Long
91df03afe8
feat: put inverted index into ram (#35222)
fix: https://github.com/milvus-io/milvus/issues/35224

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-06 11:54:16 +08:00
yihao.dai
678018d9ca
enhance: Avoid unnecessary compaction (#35148)
Estimate the import segment size based on DiskSegmentMaxSize(2G) to
avoid unnecessary compaction after import completed.

issue: https://github.com/milvus-io/milvus/issues/35147

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-06 10:30:21 +08:00
presburger
a9352a073b
enhance: change gpu default mem pool size (#34159)
Signed-off-by: yusheng.ma <yusheng.ma@zilliz.com>
2024-08-06 10:24:15 +08:00
sre-ci-robot
dea73ef288
[automated] Update cpu Builder image changes (#35283)
Update cpu Builder image changes
See changes:
d8668fe405
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-08-06 10:16:16 +08:00
chyezh
14051fed7d
enhance: streaming service client (#34656)
issue: #33285

- implement streaming service client.
- implement producing and consuming service client by streaming coord
client and streaming node client.

Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-05 21:38:15 +08:00
congqixia
a2b517523d
enhance: Add in-memory cache for casbin enforcer result (#35271)
See also #35270

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-05 18:48:15 +08:00
ThreadDao
a62118ca33
test: update cases for index describe and upsert with autoID (#35191)
issue: #33419

---------

Signed-off-by: ThreadDao <yufen.zong@zilliz.com>
2024-08-05 18:00:16 +08:00
shaoyue
1db099973f
enhance: Gin & restfulv1 handler use sonic json (#35020)
/cc @czs007 @congqixia 
/kind enhancement

Signed-off-by: haorenfsa <haorenfsa@gmail.com>
2024-08-05 16:44:17 +08:00
yihao.dai
b71e058bc5
enhance: Add import option to skip disk quota check (#35274)
Add an option to skip the disk quota check for backup-restore import.

issue: https://github.com/milvus-io/milvus/issues/33775

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-05 16:40:16 +08:00
cai.zhang
6542c1ab0e
enhance: Add monitoring metrics for task execution time in datacoord (#35139)
issue: #35138

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-05 16:26:17 +08:00
cai.zhang
d905713724
fix: Maintain load idempotency even when building new indexes (#35178)
issue: #34404

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-05 16:22:17 +08:00
sthuang
d8668fe405
enhance: upgrade go version to 1.21.11 (#35257)
issue: #35142

Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>
2024-08-05 16:12:15 +08:00
yihao.dai
678c0a708b
enhance: Refine import error log (#35067)
issue: https://github.com/milvus-io/milvus/issues/35060

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-05 15:30:16 +08:00
congqixia
a6903701a4
enhance: Remove stackerror imported unexpected (#35256)
`stackerror` pkg is imported by accident and could be replaced by
cockroachdb errors lib. This PR removes import line and fix this
problem.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-05 14:48:16 +08:00
wei liu
c22b0a636e
fix: delete by expr failed at retry progress (#35241)
issue: #35240
delete by expr shard the same err object between channels, so if one
channel's query failed, it will fail all channel, which will break
channel level retry policy, and make delete operation failed.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-05 11:18:16 +08:00
zhagnlu
4b553b0333
enhance: revert remove duplicated pk function (#35103)
issue: #34778
 Revert "fix: fix query count(*) concurrently"
 Revert "enhance: mark duplicated pk as deleted "

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-05 10:48:17 +08:00
sre-ci-robot
e6640346d0
[automated] Update gpu Builder image changes (#35246)
Update gpu Builder image changes
See changes:
7bf68d8657
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-08-05 09:26:15 +08:00
sre-ci-robot
eb5461905b
[automated] Update Pytest image changes (#35244)
Update Pytest image changes
See changes:
241c71fdde
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-08-05 09:24:33 +08:00
sre-ci-robot
e201110fbd
[automated] Update cpu Builder image changes (#35248)
Update cpu Builder image changes
See changes:
7bf68d8657
Signed-off-by: sre-ci-robot sre-ci-robot@users.noreply.github.com

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
2024-08-05 09:22:21 +08:00
chyezh
9871966415
enhance: segment alloc interceptor (#34996)
#33285

- add segment alloc interceptor for streamingnode.
- add add manual alloc segment rpc for datacoord.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
2024-08-04 07:40:15 +08:00
sre-ci-robot
a7f7d911cf Update all contributors
Signed-off-by: sre-ci-robot <sre-ci-robot@zilliz.com>
2024-08-02 12:00:48 +00:00
Chun Han
3faef63a25
enhance: add log for partition stats( #30376) (#35219)
related:  #30376

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-08-02 19:34:22 +08:00
zhagnlu
16dd53e7cf
enhance: remove timestamp_filter after retrieve (#35207)
#35226

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-02 19:32:46 +08:00
yellow-shine
241c71fdde
enhance: use docker compose instead of docker-compose (#35208)
https://github.com/milvus-io/milvus/issues/35209

---------

Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-08-02 19:32:32 +08:00
cqy123456
b99a554827
enhance:update mmap_chunk_manager config in milvus.yaml (#35180)
issue: https://github.com/milvus-io/milvus/issues/32984
related pr: https://github.com/milvus-io/milvus/pull/35186

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
2024-08-02 19:28:15 +08:00
yellow-shine
7bf68d8657
enhance: upgrade conan version (#35215)
Signed-off-by: Yellow Shine <sammy.huang@zilliz.com>
2024-08-02 19:22:15 +08:00
yihao.dai
a4439cc911
enhance: Implement flusher in streamingNode (#34942)
- Implement flusher to:
  - Manage the pipelines (creation, deletion, etc.)
  - Manage the segment write buffer
  - Manage sync operation (including receive flushMsg and execute flush)
- Add a new `GetChannelRecoveryInfo` RPC in DataCoord.
- Reorganize packages: `flushcommon` and `datanode`.

issue: https://github.com/milvus-io/milvus/issues/33285

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-02 18:30:23 +08:00
jaime
fcec4c21b9
fix: check collection health(queryable) fail for releasing collection (#34947)
issue: #34946

Signed-off-by: jaime <yun.zhang@zilliz.com>
2024-08-02 17:20:15 +08:00
Bingyi Sun
3641ae6611
fix: Fix index memory estimation (#35225)
issue: https://github.com/milvus-io/milvus/issues/35229

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2024-08-02 16:24:15 +08:00
congqixia
db06b86594
enhance: Sync otlp dependency version to fix security issue (#35192)
Related to #34434

otelgrpc DoS vulnerability due to unbound cardinality metrics

https://github.com/milvus-io/milvus/security/dependabot/91

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-02 16:22:20 +08:00
wayblink
81773bfadf
enhance: add commit time in partitionStats proto (#35125)
fix: #35110

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-02 16:16:14 +08:00
smellthemoon
475c333fa2
enhance: add valid_data in span (#35030)
#31728

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-02 15:40:14 +08:00
wei liu
f466129924
enhance: add segment state on data coord bin log size metrics (#35153)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-02 14:48:14 +08:00
wei liu
3dd3749f0b
enhance: Avoid unnecesary syncTargetVersion func call after querycoord recover (#34954)
before querycoord stop gracefully, we will save the current target to
meta store and recover it after querycoord start up, to speed the
querycoord's recovery time. but the target version hasn't been recovered
as expected, and it use latest timestamp as current target's version,
which has no effect to querycoord but an unnecessary syncTargetVersion
func call.

This PR recover the correct target version to avoid unnecessary
syncTargetVersion func call

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-02 14:46:13 +08:00
congqixia
095d77269b
enhance: Set WatchProgress for channel operation progress response (#35183)
The watch progress is always zero in CheckChannelOperationProgress
response, which is meaningless and confusing. This PR set progress value
in rpc response to fix this problem.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-02 12:06:14 +08:00
aoiasd
3655ab10b2
fix: evict paramtable cache miss (#34771)
relate: https://github.com/milvus-io/milvus/issues/33461

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-08-02 11:52:14 +08:00
congqixia
c64a078458
enhance: Support proxy/delegator qn client pooling (#35194)
See also #35196

Add param item for proxy/delegator query node client pooling and
implement pooling logic

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-02 11:24:19 +08:00
wei liu
3b735b4b02
enhance: Refine param init for MmapDirPath (#35181)
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-02 11:12:14 +08:00
wayblink
95462668ca
enhance: unify time in clustering compaction task to unix (#35167)
#34495

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-02 10:30:19 +08:00
wei liu
27b6d58981
fix: Set legacy level to l0 segment after qc restart (#35197)
issue: #35087
after qc restarts, and target is not ready yet, if dist_handler try to
update segment dist, it will set legacy level to l0 segment, which may
cause l0 segment be moved to other node, cause search/query failed.

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-02 10:18:13 +08:00
Jiquan Long
4dbd023402
enhance: add metric to record maxInsertRate and queryNodeMemoryHighWaterLevel (#35185)
fix: https://github.com/milvus-io/milvus/issues/35177

---------

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2024-08-02 10:14:13 +08:00
cai.zhang
196a7986b3
enhance: Change the fixed value to a ratio for clustering segment size (#35076)
issue: #34495

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-01 22:04:14 +08:00
congqixia
a642a26ed4
enhance: Resolve ChunkFileWriter lint issue (#35166)
See also #34483

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-01 16:52:13 +08:00
yihao.dai
7721a28584
fix: Fix unstable TestDispatchToVchannels ut (#35163)
issue: https://github.com/milvus-io/milvus/issues/35150

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-01 14:56:13 +08:00
wayblink
5bbb1c201c
enhance:support l2 single compaction (#34935)
#34928

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-01 14:36:13 +08:00
zhagnlu
f8c1b138a8
fix:fix get array error for int type (#35154)
#35055

Signed-off-by: luzhang <luzhang@zilliz.com>
Co-authored-by: luzhang <luzhang@zilliz.com>
2024-08-01 14:30:12 +08:00
Ted Xu
b4d0f4df0a
enhance: update param documents (#35152)
See: #35161

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-08-01 12:38:13 +08:00
zhenshan.cao
aa247f192d
enhance: remove unused code for StorageV2 (#35132)
issue: https://github.com/milvus-io/milvus/issues/34168

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-08-01 12:08:13 +08:00