test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-04 04:49:08 +08:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity
20,277 Commits 96 Branches 105 Tags 185 MiB
48898cefea
Commit Graph

64 Commits

Author SHA1 Message Date
jaime
a08a0c831f
fix: encountering orphan channel-cp meta after DataCoord GC (#34612) 
issue: #34545

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-11 23:01:35 +08:00
wayblink
a1232fafda
feat: Major compaction (#33620) 
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
 2024-06-10 21:34:08 +08:00
cai.zhang
27cc9f2630
enhance: Support analyze data (#33651) 
issue: #30633

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
Co-authored-by: chasingegg <chao.gao@zilliz.com>
 2024-06-06 17:37:51 +08:00
congqixia
cedb33ceec
enhance: Improve datacoord segment filtering with collection (#32831) 
See also #32165

This PR modify the `SelectSegments` interface to utilizing collection id
information when selecting segment with provided collection

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-05-08 21:37:29 +08:00
Bingyi Sun
fecd9c21ba
feat: LRU cache implementation (#32567) 
issue: https://github.com/milvus-io/milvus/issues/32783
This pr is the implementation of lru cache on branch lru-dev.

Signed-off-by: sunby <sunbingyi1992@gmail.com>
Co-authored-by: chyezh <chyezh@outlook.com>
Co-authored-by: MrPresent-Han <chun.han@zilliz.com>
Co-authored-by: Ted Xu <ted.xu@zilliz.com>
Co-authored-by: jaime <yun.zhang@zilliz.com>
Co-authored-by: wayblink <anyang.wang@zilliz.com>
 2024-05-06 20:29:30 +08:00
chyezh
77f4f0c5a8
fix: gc only do once (#32722) 
issue: #29655

produced by pr: #31740

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-04-30 14:19:27 +08:00
chyezh
2586c2f1b3
enhance: use WalkWithPrefix api for oss, enable piplined file gc (#31740) 
issue: #19095,#29655,#31718

- Change `ListWithPrefix` to `WalkWithPrefix` of OOS into a pipeline
mode.

- File garbage collection is performed in other goroutine.

- Segment Index Recycle clean index file too.

---------

Signed-off-by: chyezh <chyezh@outlook.com>
 2024-04-25 20:41:27 +08:00
congqixia
3f7c77481d
fix: Use correct ticker for gc work defer (#31456) 
See also #31362

Fix the second defer using same ticker var.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-03-20 22:17:06 +08:00
congqixia
16c661c722
enhance: Use different interval for gc scan (#31363) 
See also #31362

This PR make datacoord garbage collection scan operation using differet
interval than other opeartion.

This interval is a newly added param item, which default value is 7*24
hours.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-03-19 11:27:06 +08:00
jaime
4b0c3dd377
enhance: index meta use independent rather than global meta lock (#30869) 
issue: https://github.com/milvus-io/milvus/issues/30837

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-03-04 16:56:59 +08:00
wayblink
1635211c3f
enhance: Add log when garbage collection resumed (#30535) 
/kind enhancement

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-02-05 17:09:15 +08:00
smellthemoon
e52ce370b6
enhance:don't store logPath in meta to reduce memory (#28873) 
don't store logPath in meta to reduce memory, when service get
segmentinfo, generate logpath from logid.
#28885

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-01-18 22:06:31 +08:00
SimFG
d23f87a393
enhance: Add concurrency for datacoord segment GC (#29561) 
issue: https://github.com/milvus-io/milvus/issues/29553
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-01-03 13:16:57 +08:00
congqixia
8a63e53421
enhance: Add http method to control datacoord garbage collection (#29052) 
See also #29051

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-12-14 19:26:39 +08:00
congqixia
1d76565894
Add metrics for garbage collection (#27303) 
Also fix second metrics usage in compaction

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-22 18:47:25 +08:00
SimFG
26f06dd732
Format the code (#27275) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-21 09:45:27 +08:00
congqixia
19e4deb792
Make garbage collector fetch meta after list from storage (#27203) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-09-19 17:05:34 +08:00
PowderLi
4feb3fa7c6
support azure (#26398) 
Signed-off-by: PowderLi <min.li@zilliz.com>
 2023-09-19 10:01:23 +08:00
yah01
168e82ee10
Fix panic while handling with the nil status (#27040) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2023-09-15 10:09:21 +08:00
congqixia
f8bcf60e55
Adjust some confusing Warning log to INFO (#26356) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-08-15 17:49:33 +08:00
zhenshan.cao
7619db1edc
Fast GC compacted dropped segment (#26207) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-08-09 19:39:15 +08:00
Zhihong Yu
150947a5f4
Conditionally log warning when DropSegment fails (#25774) 
Signed-off-by: Ted Yu <yuzhihong@gmail.com>
 2023-07-25 10:39:01 +08:00
cai.zhang
b15e34db21
Add contraint for compaction based indexed segments (#25709) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-07-23 21:31:00 +08:00
Enwei Jiao
66fdc71479
Refactor logs in DataCoord & DataNode (#25574) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-07-14 15:56:31 +08:00
Xiaofan
dbf0130803
Faster garbage collect on compacted data (#25088) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-07-12 10:14:29 +08:00
congqixia
41af0a98fa
Use go-api/v2 for milvus-proto (#24770) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-09 01:28:37 +08:00
zhenshan.cao
dc5abe086c
Revert "Restrain gc files by collection (#22504)" (#23380) 
This reverts commit fa86de530d.

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-04-12 19:38:28 +08:00
cai.zhang
9288020da3
Fix bug for batch delete files on gcp of minio (#23052) (#23083) (#23090) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-04-09 16:22:30 +08:00
jaime
c9d0c157ec
Move some modules from internal to public package (#22572) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-04-06 19:14:32 +08:00
wayblink
aa6212d5b0
[Cherry-pick] Fix channel checkpoint issues and fix GC (#22559) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
Co-authored-by: Ten Thousand Leaves <69466447+soothing-rain@users.noreply.github.com>
 2023-03-09 14:13:52 +08:00
yah01
319f1773af
Fix segment not found if forward delete to empty segment (#22528) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-03-03 14:13:49 +08:00
congqixia
fa86de530d
Restrain gc files by collection (#22504) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-03-02 11:49:47 +08:00
zhenshan.cao
e768437681
Correct usage of Timer and Ticker (#22228) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-02-23 18:59:45 +08:00
congqixia
787ce7c121
Add Rated group for checkpoint gc skip case (#21949) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-02-03 14:23:56 +08:00
congqixia
42ab1b0a48
Add segment gc restraint of dml position before channel cp (#21564) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-01-06 21:33:36 +08:00
cai.zhang
aa203acfb3
Low IndexCoord weight (#21548) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-06 14:21:37 +08:00
cai.zhang
e5f408dceb
Merge IndexCoord and DataCoord (#21267) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-04 19:37:36 +08:00
SimFG
a55f739608
Separate public proto files (#19782) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-10-16 20:49:27 +08:00
yah01
89c9cb3680
Fix scan may break GC limitation (#19670) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-10-15 15:13:24 +08:00
jaime
ab88dd77e7
Remove unnecessary log for datacoord gc (#19698) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-10-13 14:43:23 +08:00
congqixia
2a2d32c4fc
Fix datacoord panics when collection info not found (#19706) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-10-11 21:39:24 +08:00
jaime
1919353f02
Fix parse segment id error from delta log (#19566) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-09-30 14:18:55 +08:00
congqixia
838a633584
Add ctx parameter for ChunkManager methods (#19546) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-09-29 16:18:56 +08:00
Ten Thousand Leaves
b30c9d4f60
Support Bulk Load in Milvus 2.2 (#18982) 
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-09-26 18:06:54 +08:00
Xiaofan
928a213e31
Fix Chunk Manager Path mis use (#19141) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-09-23 14:40:51 +08:00
SimFG
d7f38a803d
Separate some proto files (#19218) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-09-16 16:56:49 +08:00
yah01
11b352c3ae
Guarantee flushed segments are all indexed (#19062) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-09-16 11:32:48 +08:00
jaime
efa5dfaa7b
Fix Parse binlog path failure (#18583) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-08-10 17:04:37 +08:00
xing.zhao
d90313a1a8
Use chunkManager rather than minio.Client in datanode garbage collection (#18092) 
Signed-off-by: xingzhao <xing.zhao@zilliz.com>

Co-authored-by: xingzhao <xing.zhao@zilliz.com>
 2022-07-22 22:10:28 +08:00
Xiaofan
6d82ef8c20
fix garbage collector err handling (#18277) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-07-15 13:54:26 +08:00