Commit Graph

65 Commits

Author SHA1 Message Date
congqixia
41af0a98fa
Use go-api/v2 for milvus-proto (#24770)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
yah01
ebd0279d3f
Check error by Error() and NoError() for better report message (#24736)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-08 15:36:36 +08:00
congqixia
6375236533
Make SegmentInfo.size atomic (#24457)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-29 11:31:28 +08:00
yihao.dai
ed8836cd15
Add disk quota at the collection level (#23704)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2023-04-26 21:52:36 +08:00
jaime
c9d0c157ec
Move some modules from internal to public package (#22572)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
SimFG
4a90490a67
Fix the segment not found error (#22772)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-03-17 17:27:56 +08:00
congqixia
732986aa04
Remove fmt.Print from internal package (#22722)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-03-14 17:36:05 +08:00
jaime
d126f06946
Decouple mq module from internal proto definition (#22536)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-03-04 23:21:50 +08:00
yah01
319f1773af
Fix segment not found if forward delete to empty segment (#22528)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-03-03 14:13:49 +08:00
Xiaofan
3d6bded115
Fix Row Number Mismatch (#22307)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-02-27 10:41:46 +08:00
Enwei Jiao
697dedac7e
Use cockroachdb/errors to replace other error pkg (#22390)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-26 11:31:49 +08:00
Jiquan Long
9fd9bed2b9
Remove collection meta after GC finished (#21595)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-01-12 09:55:42 +08:00
jaime
73d8509600
Improve ListSegments performance for datacoord catalog (#21477)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-01-06 14:33:36 +08:00
cai.zhang
e5f408dceb
Merge IndexCoord and DataCoord (#21267)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-04 19:37:36 +08:00
Enwei Jiao
89b810a4db
Refactor all params into ParamItem (#20987)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-07 18:01:19 +08:00
SimFG
6d9d24b4ca
Make memory and etcd data as consistent as possible (#20683)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-11-24 10:19:12 +08:00
Ten Thousand Leaves
a8c7199a52
Fix segment_num, stored_rows_num and stored_rows_count metrics (#20704)
TODO: better track segment with state other than flushed and dropped.

/kind bug

issue: #20146, #20145
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-11-22 19:21:13 +08:00
SimFG
dffcd974ff
Make sure that the segment path matches the segment id (#20677)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-11-18 15:35:09 +08:00
Bingyi Sun
2390095232
Fix load uses compacted segments' binlogs (#20655)
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-11-17 20:37:10 +08:00
bigsheeper
cd19d99ad7
Add channel level checkpoint (#20350)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-11-10 22:13:04 +08:00
XuanYang-cn
22fb1d3b93
Remove dup mock_kv.go and mock_txn_kv.go (#20267)
Replace them with mockery generated mocks in kv/mocks

See also: #19289

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-11-03 14:41:35 +08:00
Ten Thousand Leaves
5708352a80
Improve DataCoord and DataNode logs (#20265)
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-11-03 12:15:35 +08:00
Xiaofan
2bfecf5b4e
Refine bloomfilter and memory usage (#20168)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-31 17:41:34 +08:00
Xiaofan
11efa0bb5f
Fix compacted segment not dropped (#20128)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-27 17:15:32 +08:00
Xiaofan
def5972e01
fix duplicated handoff request (#20108)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-27 15:15:32 +08:00
jaime
d8ca56e084
Fix empty segment handoff fails (#20073)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-26 16:49:31 +08:00
SimFG
a55f739608
Separate public proto files (#19782)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
yah01
89c9cb3680
Fix scan may break GC limitation (#19670)
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-10-15 15:13:24 +08:00
bigsheeper
bdaddd7b09
Revert DML quota check (#19790)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-10-14 17:17:23 +08:00
bigsheeper
22b32465e7
Make bulkLoad failed if quota exceeds and remove autoindex config (#19770)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-10-14 14:27:25 +08:00
bigsheeper
e6c76aba0e
Force deny writing if disk quota exceeded (#19680)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-10-12 10:03:22 +08:00
jaime
31db01b6ae
Support TTL for collection level (#19383)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-10 20:31:22 +08:00
XuanYang-cn
a0ce533727
Add a SyncSegments to sync meta between DN and DC (#19298)
See also: #19072

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-09-27 16:02:53 +08:00
Ten Thousand Leaves
b30c9d4f60
Support Bulk Load in Milvus 2.2 (#18982)
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-09-26 18:06:54 +08:00
jaime
91df8f2d6a
Use log id instead of log path of the binlog in metastore (#19123)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-09-25 15:56:51 +08:00
SimFG
d7f38a803d
Separate some proto files (#19218)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
jaime
362c663b22
Remove deprecated inner compaction type (#19024)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-09-07 19:16:34 +08:00
cai.zhang
c924f73105
Refactor for IndexCoord to support cloud (#18643)
Co-authored-by: Zach41 <zongmei.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Zach41 <zongmei.zhang@zilliz.com>
2022-08-25 15:48:54 +08:00
codeman
70254c3ec5
Unified catalog interface for segment (#18289) (#18290)
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Signed-off-by: kejiang <ke.jiang@zilliz.com>
Co-authored-by: kejiang <ke.jiang@zilliz.com>
2022-08-20 10:24:51 +08:00
yah01
ef6859a4d6
Refactor handoff handler (#17806)
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Co-authored-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-06-29 23:26:19 +08:00
XuanYang-cn
ff3c4ab2a8
Fix compaction panic (#17840)
See also: #17823

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-06-29 16:54:17 +08:00
cai.zhang
ed94ecf847
Segments can't be compacted when they have reference lock (#17649)
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
2022-06-20 21:56:12 +08:00
wayblink
074ec3060a
Support return dropped segments info in GetSegmentInfo rpc (#17617)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-06-17 18:24:12 +08:00
Bingyi Sun
d4fdfab818
Fix VDC-510 (#17554)
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2022-06-15 19:58:10 +08:00
cai.zhang
bcf3b7426a
Add distributed lock for segment refgerence (#16782)
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
2022-05-31 16:36:03 +08:00
Ten Thousand Leaves
289e468a7a
Make bulk load fully work (#16512)
issue: #15604

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-04-20 14:03:40 +08:00
yah01
0cfb6a85be
Make compaction not generate empty segment (#15707)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2022-02-28 10:17:54 +08:00
congqixia
1dcd06cc78
Add log file info for all kind of segment binlogs (#13606)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-19 20:00:42 +08:00
congqixia
06ced4d1d1
Add DropVirtualChannel for DataCoord (#12361)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-11-29 22:35:41 +08:00
Bingyi Sun
54b40da4c5
Add segment drop logic (#11621)
issue: #11558
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
2021-11-12 00:22:42 +08:00