Commit Graph

111 Commits

Author SHA1 Message Date
yah01
cc952e0486
enhance: optimize forwarding level0 deletions by respecting partition (#28456)
- Cache the level 0 deletions after loading level0 segments
- Divide the level 0 deletions by partition
related: #27349

---------

Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-21 18:24:22 +08:00
yah01
ece592a42f
Deliver L0 segments delete records (#27722)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-11-07 01:44:18 +08:00
XuanYang-cn
2f16339aac
Enhance InsertData and FieldData (#27436)
1. Add NewInsertData
2. Add GetRowNum(), GetMemorySize(), and, Append() for InsertData
3. Add AppendRow() for FieldData for compaction

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2023-10-17 17:36:11 +08:00
SimFG
26f06dd732
Format the code (#27275)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
Xu Tong
9166011c4a
Add float16 vector (#25852)
Signed-off-by: Writer-X <1256866856@qq.com>
2023-09-08 10:03:16 +08:00
congqixia
2770ac4df5
Fix nilness linter errors (#26218)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-08-09 11:31:15 +08:00
xige-16
94d6cbb238
Fix querynode panic when binlog ts wrong (#25635)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2023-07-18 10:41:20 +08:00
PowderLi
3f4356df10
fix the spelling of field (#25008)
Signed-off-by: PowderLi <min.li@zilliz.com>
2023-06-21 14:00:42 +08:00
congqixia
41af0a98fa
Use go-api/v2 for milvus-proto (#24770)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
Enwei Jiao
d3af451d92
Upgrade golangci-lint (#24707)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-06-07 19:34:36 +08:00
aoiasd
c84bdcea49
merge stats log when segment flushing or compacting (#23570)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2023-05-29 10:21:28 +08:00
Enwei Jiao
967a97b9bd
Support json & array types (#23408)
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
2023-04-20 11:32:31 +08:00
jaime
c9d0c157ec
Move some modules from internal to public package (#22572)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
yah01
081572d31c
Refactor QueryNode (#21625)
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: aoiasd <zhicheng.yue@zilliz.com>
2023-03-27 00:42:00 +08:00
Xiaofan
949d5d078f
Fix memory calculation in dataCodec (#21800)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-01-28 11:09:52 +08:00
congqixia
f745d7f489
Fix compaction target segment rowNum is always 0 (#20937)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-12-01 20:33:17 +08:00
Xiaofan
633a749880
Recude IndexCodec Load Memory (#20621)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-11-18 10:47:08 +08:00
Xiaofan
2bfecf5b4e
Refine bloomfilter and memory usage (#20168)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-31 17:41:34 +08:00
SimFG
a55f739608
Separate public proto files (#19782)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
SimFG
d7f38a803d
Separate some proto files (#19218)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
xige-16
4de1bfe5bc
Add cpp data codec (#18538)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
Co-authored-by: zhagnlu lu.zhang@zilliz.com

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-09-09 22:12:34 +08:00
xige-16
e40061b864
Update binlog event format (#18347)
Signed-off-by: xige-16 <xi.ge@zilliz.com>

Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-08-11 14:06:38 +08:00
yah01
70f8bea4b4
Avoid growing slice as deserializing binlogs (#17421)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-06-08 11:46:06 +08:00
yah01
7af02fa531
Improve load performance, load binlogs concurrently per file, deserialize binlogs concurrently per field/segment (#16514)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-04-25 15:57:47 +08:00
godchen
bb7a0766fe
Add dependency factory (#16204)
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-04-07 22:05:32 +08:00
xige-16
99984b88e1
Support delete varChar value (#16229)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-04-02 17:43:29 +08:00
Jiquan Long
ba37531456
Add support for loading multiple indexes (#16138)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2022-03-30 21:11:28 +08:00
Xiaofan
801eeffbcc
Replace cgo parquet reader to go parquet reader (#16199)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-03-30 15:21:28 +08:00
xige-16
205c92e54b
Support insert string data (#15993)
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2022-03-25 14:27:25 +08:00
godchen
384ceea223
Close payload writer when error occurs (#15013)
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2022-01-07 18:27:23 +08:00
congqixia
b22e945513
[skip e2e] Add comment for NewInsertCodec (#14907)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-01-06 18:04:03 +08:00
shaoyue
f9dc87ba6c
[skip e2e] Add comment for data_codec (#14571)
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2021-12-30 10:44:05 +08:00
congqixia
20cc6843a4
[skip e2e] Add comment for FieldData interface (#14351)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-12-28 09:34:15 +08:00
jaime
28dfccb6ea
[skip e2e] Fix golint in data_codec (#14212)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: yun.zhang <yun.zhang@zilliz.com>
2021-12-24 19:56:28 +08:00
shaoyue
e0a2f1ce82
[skip e2e] Add comment for IndexCodec (#14139)
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2021-12-24 13:31:30 +08:00
jaime
0114708c95
[skip e2e] Fix grammar mistake in data_codec (#14108)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: yun.zhang <yun.zhang@zilliz.com>
2021-12-23 22:05:51 +08:00
shaoyue
d4f9365a3d
[skip e2e] Add comment for internal/storage/data_codec.go (#14030)
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2021-12-23 14:30:34 +08:00
XuanYang-cn
bccf65ec67
[skip e2e]Update license for storage datacodec (#14039)
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-12-23 12:01:36 +08:00
shaoyue
ec91866a5c
[skip e2e] Add comment for internal/storage.Int8FieldData (#13992)
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2021-12-22 19:31:51 +08:00
JackLCL
b79d751977
[skip ci]Fix error log output format in data_codec.go (#13382)
Signed-off-by: JackLCL <chenglong.li@zilliz.com>
2021-12-22 09:59:04 +08:00
jaime
99737d30e2
[skip e2e] Fix golint in data_codec (#13804)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Co-authored-by: yun.zhang <yun.zhang@zilliz.com>
2021-12-20 22:47:07 +08:00
Xieql
35ab21372b
[skip e2e] Improve annotation (#13722)
Signed-off-by: Xieql <727092844@qq.com>
2021-12-20 19:08:38 +08:00
godchen
7e56f08747
Add payload bytes interface. (#13467)
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
2021-12-16 16:35:42 +08:00
shaoyue
0e24520192
[skip ci] Add comment for index codec (#13464)
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2021-12-15 21:13:23 +08:00
JackLCL
5b9282fdc3
[skip ci]Fix error log output format in data_ciodec.go (#13226)
Signed-off-by: JackLCL <chenglong.li@zilliz.com>
2021-12-13 13:22:14 +08:00
godchen
febdda90f4
Change binlog writer close behavior (#13046)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-12-09 12:37:06 +08:00
godchen
5c44ee8c0f
Close binlog event when error occur (#12995)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-12-08 21:11:39 +08:00
shaoyue
490f81b932
[skip ci] Add comment for DataDefinitionCodec (#12616)
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2021-12-02 18:57:32 +08:00
shaoyue
ee89ca83e1
[skip ci] Add comment for internal/storage/data_codec.go (#12520)
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2021-12-01 15:39:57 +08:00
XuanYang-cn
48b45d82e5
Add ut for binlog_io to 100 coverage (#12283)
Make DN ut coverage upto 90%
Resolves: #8058

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-11-26 17:43:17 +08:00