Commit Graph

87 Commits

Author SHA1 Message Date
smellthemoon
1c1f2a1371
enhance:change some logs (#29579)
related #29588

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-01-05 16:12:48 +08:00
cai.zhang
c45f8a2946
fix: Import data from parquet file in streaming way (#29514)
issue: #29292

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-12-27 15:30:46 +08:00
wayblink
2274aa3b50
fix: bulkinsert binlog didn't consider ts order when processing delta data (#29163)
#29162

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-12-14 14:36:40 +08:00
cai.zhang
49b8657f95
enhance: Support implicit type conversion for parquet (#29046)
issue: #29019

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-12-12 16:14:44 +08:00
cai.zhang
2b05460ef9
enhance: Make import-related error message clearer (#28978)
issue: #28976

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-12-08 10:12:38 +08:00
cai.zhang
f5f4f0872e
enhance: Support importing data with parquet file (#28608)
issue: #28272

Numpy does not support array type import. 
Array type data is imported through parquet.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-29 20:52:27 +08:00
cai.zhang
1b7a503f89
enhance: Revert import support csv format (#28760)
Revert import support csv format.
issue: #28778

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-28 14:32:27 +08:00
cai.zhang
c29b60e18e
enhance: Support Array DataType for bulk_insert (#28341)
issue: #28272 
Support array DataType for bulk_insert with json, binlog files.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2023-11-27 13:50:27 +08:00
Bingyi Sun
d7145e2c06
enhance: Update golangci_lint version (#28535)
Update golangci lint and fix some warnings

Signed-off-by: sunby <sunbingyi1992@gmail.com>
2023-11-21 10:04:21 +08:00
groot
3f6b203018
Fix bulkinsert bug that segments are compacted after import (#28192)
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2023-11-07 15:14:26 +08:00
wayblink
00ae019ff0
Use go 1.20 csv_reader to keep milvus go=1.18 limitation (#28080)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-11-03 10:40:16 +08:00
groot
abd5b199cc
Bulkinsert support pure list json (#27990)
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2023-11-01 19:02:13 +08:00
KumaJie
e88212ba4b
Add CSV file import function (#27149)
Signed-off-by: kuma <675613722@qq.com>
Co-authored-by: kuma <675613722@qq.com>
2023-10-31 22:47:23 +08:00
congqixia
2f201c25e2
Remove deprecated io/ioutil usage (#27747)
`io/ioutil` package is deprecated, use `io`,`os` package replacement
also added golangci-lint rule to block future reference

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
Co-authored-by: guoguangwu <guoguangwu@magic-shield.com>
2023-10-17 20:32:09 +08:00
yah01
be980fbc38
Refine state check (#27541)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-10-11 21:01:35 +08:00
wayblink
7dd0be1b2c
Enable bulkinsert binlog data with partitionkey (#27241)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-09-26 10:25:25 +08:00
jaime
7f7c71ea7d
Decoupling client and server API in types interface (#27186)
Co-authored-by:: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-09-26 09:57:25 +08:00
SimFG
26f06dd732
Format the code (#27275)
Signed-off-by: SimFG <bang.fu@zilliz.com>
2023-09-21 09:45:27 +08:00
yah01
b4f86ea55e
Construct all success status with merr (#27226)
Signed-off-by: yah01 <yah2er0ne@outlook.com>
2023-09-20 10:57:23 +08:00
congqixia
cc9974979f
Add staticcheck linter and fix existing problems (#27174)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-09-19 10:05:22 +08:00
Xu Tong
9166011c4a
Add float16 vector (#25852)
Signed-off-by: Writer-X <1256866856@qq.com>
2023-09-08 10:03:16 +08:00
Enwei Jiao
fb0705df1b
Decouple basetable and componentparam (#26725)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-09-05 10:31:48 +08:00
groot
1b1bafaff1
Bulkinsert read varchar by batch (#26199)
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2023-08-30 14:23:06 +08:00
groot
c2693ea284
Fix a crash bug of bulkinsert for dynamic field (#25982)
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2023-07-28 15:25:03 +08:00
groot
a6808e6484
Fix bulkinsert row count error (#25869)
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2023-07-25 11:29:00 +08:00
groot
96c987ed62
Bulkinsert supports partition keys (#25284)
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2023-07-11 15:18:28 +08:00
congqixia
41af0a98fa
Use go-api/v2 for milvus-proto (#24770)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-06-09 01:28:37 +08:00
yah01
ebd0279d3f
Check error by Error() and NoError() for better report message (#24736)
Signed-off-by: yah01 <yang.cen@zilliz.com>
2023-06-08 15:36:36 +08:00
groot
3022e37298
Fix bug of bulkinsert for dynamic field (#24569)
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2023-06-01 14:14:31 +08:00
groot
5358da2a10
Fix a crash bug for backup/restore (#24529)
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2023-05-31 14:45:30 +08:00
groot
0d0e97ab7f
Fix crash issue of bulkinsert by invalid numpy array file (#24480) (#24528)
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2023-05-31 14:43:39 +08:00
groot
3cb75997a3
Fix a bug of bulkinsert dynamic schema (#24405) (#24527)
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2023-05-31 14:41:41 +08:00
groot
97cd0409e4
Support dynamic field for bulkinsert (#24265) (#24304)
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2023-05-23 14:17:25 +08:00
MrPresent-Han
7744573d3d
support parms for import maxfilesize(#24191) (#24192)
Signed-off-by: MrPresent-Han <chun.han@zilliz.com>
2023-05-18 18:59:27 +08:00
groot
b89f06dd19
Support JSON for bulkinsert (#24130)
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2023-05-18 16:33:23 +08:00
congqixia
73a181d226
Fix get vector it timeout and improve some string const usage (#24141)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-05-16 17:41:22 +08:00
jaime
c9d0c157ec
Move some modules from internal to public package (#22572)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-04-06 19:14:32 +08:00
groot
6f6bd98c27
Verify vector float data for bulkinsert and insert (#22728)
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2023-03-14 14:03:58 +08:00
Enwei Jiao
697dedac7e
Use cockroachdb/errors to replace other error pkg (#22390)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2023-02-26 11:31:49 +08:00
wayblink
b7c0d12d82
Skip file with prefix '.' may exist in filesystem (#22138)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-02-21 10:54:29 +08:00
congqixia
f2575e5fa8
Add unconvert & durationcheck linters and fix issues (#22161)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2023-02-15 17:22:34 +08:00
groot
c21e11eb8a
Fix datanode crash bug for bulkinsert (#22041)
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2023-02-08 14:06:31 +08:00
Xiaofan
949d5d078f
Fix memory calculation in dataCodec (#21800)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2023-01-28 11:09:52 +08:00
groot
c0e7a122d4
Fix bulkinsert progress bug (#21680)
Signed-off-by: groot <yihua.mo@zilliz.com>
2023-01-13 16:25:40 +08:00
groot
9f8095996b
Report bulkinsert progress (#21612) (#21638)
Signed-off-by: groot <yihua.mo@zilliz.com>
2023-01-11 17:37:44 +08:00
groot
aab20a4031
Streaming read numpy (#21579)
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2023-01-10 14:53:38 +08:00
Ten Thousand Leaves
b3da554705
Increase bulk insert file size limit (#21526)
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2023-01-06 13:09:36 +08:00
groot
33a102c6fc
Refine bulkinsert (#20986)
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-12-07 14:53:18 +08:00
groot
bf2107ecf5
Fix a regression of local storage (#20653)
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-11-21 10:19:10 +08:00
groot
8da44638ac
Refine json parser for import (#20545)
Signed-off-by: yhmo <yihua.mo@zilliz.com>

Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-11-16 19:05:08 +08:00