7 Commits

Author	SHA1	Message	Date
yihao.dai	b45cf2d49f	enhance: Add max length check for csv import (#37077) ... 1. Add max length check for csv import. 2. Tidy import options. 3. Tidy common import util functions. issue: https://github.com/milvus-io/milvus/issues/34150 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-10-25 14:37:29 +08:00
yihao.dai	4de063ae14	fix: Make the dynamic column optional in parquet import (#32738) ... issue: https://github.com/milvus-io/milvus/issues/32729 Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-05-07 11:21:29 +08:00
yihao.dai	1594122c0a	enhance: Make the dynamic field file optional during numpy import (#32596) ... 1. Make the dynamic field file optional during numpy import 2. Add integration importing test with dynamic 3. Disallow file of pk when autoID=true during numpy import issue: https://github.com/milvus-io/milvus/issues/32542 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-04-28 19:39:25 +08:00
Cai Yudong	5fc439c600	feat: Bulk insert support fp16/bf16 (#32157) ... Issue: #22837 Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>	2024-04-22 10:05:22 +08:00
yihao.dai	31cf849f68	enhance: Support retriving file size from importutilv2.Reader (#31533) ... To reduce the overhead caused by listing the S3 objects, add an interface to importutil.Reader to retrieve file sizes. issue: https://github.com/milvus-io/milvus/issues/31532, https://github.com/milvus-io/milvus/issues/28521 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-03-25 20:29:07 +08:00
yihao.dai	c5918290e6	feat: Add import executor and manager for datanode (#29438) ... This PR introduces novel importv2 roles for datanode: 1. Executor: To execute tasks, a import task will be divided into the following steps: read data -> hash data -> sync data; 2. Manager: To manage all the tasks; issue: https://github.com/milvus-io/milvus/issues/28521 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-01-31 20:45:04 +08:00
yihao.dai	3d07b6682c	feat: Add import reader for numpy (#29253) ... This PR implements a new numpy reader for import. issue: https://github.com/milvus-io/milvus/issues/28521 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-01-08 19:42:49 +08:00