9 Commits

Author	SHA1	Message	Date
yihao.dai	31cf849f68	enhance: Support retriving file size from importutilv2.Reader (#31533) ... To reduce the overhead caused by listing the S3 objects, add an interface to importutil.Reader to retrieve file sizes. issue: https://github.com/milvus-io/milvus/issues/31532, https://github.com/milvus-io/milvus/issues/28521 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-03-25 20:29:07 +08:00
yihao.dai	9a13b9822f	enhance: Return more fields in import progress response (#31539) ... Return more fields in import progress response, include importedRows and totalRows. Additionally, ensure compatibility with the old import progress response by retaining fields of create timestamp and row count. issue: https://github.com/milvus-io/milvus/issues/31448 https://github.com/milvus-io/milvus/issues/31237 https://github.com/milvus-io/milvus/issues/28521 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-03-24 21:57:06 +08:00
yihao.dai	87b3c25b15	fix: Fix binlog import (#31205) ... 1. File type validation is omitted during binlog import. 2. System fields are appended to the schema during binlog import. issue: https://github.com/milvus-io/milvus/issues/28521 Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-03-13 10:35:04 +08:00
cai.zhang	de2c95d00c	enhance: Constraint dynamic field as key-value format (#31183) ... issue: #31051 Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>	2024-03-12 12:45:03 +08:00
yihao.dai	c5918290e6	feat: Add import executor and manager for datanode (#29438) ... This PR introduces novel importv2 roles for datanode: 1. Executor: To execute tasks, a import task will be divided into the following steps: read data -> hash data -> sync data; 2. Manager: To manage all the tasks; issue: https://github.com/milvus-io/milvus/issues/28521 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-01-31 20:45:04 +08:00
yihao.dai	3d07b6682c	feat: Add import reader for numpy (#29253) ... This PR implements a new numpy reader for import. issue: https://github.com/milvus-io/milvus/issues/28521 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-01-08 19:42:49 +08:00
yihao.dai	156a0dd450	feat: Add import reader for Parquet (#29618) ... This PR implements a Parquet reader for import. issue: https://github.com/milvus-io/milvus/issues/28521 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-01-07 19:38:49 +08:00
yihao.dai	23183ffb0f	feat: Add import reader for json (#29252) ... This PR implements a new json reader for import. issue: https://github.com/milvus-io/milvus/issues/28521 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-01-05 18:12:48 +08:00
yihao.dai	3561586edf	feat: Add import reader for binlog (#28910) ... This PR defines the new import reader interfaces and implement a binlog reader for import. issue: https://github.com/milvus-io/milvus/issues/28521 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-01-05 11:48:47 +08:00