milvus

mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-03 12:29:36 +08:00

Author	SHA1	Message	Date
cai.zhang	04c306e63f	fix: Fix clustering compaction task leak (#36800 ) issue: #36686 bug reason: - The clustering compaction tasks on the datanode were never cleaned up. - The clustering compaction task contains a mapping from clustering key to buffer, this caused a large memory leak. fix: - clean the tasks on datanode by datacoord when clustering compaction finished. - reset the mapping that from clustering key to buffer on datanode when clustering finished. Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>	2024-10-17 20:43:30 +08:00
XuanYang-cn	b172ea1093	fix: Remove enableLevelZeroSegment config (#36535 ) See also: #36504 --------- Signed-off-by: yangxuan <xuan.yang@zilliz.com>	2024-10-17 11:59:24 +08:00
aoiasd	5ec4163d0f	feat: support bm25 logs mixcompaction (#36072 ) relate: https://github.com/milvus-io/milvus/issues/35853 --------- Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>	2024-10-14 16:57:22 +08:00
Buqian Zheng	82c5cf2fa2	feat: add bulk insert support for Functions (#36715 ) issue: https://github.com/milvus-io/milvus/issues/35853 and https://github.com/milvus-io/milvus/issues/35856 Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>	2024-10-12 17:19:20 +08:00
CharlesFeng	7c8b71e26c	fix: BinlogDeserializeReader leak in mix_compactor.go (#36270 ) https://github.com/milvus-io/milvus/issues/36269 Signed-off-by: fengjun2016 <jornfeng@gmail.com>	2024-10-11 15:41:20 +08:00
XuanYang-cn	290ceb4e84	enhance: Add more info in logs (#36731 ) Signed-off-by: yangxuan <xuan.yang@zilliz.com>	2024-10-10 17:51:25 +08:00
yihao.dai	0fc2a4aa53	enhance: Optimize import scheduling and add time cost metric (#36601 ) 1. Optimize import scheduling strategic: a. Revise slot weights, calculating them based on the number of files and segments for both import and pre-import tasks. b. Ensure that the DN executes tasks in ascending order of task ID. 2. Add time cost metric and log. issue: https://github.com/milvus-io/milvus/issues/36600, https://github.com/milvus-io/milvus/issues/36518 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-10-09 14:41:20 +08:00
wayblink	00a5025949	enhance: support clustering compaction on null value (#36372 ) issue: #36055 Signed-off-by: wayblink <anyang.wang@zilliz.com>	2024-09-30 14:33:17 +08:00
yihao.dai	80f25d497f	enhance: Add metrics to monitor import throughput and imported rows (#36519 ) issue: https://github.com/milvus-io/milvus/issues/36518 Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-09-28 17:31:15 +08:00
cai.zhang	2adca8b754	fix: Fix data race for cluerting compaction (#36440 ) issue: #36438 Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>	2024-09-28 17:19:21 +08:00
yihao.dai	9e8cafcbe2	enhance: Skip loading bf in datanode (#36367 ) Skip loading bf in datanode: 1. When watching vchannels, skip loading bloom filters for segments. 2. Bypass bloom filter checks for delete messages, directly writing to L0 segments. 3. Remove flushed segments proactively after flush. issue: https://github.com/milvus-io/milvus/issues/34585 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-09-26 10:11:15 +08:00
aoiasd	139787371e	feat: support embedding bm25 sparse vector and flush bm25 stats log (#36036 ) relate: https://github.com/milvus-io/milvus/issues/35853 --------- Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>	2024-09-19 10:57:12 +08:00
congqixia	3352030a84	enhance: Graceful stop flowgraph manager when stopping datanode (#36229 ) Flowgraph manager is not stopped durong datanode stopping procedure which may lead to unexpect flowgraph behavior during/after datanode stop progress. --------- Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>	2024-09-14 15:55:08 +08:00
cai.zhang	8395c8a8db	enhance: Update stats task to optional (#35947 ) issue: #33744 --------- Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>	2024-09-12 20:37:08 +08:00
smellthemoon	3f75bf1f20	fix: clustering compact not support null (#36152 ) #36055 Signed-off-by: lixinguo <xinguo.li@zilliz.com> Co-authored-by: lixinguo <xinguo.li@zilliz.com>	2024-09-11 14:49:06 +08:00
XuanYang-cn	2687747278	fix: Set an empty segment if compaction deleted all inserts (#36044 ) See also: #36038 --------- Signed-off-by: yangxuan <xuan.yang@zilliz.com>	2024-09-09 14:23:05 +08:00
Chun Han	e480b103bd	feat: supporing hybrid search group_by (#35982 ) related: #35096 Signed-off-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: MrPresent-Han <chun.han@gmail.com>	2024-09-08 17:09:04 +08:00
SimFG	5247631289	fix: fill the metric type field in the LoadMetaInfo object (#35962 ) - issue: #35960 Signed-off-by: SimFG <bang.fu@zilliz.com>	2024-09-05 20:50:23 -07:00
cai.zhang	90bdb171ab	fix: Fix data race for clustering compaction writer (#35957 ) issue: #35950 Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>	2024-09-05 04:07:10 +08:00
CharlesFeng	6eb8b3f745	fix: err degenerated to a new variable (#35891 ) https://github.com/milvus-io/milvus/issues/35890 Signed-off-by: fengjun2016 <jornfeng@gmail.com>	2024-09-04 14:57:04 +08:00
yihao.dai	6fd33285e1	fix: Fix compile error (#35901 ) /kind improvement Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-09-02 14:50:35 +08:00
cai.zhang	2c9bb4dfa3	feat: Support stats task to sort segment by PK (#35054 ) issue: #33744 This PR includes the following changes: 1. Added a new task type to the task scheduler in datacoord: stats task, which sorts segments by primary key. 2. Implemented segment sorting in indexnode. 3. Added a new field `FieldStatsLog` to SegmentInfo to store token index information. --------- Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>	2024-09-02 14:19:03 +08:00
yihao.dai	1413ffe9b1	enhance: Rename preAllocatedSegments (#35871 ) Rename `preAllocatedSegments` to `preAllocatedSegmentIDs` to avoid confusion. Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-09-01 17:09:01 +08:00
XuanYang-cn	323400c190	enhance: Enable to write multiple segments in mix compactor (#35705 ) Prevent segments to be written larger than maxSize * expansionRate See also: #35584 Signed-off-by: yangxuan <xuan.yang@zilliz.com> --------- Signed-off-by: yangxuan <xuan.yang@zilliz.com>	2024-08-30 11:29:01 +08:00
Zhen Ye	99dff06391	enhance: using streaming service in insert/upsert/flush/delete/querynode (#35406 ) issue: #33285 - using streaming service in insert/upsert/flush/delete/querynode - fixup flusher bugs and refactor the flush operation - enable streaming service for dml and ddl - pass the e2e when enabling streaming service - pass the integration tst when enabling streaming service --------- Signed-off-by: chyezh <chyezh@outlook.com> Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-08-29 10:03:08 +08:00
yihao.dai	9868fe4e6c	fix: Fix panic due to empty candidate import segments (#35673 ) issue: https://github.com/milvus-io/milvus/issues/35662 Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-08-27 17:08:59 +08:00
congqixia	59387f0b3a	fix: Use correct release func when datanode unwatch a channel (#35655 ) See also #35654 Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>	2024-08-23 14:09:03 +08:00
congqixia	ab532ae199	enhance: Add back BF lazy load logic for datanode watch channel (#35646 ) Add back lazy loading statslog when watch dml channel on datanode. Related to #22994 #27675 Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>	2024-08-22 19:42:57 +08:00
Ted Xu	41646c8439	feat: integrate new deltalog format (#35522 ) See #34123 --------- Signed-off-by: Ted Xu <ted.xu@zilliz.com>	2024-08-20 19:06:56 +08:00
smellthemoon	80a7c78f28	enhance: import supports null in parquet and json formats (#35558 ) #31728 --------- Signed-off-by: lixinguo <xinguo.li@zilliz.com> Co-authored-by: lixinguo <xinguo.li@zilliz.com>	2024-08-20 16:50:55 +08:00
XuanYang-cn	967f38672a	enhance: Add integration tests for l0 (#35429 ) See also: #34796 --------- Signed-off-by: yangxuan <xuan.yang@zilliz.com>	2024-08-19 10:56:54 +08:00
cai.zhang	1bbf7a3c0e	enhance: Optimize the use of locks and avoid double flush clustering buffer writer (#35486 ) issue: #35436 --------- Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>	2024-08-16 02:24:58 +08:00
wei liu	c0200eec39	enhance: limit getSegmentInfo batch size to avoid excced grpc message limit (#35394 ) issue: #35395 Signed-off-by: Wei Liu <wei.liu@zilliz.com>	2024-08-15 19:17:00 +08:00
CharlesFeng	b4b1ed6910	fix: before exiting, make sure the goroutine has exited (#35411 ) https://github.com/milvus-io/milvus/issues/35409 Signed-off-by: fengjun2016 <jornfeng@gmail.com>	2024-08-15 18:16:53 +08:00
cai.zhang	196b343a94	fix: Fix data race for clustering compaction (#35435 ) issue: #35436 Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>	2024-08-13 17:10:20 +08:00
cai.zhang	aaab827a16	fix: Fix the issue of missing stats log after clustering compaction (#35266 ) issue: #35265 Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>	2024-08-08 14:24:17 +08:00
yihao.dai	a4439cc911	enhance: Implement flusher in streamingNode (#34942 ) - Implement flusher to: - Manage the pipelines (creation, deletion, etc.) - Manage the segment write buffer - Manage sync operation (including receive flushMsg and execute flush) - Add a new `GetChannelRecoveryInfo` RPC in DataCoord. - Reorganize packages: `flushcommon` and `datanode`. issue: https://github.com/milvus-io/milvus/issues/33285 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-08-02 18:30:23 +08:00
congqixia	095d77269b	enhance: Set WatchProgress for channel operation progress response (#35183 ) The watch progress is always zero in CheckChannelOperationProgress response, which is meaningless and confusing. This PR set progress value in rpc response to fix this problem. Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>	2024-08-02 12:06:14 +08:00
wayblink	95462668ca	enhance: unify time in clustering compaction task to unix (#35167 ) #34495 Signed-off-by: wayblink <anyang.wang@zilliz.com>	2024-08-02 10:30:19 +08:00
zhenshan.cao	aa247f192d	enhance: remove unused code for StorageV2 (#35132 ) issue: https://github.com/milvus-io/milvus/issues/34168 Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>	2024-08-01 12:08:13 +08:00
cai.zhang	9412002d7d	fix: Fix data race for clustering buffer writer (#35145 ) issue: #34495 Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>	2024-08-01 11:20:13 +08:00
congqixia	de8a266d8a	enhance: Enable linux code checker (#35084 ) See also #34483 --------- Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>	2024-07-30 15:53:51 +08:00
wei liu	c45f38aa61	enhance: Update protobuf-go to protobuf-go v2 (#34394 ) issue: #34252 Signed-off-by: Wei Liu <wei.liu@zilliz.com>	2024-07-29 11:31:51 +08:00
cai.zhang	2372452fac	enhance: Optimized the GC logic to ensure that memory is released in time (#34949 ) issue: #34703 Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>	2024-07-28 23:53:47 +08:00
cai.zhang	ea13b97b6a	fix: Fix bug for block clustering compaction (#35019 ) issue: #34703 Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>	2024-07-26 11:33:58 +08:00
congqixia	e8038a76e1	fix: Remove timeout in datanode watch ctx (#35011 ) See also #35008 Use tickle timeout logic instead of hardcode context timeout Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>	2024-07-25 22:17:45 +08:00
cai.zhang	260a6e2ba9	enhance: Send flush signal when the water level reaches the high watermark (#34907 ) issue: #30633 Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>	2024-07-23 10:29:42 +08:00
cai.zhang	4c45bc412f	enhance: Add integration test for clustering compaction (#34881 ) issue: #34792 --------- Signed-off-by: cai.zhang <cai.zhang@zilliz.com> Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>	2024-07-23 10:13:43 +08:00
cai.zhang	643b9d521c	fix: Fix the issue of concurrent packing of the same segment (#34840 ) issue: #34703 Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>	2024-07-23 10:11:49 +08:00
yihao.dai	8aab6cbfac	enhance: Organize the common modules of streamingNode and dataNode (#34773 ) 1. Move the common modules of streamingNode and dataNode to flushcommon 2. Add new GetVChannels interface for rootcoord issue: https://github.com/milvus-io/milvus/issues/33285 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-07-22 11:33:51 +08:00

1 2 3 4 5 ...

1108 Commits