1123 Commits

Author	SHA1	Message	Date
congqixia	b0bd290a6e	enhance: Use internal json(sonic) to replace std json lib (#37708) ... Related to #35020 Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>	2024-11-18 10:46:31 +08:00
congqixia	5e90f348fc	enhance: Handle legacy proxy load fields request (#37565) ... Related to #35415 In rolling upgrade, legacy proxy may dispatch load request wit empty load field list. The upgraded querycoord may report error by mistake that load field list is changed. This PR: - Auto field empty load field list with all user field ids - Refine the error messag when load field list updates - Refine load job unit test with service cases Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>	2024-11-11 10:14:26 +08:00
sthuang	70605cf5b3	enhance: Support custom privilege group for RBAC (#37087) ... issue: #37031 --------- Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>	2024-11-09 08:44:28 +08:00
yihao.dai	81879425e1	enhance: Optimize the performance of stats task (#37374) ... 1. Increase the writer's `batchSize` to avoid multiple serialization operations. 2. Perform asynchronous upload of binlog files to prevent blocking the data processing flow. 3. Reduce multiple calls to `writer.Flush()`. issue: https://github.com/milvus-io/milvus/issues/37373 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-11-08 10:08:27 +08:00
Ted Xu	bc9562feb1	enhance: avoid memory copy and serde in mix compaction (#37479) ... See: #37234 --------- Signed-off-by: Ted Xu <ted.xu@zilliz.com>	2024-11-07 16:30:57 -08:00
wei liu	00f6d0ec51	fix: watch channel stuck due to misuse of timer.Reset (#37433) ... issue: #37166 cause the misuse of timer.Reset, which cause dispatcher failed to send msg to virtual channel buffer, and dispatcher do splitting again and again, which hold the dispatcher manager's lock, block watching channel progress. This PR fix the misuse of timer.Reset Signed-off-by: Wei Liu <wei.liu@zilliz.com>	2024-11-07 14:34:24 +08:00
jaime	f348bd9441	feat: add segment,pipeline, replica and resourcegroup api for WebUI (#37344) ... issue: #36621 Signed-off-by: jaime <yun.zhang@zilliz.com>	2024-11-07 11:52:25 +08:00
aoiasd	b4c749dcd5	fix: merge sort segment loss data (#37400) ... relate: https://github.com/milvus-io/milvus/issues/37238 Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>	2024-11-07 11:18:26 +08:00
Ted Xu	b792b199d7	enhance: load deltalogs on demand when doing compactions (#37310) ... See #37234 --------- Signed-off-by: Ted Xu <ted.xu@zilliz.com>	2024-11-01 16:40:21 +08:00
Zhen Ye	448cc08960	fix: flowgraph crash when channel releasing (#37285) ... issue: #37284 Signed-off-by: chyezh <chyezh@outlook.com>	2024-10-31 16:30:21 +08:00
Ted Xu	262a994d6d	enhance: generally improve the performance of mix compactions (#37163) ... See #37234 --------- Signed-off-by: Ted Xu <ted.xu@zilliz.com>	2024-10-29 18:12:20 +08:00
congqixia	3106384fc4	enhance: Return deltadata for DeleteCodec.Deserialize (#37214) ... Related to #35303 #30404 This PR change return type of `DeleteCodec.Deserialize` from `storage.DeleteData` to `DeltaData`, which reduces the memory usage of interface header. Also refine `storage.DeltaData` methods to make it easier to usage. Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>	2024-10-29 12:04:24 +08:00
jaime	9d16b972ea	feat: add tasks page into management WebUI (#37002) ... issue: #36621 1. Add API to access task runtime metrics, including: - build index task - compaction task - import task - balance (including load/release of segments/channels and some leader tasks on querycoord) - sync task 2. Add a debug model to the webpage by using debug=true or debug=false in the URL query parameters to enable or disable debug mode. Signed-off-by: jaime <yun.zhang@zilliz.com>	2024-10-28 10:13:29 +08:00
yihao.dai	d7b2906318	enhance: Make dataNode.import.maxConcurrentTaskNum dynamic (#37102) ... Resize import execution pool when config `dataNode.import.maxConcurrentTaskNum` update. issue: https://github.com/milvus-io/milvus/issues/37095 Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-10-25 16:51:29 +08:00
jaime	4746f47282	feat: management WebUI homepage (#36822) ... issue: #36784 1. Implement an embedded web server for WebUI access. 2. Complete the homepage development. Home page demo: <img width="2177" alt="iShot_2024-10-10_17 57 34" src="https://github.com/user-attachments/assets/38539917-ce09-4e54-a5b5-7f4f7eaac353"> Signed-off-by: jaime <yun.zhang@zilliz.com>	2024-10-23 11:29:28 +08:00
cai.zhang	04c306e63f	fix: Fix clustering compaction task leak (#36800) ... issue: #36686 bug reason: - The clustering compaction tasks on the datanode were never cleaned up. - The clustering compaction task contains a mapping from clustering key to buffer, this caused a large memory leak. fix: - clean the tasks on datanode by datacoord when clustering compaction finished. - reset the mapping that from clustering key to buffer on datanode when clustering finished. Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>	2024-10-17 20:43:30 +08:00
XuanYang-cn	b172ea1093	fix: Remove enableLevelZeroSegment config (#36535) ... See also: #36504 --------- Signed-off-by: yangxuan <xuan.yang@zilliz.com>	2024-10-17 11:59:24 +08:00
aoiasd	5ec4163d0f	feat: support bm25 logs mixcompaction (#36072) ... relate: https://github.com/milvus-io/milvus/issues/35853 --------- Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>	2024-10-14 16:57:22 +08:00
Buqian Zheng	82c5cf2fa2	feat: add bulk insert support for Functions (#36715) ... issue: https://github.com/milvus-io/milvus/issues/35853 and https://github.com/milvus-io/milvus/issues/35856 Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>	2024-10-12 17:19:20 +08:00
CharlesFeng	7c8b71e26c	fix: BinlogDeserializeReader leak in mix_compactor.go (#36270) ... https://github.com/milvus-io/milvus/issues/36269 Signed-off-by: fengjun2016 <jornfeng@gmail.com>	2024-10-11 15:41:20 +08:00
XuanYang-cn	290ceb4e84	enhance: Add more info in logs (#36731) ... Signed-off-by: yangxuan <xuan.yang@zilliz.com>	2024-10-10 17:51:25 +08:00
yihao.dai	0fc2a4aa53	enhance: Optimize import scheduling and add time cost metric (#36601) ... 1. Optimize import scheduling strategic: a. Revise slot weights, calculating them based on the number of files and segments for both import and pre-import tasks. b. Ensure that the DN executes tasks in ascending order of task ID. 2. Add time cost metric and log. issue: https://github.com/milvus-io/milvus/issues/36600, https://github.com/milvus-io/milvus/issues/36518 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-10-09 14:41:20 +08:00
wayblink	00a5025949	enhance: support clustering compaction on null value (#36372) ... issue: #36055 Signed-off-by: wayblink <anyang.wang@zilliz.com>	2024-09-30 14:33:17 +08:00
yihao.dai	80f25d497f	enhance: Add metrics to monitor import throughput and imported rows (#36519) ... issue: https://github.com/milvus-io/milvus/issues/36518 Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-09-28 17:31:15 +08:00
cai.zhang	2adca8b754	fix: Fix data race for cluerting compaction (#36440) ... issue: #36438 Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>	2024-09-28 17:19:21 +08:00
yihao.dai	9e8cafcbe2	enhance: Skip loading bf in datanode (#36367) ... Skip loading bf in datanode: 1. When watching vchannels, skip loading bloom filters for segments. 2. Bypass bloom filter checks for delete messages, directly writing to L0 segments. 3. Remove flushed segments proactively after flush. issue: https://github.com/milvus-io/milvus/issues/34585 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-09-26 10:11:15 +08:00
aoiasd	139787371e	feat: support embedding bm25 sparse vector and flush bm25 stats log (#36036) ... relate: https://github.com/milvus-io/milvus/issues/35853 --------- Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>	2024-09-19 10:57:12 +08:00
congqixia	3352030a84	enhance: Graceful stop flowgraph manager when stopping datanode (#36229) ... Flowgraph manager is not stopped durong datanode stopping procedure which may lead to unexpect flowgraph behavior during/after datanode stop progress. --------- Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>	2024-09-14 15:55:08 +08:00
cai.zhang	8395c8a8db	enhance: Update stats task to optional (#35947) ... issue: #33744 --------- Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>	2024-09-12 20:37:08 +08:00
smellthemoon	3f75bf1f20	fix: clustering compact not support null (#36152) ... #36055 Signed-off-by: lixinguo <xinguo.li@zilliz.com> Co-authored-by: lixinguo <xinguo.li@zilliz.com>	2024-09-11 14:49:06 +08:00
XuanYang-cn	2687747278	fix: Set an empty segment if compaction deleted all inserts (#36044) ... See also: #36038 --------- Signed-off-by: yangxuan <xuan.yang@zilliz.com>	2024-09-09 14:23:05 +08:00
Chun Han	e480b103bd	feat: supporing hybrid search group_by (#35982) ... related: #35096 Signed-off-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: MrPresent-Han <chun.han@gmail.com>	2024-09-08 17:09:04 +08:00
SimFG	5247631289	fix: fill the metric type field in the LoadMetaInfo object (#35962) ... - issue: #35960 Signed-off-by: SimFG <bang.fu@zilliz.com>	2024-09-05 20:50:23 -07:00
cai.zhang	90bdb171ab	fix: Fix data race for clustering compaction writer (#35957) ... issue: #35950 Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>	2024-09-05 04:07:10 +08:00
CharlesFeng	6eb8b3f745	fix: err degenerated to a new variable (#35891) ... https://github.com/milvus-io/milvus/issues/35890 Signed-off-by: fengjun2016 <jornfeng@gmail.com>	2024-09-04 14:57:04 +08:00
yihao.dai	6fd33285e1	fix: Fix compile error (#35901) ... /kind improvement Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-09-02 14:50:35 +08:00
cai.zhang	2c9bb4dfa3	feat: Support stats task to sort segment by PK (#35054) ... issue: #33744 This PR includes the following changes: 1. Added a new task type to the task scheduler in datacoord: stats task, which sorts segments by primary key. 2. Implemented segment sorting in indexnode. 3. Added a new field `FieldStatsLog` to SegmentInfo to store token index information. --------- Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>	2024-09-02 14:19:03 +08:00
yihao.dai	1413ffe9b1	enhance: Rename preAllocatedSegments (#35871) ... Rename `preAllocatedSegments` to `preAllocatedSegmentIDs` to avoid confusion. Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-09-01 17:09:01 +08:00
XuanYang-cn	323400c190	enhance: Enable to write multiple segments in mix compactor (#35705) ... Prevent segments to be written larger than maxSize * expansionRate See also: #35584 Signed-off-by: yangxuan <xuan.yang@zilliz.com> --------- Signed-off-by: yangxuan <xuan.yang@zilliz.com>	2024-08-30 11:29:01 +08:00
Zhen Ye	99dff06391	enhance: using streaming service in insert/upsert/flush/delete/querynode (#35406) ... issue: #33285 - using streaming service in insert/upsert/flush/delete/querynode - fixup flusher bugs and refactor the flush operation - enable streaming service for dml and ddl - pass the e2e when enabling streaming service - pass the integration tst when enabling streaming service --------- Signed-off-by: chyezh <chyezh@outlook.com> Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-08-29 10:03:08 +08:00
yihao.dai	9868fe4e6c	fix: Fix panic due to empty candidate import segments (#35673) ... issue: https://github.com/milvus-io/milvus/issues/35662 Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-08-27 17:08:59 +08:00
congqixia	59387f0b3a	fix: Use correct release func when datanode unwatch a channel (#35655) ... See also #35654 Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>	2024-08-23 14:09:03 +08:00
congqixia	ab532ae199	enhance: Add back BF lazy load logic for datanode watch channel (#35646) ... Add back lazy loading statslog when watch dml channel on datanode. Related to #22994 #27675 Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>	2024-08-22 19:42:57 +08:00
Ted Xu	41646c8439	feat: integrate new deltalog format (#35522) ... See #34123 --------- Signed-off-by: Ted Xu <ted.xu@zilliz.com>	2024-08-20 19:06:56 +08:00
smellthemoon	80a7c78f28	enhance: import supports null in parquet and json formats (#35558) ... #31728 --------- Signed-off-by: lixinguo <xinguo.li@zilliz.com> Co-authored-by: lixinguo <xinguo.li@zilliz.com>	2024-08-20 16:50:55 +08:00
XuanYang-cn	967f38672a	enhance: Add integration tests for l0 (#35429) ... See also: #34796 --------- Signed-off-by: yangxuan <xuan.yang@zilliz.com>	2024-08-19 10:56:54 +08:00
cai.zhang	1bbf7a3c0e	enhance: Optimize the use of locks and avoid double flush clustering buffer writer (#35486) ... issue: #35436 --------- Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>	2024-08-16 02:24:58 +08:00
wei liu	c0200eec39	enhance: limit getSegmentInfo batch size to avoid excced grpc message limit (#35394) ... issue: #35395 Signed-off-by: Wei Liu <wei.liu@zilliz.com>	2024-08-15 19:17:00 +08:00
CharlesFeng	b4b1ed6910	fix: before exiting, make sure the goroutine has exited (#35411) ... https://github.com/milvus-io/milvus/issues/35409 Signed-off-by: fengjun2016 <jornfeng@gmail.com>	2024-08-15 18:16:53 +08:00
cai.zhang	196b343a94	fix: Fix data race for clustering compaction (#35435) ... issue: #35436 Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>	2024-08-13 17:10:20 +08:00