9706 Commits

Author	SHA1	Message	Date
wei liu	c5485bb1b1	fix: Make GetShardLeaders skip retry on not loaded collection (#37684) ... issue: #37532 Signed-off-by: Wei Liu <wei.liu@zilliz.com>	2024-11-14 21:14:40 +08:00
Bingyi Sun	1b4f7e3ac1	enhance: Add more expr ut for chunked segment (#37600) ... related pr: #37570 --------- Signed-off-by: sunby <sunbingyi1992@gmail.com>	2024-11-14 18:40:32 +08:00
aoiasd	993051bb49	fix: brute force bm25 search lack avgdl param (#37650) ... relate: https://github.com/milvus-io/milvus/issues/35853 Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>	2024-11-14 14:58:31 +08:00
XuanYang-cn	31a8d08bdd	fix: Correct varchar primarykey size calculation (#37617) ... See also: #37582 --------- Signed-off-by: yangxuan <xuan.yang@zilliz.com>	2024-11-14 14:16:38 +08:00
Zhen Ye	cd181e4c6d	fix: make NewDataSyncService idempotent of dispatcher (#37576) ... issue: #37547 Signed-off-by: chyezh <chyezh@outlook.com>	2024-11-14 11:00:31 +08:00
Buqian Zheng	0565300b7f	fix: Sparse to use CC index as growing/temp index (#37591) ... relate: https://github.com/milvus-io/milvus/issues/35853 Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>	2024-11-14 10:54:31 +08:00
aoiasd	1c5b5e1e3d	feat: Add chinese and english analyzer with refactor jieba tokenizer (#37494) ... relate: https://github.com/milvus-io/milvus/issues/35853 Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>	2024-11-14 10:34:31 +08:00
wei liu	1304b40552	fix: Balance channel may stuck at increasing replica number case (#37641) ... issue: #37640 fix the pr #36549 cause balance channel will wait until new delegator becomes serviceable, but new delegator need to sync target version then becomes serviceable, and sync target version need to be wait all replica load done. so if increasing replica number and balance channel happens at same time, logic dead lock occurs. Signed-off-by: Wei Liu <wei.liu@zilliz.com>	2024-11-14 10:08:31 +08:00
congqixia	e7fb87d491	enhance: Fix lb policy retry log after targetNode type changed (#37646) ... Related to previous pr: #37371 Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>	2024-11-13 19:52:29 +08:00
smellthemoon	a654487995	fix: some error in restful (#37487) ... #37370 Signed-off-by: lixinguo <xinguo.li@zilliz.com> Co-authored-by: lixinguo <xinguo.li@zilliz.com>	2024-11-13 17:12:39 +08:00
foxspy	cf883b114e	enhance: update knowhere version (#37510) ... issue: #36925 Signed-off-by: xianliang.li <xianliang.li@zilliz.com>	2024-11-13 16:36:27 +08:00
Ted Xu	1a49da2cc0	fix: refuse schedule compaction tasks if there is no slot (#37589) ... See #37621 --------- Signed-off-by: Ted Xu <ted.xu@zilliz.com>	2024-11-13 15:12:30 +08:00
smellthemoon	3389a6b500	enhance: support null in text match index (#37517) ... #37508 Signed-off-by: lixinguo <xinguo.li@zilliz.com> Co-authored-by: lixinguo <xinguo.li@zilliz.com>	2024-11-13 11:08:29 +08:00
Zhen Ye	3c225e5c94	fix: data race when using fields_ (#37612) ... issue: #37609 Signed-off-by: chyezh <chyezh@outlook.com>	2024-11-13 04:06:30 +08:00
XuanYang-cn	5ab3c562da	enhance: tidy compaction logs (#37595) ... Remove some annoying logs and lower a log level from warn to info Signed-off-by: yangxuan <xuan.yang@zilliz.com>	2024-11-12 21:18:29 +08:00
cai.zhang	14e007d6fb	fix: Fix the bug that retrieved from wrong field for L0 segments (#37598) ... issue: #37574 Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>	2024-11-12 20:58:29 +08:00
jaime	1e8ea4a7e7	feat: add segment/channel/task/slow query render (#37561) ... issue: #36621 Signed-off-by: jaime <yun.zhang@zilliz.com>	2024-11-12 17:44:29 +08:00
wei liu	266f8ef1f5	fix: Search may return less result after qn recover (#36549) ... issue: #36293 #36242 after qn recover, delegator may be loaded in new node, after all segment has been loaded, delegator becomes serviceable. but delegator's target version hasn't been synced, and if search/query comes, delegator will use wrong target version to filter out a empty segment list, which caused empty search result. This pr will block delegator's serviceable status until target version is synced --------- Signed-off-by: Wei Liu <wei.liu@zilliz.com>	2024-11-12 16:34:28 +08:00
Zhen Ye	b5b003551e	enhance: use localhost for it and ut (#37529) ... issue: #37528 Signed-off-by: chyezh <chyezh@outlook.com>	2024-11-12 11:36:27 +08:00
aoiasd	4682bba586	enhance: Support gc with bm25 stats logs (#37408) ... Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>	2024-11-12 11:30:28 +08:00
XuanYang-cn	a45a288a25	fix: Separate L0 and Mix trigger interval (#37190) ... See also: #37108 - Add MixCompactionTriggerInterval, default 60s - Add L0CompactionTriggerInterval, default 10s - Export Single related compaction configs - Raise SingleCompactionDeltaLogMaxSize from 2MB to 16MB --------- Signed-off-by: yangxuan <xuan.yang@zilliz.com>	2024-11-12 10:56:37 +08:00
wei liu	2a4c00de9d	enhance: Decouple shard client manager from shard cache (#37371) ... issue: #37115 the old implementation update shard cache and shard client manager at same time, which causes lots of conor case due to concurrent issue without lock. This PR decouple shard client manager from shard cache, so only shard cache will be updated if delegator changes. and make sure shard client manager will always return the right client, and create a new client if not exist. in case of client leak, shard client manager will purge client in async for every 10 minutes. --------- Signed-off-by: Wei Liu <wei.liu@zilliz.com>	2024-11-12 10:30:28 +08:00
congqixia	f5b06a3c9f	enhance: Invalidate collection cache when release collection (#37577) ... Related to #37395 --------- Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>	2024-11-12 10:16:29 +08:00
Chun Han	2d29dcd30c	enhance:refine group_strict_size parameter(#37482) (#37483) ... related: #37482 Signed-off-by: MrPresent-Han <chun.han@gmail.com> Co-authored-by: MrPresent-Han <chun.han@gmail.com>	2024-11-12 09:56:28 +08:00
Bingyi Sun	c1eccce2fa	enhance: enable multiple chunked segment by default (#37570) ... Signed-off-by: sunby <sunbingyi1992@gmail.com>	2024-11-12 09:20:28 +08:00
Zhen Ye	1b6edd0b4b	enhance: refactor the consumer grpc proto for reusing grpc stream for multi-consumer (#37564) ... issue: #33285 - Modify the proto of consumer of streaming service. - Make VChannel as a required option for streaming --------- Signed-off-by: chyezh <chyezh@outlook.com>	2024-11-11 17:24:29 +08:00
XuanYang-cn	5e6c3df253	fix: l0RowCount metrics value always empty (#37306) ... See also: #36953 --------- Signed-off-by: yangxuan <xuan.yang@zilliz.com>	2024-11-11 17:00:27 +08:00
wei liu	61a5b15ada	fix: Lost loading collection's updateTs after qc restart (#37538) ... issue: #37537 Signed-off-by: Wei Liu <wei.liu@zilliz.com>	2024-11-11 14:34:28 +08:00
Ted Xu	31d0c84f67	fix: panic calclulating data size on writing binlogs (#37575) ... See: #37568 Signed-off-by: Ted Xu <ted.xu@zilliz.com>	2024-11-11 14:08:27 +08:00
Zhen Ye	fca946dee1	enhance: move the search utilities from querynode into new module (#37531) ... issue: #33285 - The search utilities will be shared between query node and streaming node. Signed-off-by: chyezh <chyezh@outlook.com>	2024-11-11 11:10:27 +08:00
congqixia	5e90f348fc	enhance: Handle legacy proxy load fields request (#37565) ... Related to #35415 In rolling upgrade, legacy proxy may dispatch load request wit empty load field list. The upgraded querycoord may report error by mistake that load field list is changed. This PR: - Auto field empty load field list with all user field ids - Refine the error messag when load field list updates - Refine load job unit test with service cases Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>	2024-11-11 10:14:26 +08:00
aoiasd	12951f0abb	enhance: rename tokenizer to analyzer and check analyzer params (#37478) ... relate: https://github.com/milvus-io/milvus/issues/35853 --------- Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>	2024-11-10 16:12:26 +08:00
sthuang	ff00a12805	enhance: RBAC custom privilege group ut coverage (#37558) ... issue: https://github.com/milvus-io/milvus/issues/37031 Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>	2024-11-09 20:40:25 +08:00
sthuang	70605cf5b3	enhance: Support custom privilege group for RBAC (#37087) ... issue: #37031 --------- Signed-off-by: shaoting-huang <shaoting.huang@zilliz.com>	2024-11-09 08:44:28 +08:00
yihao.dai	ff9bdf7029	fix: Fix load slowly (#37454) ... When there're a lot of loaded collections, they would occupy the target observer scheduler’s pool. This prevents loading collections from updating the current target in time, slowing down the load process. This PR adds a separate target dispatcher for loading collections. issue: https://github.com/milvus-io/milvus/issues/37166 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-11-09 07:48:26 +08:00
XuanYang-cn	ce2fa3de01	fix: Set correct memory size (#37551) ... See also: #37549 Signed-off-by: yangxuan <xuan.yang@zilliz.com>	2024-11-09 01:42:26 +08:00
yihao.dai	ebc3c82761	fix: Fix subscription leak (#37382) ... Close (unsubscribe) the msg stream after completing the `PreCreatedTopic` check to prevent backlog issue. issue: https://github.com/milvus-io/milvus/issues/36021 Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-11-08 16:16:26 +08:00
congqixia	0f514708ff	enhance: Accelerate Build stage by skipping cpp ut compilation (#37530) ... Since master branch run cpp unittest in jenken, the cpp unit test building could be skipped since the output is not needed in github runner. Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>	2024-11-08 14:06:26 +08:00
congqixia	dcc1b506dc	enhance: Add context trace for querycoord queryable check (#37524) ... When check health logic failed to collection not-queryable, the related reason is hard to find in log. This PR add context for log with trace id and print unqueryable collection info log. Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>	2024-11-08 14:00:26 +08:00
wei liu	a03157838b	enhance: Enable node assign policy on resource group (#36968) ... issue: #36977 with node_label_filter on resource group, user can add label on querynode with env `MILVUS_COMPONENT_LABEL`, then resource group will prefer to accept node which match it's node_label_filter. then querynode's can't be group by labels, and put querynodes with same label to same resource groups. Signed-off-by: Wei Liu <wei.liu@zilliz.com>	2024-11-08 11:18:27 +08:00
Xiaofan	e073906a19	enhance: optimize describe collection and index (#37490) ... fix #37489 combine multiple describe collection and list index into one call Signed-off-by: xiaofanluan <xiaofan.luan@zilliz.com>	2024-11-08 10:18:34 +08:00
Zhen Ye	49657c4690	enhance: add create segment message, enable empty segment flush (#37407) ... issue: #37172 - add redo interceptor to implement append context refresh. (make new timetick) - add create segment handler for flusher. - make empty segment flushable and directly change it into dropped. - add create segment message into wal when creating new growing segment. - make the insert operation into following seq: createSegment -> insert -> insert -> flushSegment. - make manual flush into following seq: flushTs -> flushsegment -> flushsegment -> manualflush. --------- Signed-off-by: chyezh <chyezh@outlook.com>	2024-11-08 10:16:34 +08:00
yihao.dai	81879425e1	enhance: Optimize the performance of stats task (#37374) ... 1. Increase the writer's `batchSize` to avoid multiple serialization operations. 2. Perform asynchronous upload of binlog files to prevent blocking the data processing flow. 3. Reduce multiple calls to `writer.Flush()`. issue: https://github.com/milvus-io/milvus/issues/37373 --------- Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-11-08 10:08:27 +08:00
Ted Xu	bc9562feb1	enhance: avoid memory copy and serde in mix compaction (#37479) ... See: #37234 --------- Signed-off-by: Ted Xu <ted.xu@zilliz.com>	2024-11-07 16:30:57 -08:00
Bingyi Sun	40ba5a3414	fix: fix chunked segment term filter expression and add ut (#37392) ... issue: https://github.com/milvus-io/milvus/issues/37143 --------- Signed-off-by: sunby <sunbingyi1992@gmail.com>	2024-11-07 11:04:19 -08:00
congqixia	5310d3469f	fix: Escape brace of dumped JSON for index err message (#37504) ... Related to #37503 Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>	2024-11-07 18:58:25 +08:00
Bingyi Sun	7ec449fee5	enhance: regenerate datacoord mockery (#37501) ... Signed-off-by: sunby <sunbingyi1992@gmail.com>	2024-11-07 17:38:24 +08:00
yihao.dai	36bb87d9b6	fix: Fix large growing segment (#37388) ... Consider the `sealProportion` factor during segment allocation. issue: https://github.com/milvus-io/milvus/issues/37387 Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-11-07 17:12:25 +08:00
yihao.dai	994f52fab8	fix: Revert "enhance: Support db for bulkinsert (#37012)" (#37420) ... This reverts commit 6e90f9e8d9. issue: https://github.com/milvus-io/milvus/issues/31273 Signed-off-by: bigsheeper <yihao.dai@zilliz.com>	2024-11-07 17:02:25 +08:00
cai.zhang	de627644f5	enhance: Update the template expression proto to improve transmission efficiency (#37484) ... issue: #36672 Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>	2024-11-07 16:28:25 +08:00