aoiasd
5ec4163d0f
feat: support bm25 logs mixcompaction ( #36072 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-10-14 16:57:22 +08:00
Buqian Zheng
82c5cf2fa2
feat: add bulk insert support for Functions ( #36715 )
...
issue: https://github.com/milvus-io/milvus/issues/35853 and
https://github.com/milvus-io/milvus/issues/35856
Signed-off-by: Buqian Zheng <zhengbuqian@gmail.com>
2024-10-12 17:19:20 +08:00
CharlesFeng
7c8b71e26c
fix: BinlogDeserializeReader leak in mix_compactor.go ( #36270 )
...
https://github.com/milvus-io/milvus/issues/36269
Signed-off-by: fengjun2016 <jornfeng@gmail.com>
2024-10-11 15:41:20 +08:00
XuanYang-cn
290ceb4e84
enhance: Add more info in logs ( #36731 )
...
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-10-10 17:51:25 +08:00
yihao.dai
0fc2a4aa53
enhance: Optimize import scheduling and add time cost metric ( #36601 )
...
1. Optimize import scheduling strategic:
a. Revise slot weights, calculating them based on the number of files
and segments for both import and pre-import tasks.
b. Ensure that the DN executes tasks in ascending order of task ID.
2. Add time cost metric and log.
issue: https://github.com/milvus-io/milvus/issues/36600 ,
https://github.com/milvus-io/milvus/issues/36518
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-10-09 14:41:20 +08:00
wayblink
00a5025949
enhance: support clustering compaction on null value ( #36372 )
...
issue: #36055
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-09-30 14:33:17 +08:00
yihao.dai
80f25d497f
enhance: Add metrics to monitor import throughput and imported rows ( #36519 )
...
issue: https://github.com/milvus-io/milvus/issues/36518
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-28 17:31:15 +08:00
cai.zhang
2adca8b754
fix: Fix data race for cluerting compaction ( #36440 )
...
issue: #36438
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-28 17:19:21 +08:00
yihao.dai
9e8cafcbe2
enhance: Skip loading bf in datanode ( #36367 )
...
Skip loading bf in datanode:
1. When watching vchannels, skip loading bloom filters for segments.
2. Bypass bloom filter checks for delete messages, directly writing to
L0 segments.
3. Remove flushed segments proactively after flush.
issue: https://github.com/milvus-io/milvus/issues/34585
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-26 10:11:15 +08:00
aoiasd
139787371e
feat: support embedding bm25 sparse vector and flush bm25 stats log ( #36036 )
...
relate: https://github.com/milvus-io/milvus/issues/35853
---------
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2024-09-19 10:57:12 +08:00
congqixia
3352030a84
enhance: Graceful stop flowgraph manager when stopping datanode ( #36229 )
...
Flowgraph manager is not stopped durong datanode stopping procedure
which may lead to unexpect flowgraph behavior during/after datanode stop
progress.
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-09-14 15:55:08 +08:00
cai.zhang
8395c8a8db
enhance: Update stats task to optional ( #35947 )
...
issue: #33744
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-12 20:37:08 +08:00
smellthemoon
3f75bf1f20
fix: clustering compact not support null ( #36152 )
...
#36055
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-09-11 14:49:06 +08:00
XuanYang-cn
2687747278
fix: Set an empty segment if compaction deleted all inserts ( #36044 )
...
See also: #36038
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-09-09 14:23:05 +08:00
Chun Han
e480b103bd
feat: supporing hybrid search group_by ( #35982 )
...
related: #35096
Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
2024-09-08 17:09:04 +08:00
SimFG
5247631289
fix: fill the metric type field in the LoadMetaInfo object ( #35962 )
...
- issue: #35960
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-09-05 20:50:23 -07:00
cai.zhang
90bdb171ab
fix: Fix data race for clustering compaction writer ( #35957 )
...
issue: #35950
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-05 04:07:10 +08:00
CharlesFeng
6eb8b3f745
fix: err degenerated to a new variable ( #35891 )
...
https://github.com/milvus-io/milvus/issues/35890
Signed-off-by: fengjun2016 <jornfeng@gmail.com>
2024-09-04 14:57:04 +08:00
yihao.dai
6fd33285e1
fix: Fix compile error ( #35901 )
...
/kind improvement
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-02 14:50:35 +08:00
cai.zhang
2c9bb4dfa3
feat: Support stats task to sort segment by PK ( #35054 )
...
issue: #33744
This PR includes the following changes:
1. Added a new task type to the task scheduler in datacoord: stats task,
which sorts segments by primary key.
2. Implemented segment sorting in indexnode.
3. Added a new field `FieldStatsLog` to SegmentInfo to store token index
information.
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-09-02 14:19:03 +08:00
yihao.dai
1413ffe9b1
enhance: Rename preAllocatedSegments ( #35871 )
...
Rename `preAllocatedSegments` to `preAllocatedSegmentIDs` to avoid
confusion.
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-09-01 17:09:01 +08:00
XuanYang-cn
323400c190
enhance: Enable to write multiple segments in mix compactor ( #35705 )
...
Prevent segments to be written larger than maxSize * expansionRate
See also: #35584
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-30 11:29:01 +08:00
Zhen Ye
99dff06391
enhance: using streaming service in insert/upsert/flush/delete/querynode ( #35406 )
...
issue: #33285
- using streaming service in insert/upsert/flush/delete/querynode
- fixup flusher bugs and refactor the flush operation
- enable streaming service for dml and ddl
- pass the e2e when enabling streaming service
- pass the integration tst when enabling streaming service
---------
Signed-off-by: chyezh <chyezh@outlook.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-29 10:03:08 +08:00
yihao.dai
9868fe4e6c
fix: Fix panic due to empty candidate import segments ( #35673 )
...
issue: https://github.com/milvus-io/milvus/issues/35662
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-27 17:08:59 +08:00
congqixia
59387f0b3a
fix: Use correct release func when datanode unwatch a channel ( #35655 )
...
See also #35654
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-23 14:09:03 +08:00
congqixia
ab532ae199
enhance: Add back BF lazy load logic for datanode watch channel ( #35646 )
...
Add back lazy loading statslog when watch dml channel on datanode.
Related to #22994 #27675
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-22 19:42:57 +08:00
Ted Xu
41646c8439
feat: integrate new deltalog format ( #35522 )
...
See #34123
---------
Signed-off-by: Ted Xu <ted.xu@zilliz.com>
2024-08-20 19:06:56 +08:00
smellthemoon
80a7c78f28
enhance: import supports null in parquet and json formats ( #35558 )
...
#31728
---------
Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
2024-08-20 16:50:55 +08:00
XuanYang-cn
967f38672a
enhance: Add integration tests for l0 ( #35429 )
...
See also: #34796
---------
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2024-08-19 10:56:54 +08:00
cai.zhang
1bbf7a3c0e
enhance: Optimize the use of locks and avoid double flush clustering buffer writer ( #35486 )
...
issue: #35436
---------
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-16 02:24:58 +08:00
wei liu
c0200eec39
enhance: limit getSegmentInfo batch size to avoid excced grpc message limit ( #35394 )
...
issue: #35395
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-08-15 19:17:00 +08:00
CharlesFeng
b4b1ed6910
fix: before exiting, make sure the goroutine has exited ( #35411 )
...
https://github.com/milvus-io/milvus/issues/35409
Signed-off-by: fengjun2016 <jornfeng@gmail.com>
2024-08-15 18:16:53 +08:00
cai.zhang
196b343a94
fix: Fix data race for clustering compaction ( #35435 )
...
issue: #35436
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-13 17:10:20 +08:00
cai.zhang
aaab827a16
fix: Fix the issue of missing stats log after clustering compaction ( #35266 )
...
issue: #35265
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-08 14:24:17 +08:00
yihao.dai
a4439cc911
enhance: Implement flusher in streamingNode ( #34942 )
...
- Implement flusher to:
- Manage the pipelines (creation, deletion, etc.)
- Manage the segment write buffer
- Manage sync operation (including receive flushMsg and execute flush)
- Add a new `GetChannelRecoveryInfo` RPC in DataCoord.
- Reorganize packages: `flushcommon` and `datanode`.
issue: https://github.com/milvus-io/milvus/issues/33285
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-08-02 18:30:23 +08:00
congqixia
095d77269b
enhance: Set WatchProgress for channel operation progress response ( #35183 )
...
The watch progress is always zero in CheckChannelOperationProgress
response, which is meaningless and confusing. This PR set progress value
in rpc response to fix this problem.
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-08-02 12:06:14 +08:00
wayblink
95462668ca
enhance: unify time in clustering compaction task to unix ( #35167 )
...
#34495
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-08-02 10:30:19 +08:00
zhenshan.cao
aa247f192d
enhance: remove unused code for StorageV2 ( #35132 )
...
issue: https://github.com/milvus-io/milvus/issues/34168
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2024-08-01 12:08:13 +08:00
cai.zhang
9412002d7d
fix: Fix data race for clustering buffer writer ( #35145 )
...
issue: #34495
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-08-01 11:20:13 +08:00
congqixia
de8a266d8a
enhance: Enable linux code checker ( #35084 )
...
See also #34483
---------
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-30 15:53:51 +08:00
wei liu
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 ( #34394 )
...
issue: #34252
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
2024-07-29 11:31:51 +08:00
cai.zhang
2372452fac
enhance: Optimized the GC logic to ensure that memory is released in time ( #34949 )
...
issue: #34703
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-28 23:53:47 +08:00
cai.zhang
ea13b97b6a
fix: Fix bug for block clustering compaction ( #35019 )
...
issue: #34703
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-26 11:33:58 +08:00
congqixia
e8038a76e1
fix: Remove timeout in datanode watch ctx ( #35011 )
...
See also #35008
Use tickle timeout logic instead of hardcode context timeout
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2024-07-25 22:17:45 +08:00
cai.zhang
260a6e2ba9
enhance: Send flush signal when the water level reaches the high watermark ( #34907 )
...
issue: #30633
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-23 10:29:42 +08:00
cai.zhang
4c45bc412f
enhance: Add integration test for clustering compaction ( #34881 )
...
issue: #34792
---------
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-23 10:13:43 +08:00
cai.zhang
643b9d521c
fix: Fix the issue of concurrent packing of the same segment ( #34840 )
...
issue: #34703
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
2024-07-23 10:11:49 +08:00
yihao.dai
8aab6cbfac
enhance: Organize the common modules of streamingNode and dataNode ( #34773 )
...
1. Move the common modules of streamingNode and dataNode to flushcommon
2. Add new GetVChannels interface for rootcoord
issue: https://github.com/milvus-io/milvus/issues/33285
---------
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2024-07-22 11:33:51 +08:00
wayblink
d294fdd060
enhance: add ut for clustering_compactor ( #34852 )
...
issue: #34792
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2024-07-21 19:29:42 +08:00
SimFG
16c002b51b
enhance: unified channel name generation and conversion ( #34805 )
...
/kind improvement
- issue: #34804
Signed-off-by: SimFG <bang.fu@zilliz.com>
2024-07-20 23:11:39 +08:00
