test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-04 21:09:06 +08:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity
20,710 Commits 96 Branches 105 Tags 186 MiB
2694551757
Commit Graph

1093 Commits

Author SHA1 Message Date
XuanYang-cn
2687747278
fix: Set an empty segment if compaction deleted all inserts (#36044) 
See also: #36038

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-09-09 14:23:05 +08:00
Chun Han
e480b103bd
feat: supporing hybrid search group_by (#35982) 
related: #35096

Signed-off-by: MrPresent-Han <chun.han@gmail.com>
Co-authored-by: MrPresent-Han <chun.han@gmail.com>
 2024-09-08 17:09:04 +08:00
SimFG
5247631289
fix: fill the metric type field in the LoadMetaInfo object (#35962) 
- issue: #35960

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-09-05 20:50:23 -07:00
cai.zhang
90bdb171ab
fix: Fix data race for clustering compaction writer (#35957) 
issue: #35950

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-05 04:07:10 +08:00
CharlesFeng
6eb8b3f745
fix: err degenerated to a new variable (#35891) 
https://github.com/milvus-io/milvus/issues/35890

Signed-off-by: fengjun2016 <jornfeng@gmail.com>
 2024-09-04 14:57:04 +08:00
yihao.dai
6fd33285e1
fix: Fix compile error (#35901) 
/kind improvement

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-09-02 14:50:35 +08:00
cai.zhang
2c9bb4dfa3
feat: Support stats task to sort segment by PK (#35054) 
issue: #33744 

This PR includes the following changes:
1. Added a new task type to the task scheduler in datacoord: stats task,
which sorts segments by primary key.
2. Implemented segment sorting in indexnode.
3. Added a new field `FieldStatsLog` to SegmentInfo to store token index
information.

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-09-02 14:19:03 +08:00
yihao.dai
1413ffe9b1
enhance: Rename preAllocatedSegments (#35871) 
Rename `preAllocatedSegments` to `preAllocatedSegmentIDs` to avoid
confusion.

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-09-01 17:09:01 +08:00
XuanYang-cn
323400c190
enhance: Enable to write multiple segments in mix compactor (#35705) 
Prevent segments to be written larger than maxSize * expansionRate

See also: #35584

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-08-30 11:29:01 +08:00
Zhen Ye
99dff06391
enhance: using streaming service in insert/upsert/flush/delete/querynode (#35406) 
issue: #33285

- using streaming service in insert/upsert/flush/delete/querynode
- fixup flusher bugs and refactor the flush operation
- enable streaming service for dml and ddl
- pass the e2e when enabling streaming service
- pass the integration tst when enabling streaming service

---------

Signed-off-by: chyezh <chyezh@outlook.com>
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-29 10:03:08 +08:00
yihao.dai
9868fe4e6c
fix: Fix panic due to empty candidate import segments (#35673) 
issue: https://github.com/milvus-io/milvus/issues/35662

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-27 17:08:59 +08:00
congqixia
59387f0b3a
fix: Use correct release func when datanode unwatch a channel (#35655) 
See also #35654

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-23 14:09:03 +08:00
congqixia
ab532ae199
enhance: Add back BF lazy load logic for datanode watch channel (#35646) 
Add back lazy loading statslog when watch dml channel on datanode.

Related to #22994 #27675

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-22 19:42:57 +08:00
Ted Xu
41646c8439
feat: integrate new deltalog format (#35522) 
See #34123

---------

Signed-off-by: Ted Xu <ted.xu@zilliz.com>
 2024-08-20 19:06:56 +08:00
smellthemoon
80a7c78f28
enhance: import supports null in parquet and json formats (#35558) 
#31728

---------

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-08-20 16:50:55 +08:00
XuanYang-cn
967f38672a
enhance: Add integration tests for l0 (#35429) 
See also: #34796

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-08-19 10:56:54 +08:00
cai.zhang
1bbf7a3c0e
enhance: Optimize the use of locks and avoid double flush clustering buffer writer (#35486) 
issue: #35436

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-08-16 02:24:58 +08:00
wei liu
c0200eec39
enhance: limit getSegmentInfo batch size to avoid excced grpc message limit (#35394) 
issue: #35395

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-08-15 19:17:00 +08:00
CharlesFeng
b4b1ed6910
fix: before exiting, make sure the goroutine has exited (#35411) 
https://github.com/milvus-io/milvus/issues/35409

Signed-off-by: fengjun2016 <jornfeng@gmail.com>
 2024-08-15 18:16:53 +08:00
cai.zhang
196b343a94
fix: Fix data race for clustering compaction (#35435) 
issue: #35436

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-08-13 17:10:20 +08:00
cai.zhang
aaab827a16
fix: Fix the issue of missing stats log after clustering compaction (#35266) 
issue: #35265

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-08-08 14:24:17 +08:00
yihao.dai
a4439cc911
enhance: Implement flusher in streamingNode (#34942) 
- Implement flusher to:
  - Manage the pipelines (creation, deletion, etc.)
  - Manage the segment write buffer
  - Manage sync operation (including receive flushMsg and execute flush)
- Add a new `GetChannelRecoveryInfo` RPC in DataCoord.
- Reorganize packages: `flushcommon` and `datanode`.

issue: https://github.com/milvus-io/milvus/issues/33285

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-02 18:30:23 +08:00
congqixia
095d77269b
enhance: Set WatchProgress for channel operation progress response (#35183) 
The watch progress is always zero in CheckChannelOperationProgress
response, which is meaningless and confusing. This PR set progress value
in rpc response to fix this problem.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-02 12:06:14 +08:00
wayblink
95462668ca
enhance: unify time in clustering compaction task to unix (#35167) 
#34495

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-08-02 10:30:19 +08:00
zhenshan.cao
aa247f192d
enhance: remove unused code for StorageV2 (#35132) 
issue: https://github.com/milvus-io/milvus/issues/34168

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-08-01 12:08:13 +08:00
cai.zhang
9412002d7d
fix: Fix data race for clustering buffer writer (#35145) 
issue: #34495

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-08-01 11:20:13 +08:00
congqixia
de8a266d8a
enhance: Enable linux code checker (#35084) 
See also #34483

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-30 15:53:51 +08:00
wei liu
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 (#34394) 
issue: #34252

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-29 11:31:51 +08:00
cai.zhang
2372452fac
enhance: Optimized the GC logic to ensure that memory is released in time (#34949) 
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-28 23:53:47 +08:00
cai.zhang
ea13b97b6a
fix: Fix bug for block clustering compaction (#35019) 
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-26 11:33:58 +08:00
congqixia
e8038a76e1
fix: Remove timeout in datanode watch ctx (#35011) 
See also #35008

Use tickle timeout logic instead of hardcode context timeout

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-25 22:17:45 +08:00
cai.zhang
260a6e2ba9
enhance: Send flush signal when the water level reaches the high watermark (#34907) 
issue: #30633

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-23 10:29:42 +08:00
cai.zhang
4c45bc412f
enhance: Add integration test for clustering compaction (#34881) 
issue: #34792

---------

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-23 10:13:43 +08:00
cai.zhang
643b9d521c
fix: Fix the issue of concurrent packing of the same segment (#34840) 
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-23 10:11:49 +08:00
yihao.dai
8aab6cbfac
enhance: Organize the common modules of streamingNode and dataNode (#34773) 
1. Move the common modules of streamingNode and dataNode to flushcommon
2. Add new GetVChannels interface for rootcoord

issue: https://github.com/milvus-io/milvus/issues/33285

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-22 11:33:51 +08:00
wayblink
d294fdd060
enhance: add ut for clustering_compactor (#34852) 
issue: #34792

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-21 19:29:42 +08:00
SimFG
16c002b51b
enhance: unified channel name generation and conversion (#34805) 
/kind improvement
- issue: #34804

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-07-20 23:11:39 +08:00
wayblink
c79d1af390
enhance: Add compaction task slot usage logic (#34581) 
#34544

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-18 10:27:41 +08:00
cai.zhang
0c0ca4cf0e
fix: Fix bug where binlogs already flushed with new segment during pack (#34762) 
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-18 09:15:48 +08:00
yihao.dai
ca758c36cc
enhance: Pre-allocate ids for compaction (#34187) 
This PR removes the dependency of compaction on the ID allocator by
pre-allocating the logID and segmentID.

issue: https://github.com/milvus-io/milvus/issues/33957

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-17 13:23:42 +08:00
cai.zhang
17c96e1bde
fix: Fix the bug that caused small segment flush frequently (#34725) 
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-17 12:15:40 +08:00
cai.zhang
fbc10ac695
fix:Reset flushed row num after pack segment for clustering compaction (#34702) 
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-16 15:37:38 +08:00
SimFG
b58a5617ef
enhance: add the seal segment when dispatch delete msgs (#34565) 
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-07-10 19:14:51 +08:00
SimFG
5016038781
enhance: release the record in delete codec and add some log for compaction (#34454) 
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-07-09 15:40:17 +08:00
yihao.dai
e15ac2b472
fix: Fix incorrect segment num rows (#34441) 
Repeated calls to UpdateStatistics, this PR correct it.

issue: https://github.com/milvus-io/milvus/issues/34440

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-08 20:00:14 +08:00
yihao.dai
4e5f1d5f75
enhance: Pre-allocate ids for import (#33958) 
The import is dependent on syncTask, which in turn relies on the
allocator. This PR pre-allocate the necessary IDs for import syncTask.

issue: https://github.com/milvus-io/milvus/issues/33957

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-07 21:26:14 +08:00
yihao.dai
43fd8d19c2
enhance: Check segment existence when FlushSegments and add some key logs (#34438) 
Check if the segment exists during FlushSegments and add some key logs
in write path.

issue: https://github.com/milvus-io/milvus/issues/34255

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-06 08:50:11 +08:00
jaime
21fc5f5d46
enhance: Remove datanode reporting TT based on MQ implementation (#34421) 
issue: #34420

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-05 15:48:09 +08:00
yihao.dai
0b404bff22
fix: Correct the update logic of timerecorder (#34339) 
Correct the update logic of timerecorder in the flowgraph to avoid false
failure: "some node(s) haven't received input".

issue: https://github.com/milvus-io/milvus/issues/34337

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-04 16:34:17 +08:00
XuanYang-cn
8a2be8a457
fix: DataNode might OOM by estimating based on MemorySize (#34201) 
See also: #34136

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-07-04 10:16:17 +08:00