test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-12 13:15:09 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity
20,429 Commits 99 Branches 105 Tags 173 MiB
a4439cc911
Commit Graph

1072 Commits

Author SHA1 Message Date
yihao.dai
a4439cc911
enhance: Implement flusher in streamingNode (#34942) 
- Implement flusher to:
  - Manage the pipelines (creation, deletion, etc.)
  - Manage the segment write buffer
  - Manage sync operation (including receive flushMsg and execute flush)
- Add a new `GetChannelRecoveryInfo` RPC in DataCoord.
- Reorganize packages: `flushcommon` and `datanode`.

issue: https://github.com/milvus-io/milvus/issues/33285

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-08-02 18:30:23 +08:00
congqixia
095d77269b
enhance: Set WatchProgress for channel operation progress response (#35183) 
The watch progress is always zero in CheckChannelOperationProgress
response, which is meaningless and confusing. This PR set progress value
in rpc response to fix this problem.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-08-02 12:06:14 +08:00
wayblink
95462668ca
enhance: unify time in clustering compaction task to unix (#35167) 
#34495

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-08-02 10:30:19 +08:00
zhenshan.cao
aa247f192d
enhance: remove unused code for StorageV2 (#35132) 
issue: https://github.com/milvus-io/milvus/issues/34168

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2024-08-01 12:08:13 +08:00
cai.zhang
9412002d7d
fix: Fix data race for clustering buffer writer (#35145) 
issue: #34495

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-08-01 11:20:13 +08:00
congqixia
de8a266d8a
enhance: Enable linux code checker (#35084) 
See also #34483

---------

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-30 15:53:51 +08:00
wei liu
c45f38aa61
enhance: Update protobuf-go to protobuf-go v2 (#34394) 
issue: #34252

Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-07-29 11:31:51 +08:00
cai.zhang
2372452fac
enhance: Optimized the GC logic to ensure that memory is released in time (#34949) 
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-28 23:53:47 +08:00
cai.zhang
ea13b97b6a
fix: Fix bug for block clustering compaction (#35019) 
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-26 11:33:58 +08:00
congqixia
e8038a76e1
fix: Remove timeout in datanode watch ctx (#35011) 
See also #35008

Use tickle timeout logic instead of hardcode context timeout

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-07-25 22:17:45 +08:00
cai.zhang
260a6e2ba9
enhance: Send flush signal when the water level reaches the high watermark (#34907) 
issue: #30633

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-23 10:29:42 +08:00
cai.zhang
4c45bc412f
enhance: Add integration test for clustering compaction (#34881) 
issue: #34792

---------

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-23 10:13:43 +08:00
cai.zhang
643b9d521c
fix: Fix the issue of concurrent packing of the same segment (#34840) 
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-23 10:11:49 +08:00
yihao.dai
8aab6cbfac
enhance: Organize the common modules of streamingNode and dataNode (#34773) 
1. Move the common modules of streamingNode and dataNode to flushcommon
2. Add new GetVChannels interface for rootcoord

issue: https://github.com/milvus-io/milvus/issues/33285

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-22 11:33:51 +08:00
wayblink
d294fdd060
enhance: add ut for clustering_compactor (#34852) 
issue: #34792

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-21 19:29:42 +08:00
SimFG
16c002b51b
enhance: unified channel name generation and conversion (#34805) 
/kind improvement
- issue: #34804

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-07-20 23:11:39 +08:00
wayblink
c79d1af390
enhance: Add compaction task slot usage logic (#34581) 
#34544

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-07-18 10:27:41 +08:00
cai.zhang
0c0ca4cf0e
fix: Fix bug where binlogs already flushed with new segment during pack (#34762) 
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-18 09:15:48 +08:00
yihao.dai
ca758c36cc
enhance: Pre-allocate ids for compaction (#34187) 
This PR removes the dependency of compaction on the ID allocator by
pre-allocating the logID and segmentID.

issue: https://github.com/milvus-io/milvus/issues/33957

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-17 13:23:42 +08:00
cai.zhang
17c96e1bde
fix: Fix the bug that caused small segment flush frequently (#34725) 
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-17 12:15:40 +08:00
cai.zhang
fbc10ac695
fix:Reset flushed row num after pack segment for clustering compaction (#34702) 
issue: #34703

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-16 15:37:38 +08:00
SimFG
b58a5617ef
enhance: add the seal segment when dispatch delete msgs (#34565) 
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-07-10 19:14:51 +08:00
SimFG
5016038781
enhance: release the record in delete codec and add some log for compaction (#34454) 
/kind improvement

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2024-07-09 15:40:17 +08:00
yihao.dai
e15ac2b472
fix: Fix incorrect segment num rows (#34441) 
Repeated calls to UpdateStatistics, this PR correct it.

issue: https://github.com/milvus-io/milvus/issues/34440

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-08 20:00:14 +08:00
yihao.dai
4e5f1d5f75
enhance: Pre-allocate ids for import (#33958) 
The import is dependent on syncTask, which in turn relies on the
allocator. This PR pre-allocate the necessary IDs for import syncTask.

issue: https://github.com/milvus-io/milvus/issues/33957

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-07 21:26:14 +08:00
yihao.dai
43fd8d19c2
enhance: Check segment existence when FlushSegments and add some key logs (#34438) 
Check if the segment exists during FlushSegments and add some key logs
in write path.

issue: https://github.com/milvus-io/milvus/issues/34255

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-06 08:50:11 +08:00
jaime
21fc5f5d46
enhance: Remove datanode reporting TT based on MQ implementation (#34421) 
issue: #34420

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-05 15:48:09 +08:00
yihao.dai
0b404bff22
fix: Correct the update logic of timerecorder (#34339) 
Correct the update logic of timerecorder in the flowgraph to avoid false
failure: "some node(s) haven't received input".

issue: https://github.com/milvus-io/milvus/issues/34337

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-07-04 16:34:17 +08:00
XuanYang-cn
8a2be8a457
fix: DataNode might OOM by estimating based on MemorySize (#34201) 
See also: #34136

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-07-04 10:16:17 +08:00
cai.zhang
2e434e4453
fix: Only load or release Flushed segment in datanode meta (#34390) 
issue: #34376 , #34379 , #34375

---------

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-03 17:52:09 +08:00
cai.zhang
8165961223
enhance: Optimeize clustering compaction (#34313) 
issue: #30633

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-03 15:14:08 +08:00
cai.zhang
4cf1a358ba
fix: Sync the sealed and flushed segments to datanode (#34301) 
issue: #33696

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-07-01 22:42:08 +08:00
jaime
d6afb31b94
enhance: make subfunctions of datanode component modular (#33992) 
issue: #33994

also remove deprecated channel manager based on the etcd implementation

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-07-01 14:46:07 +08:00
wayblink
e5d691d854
Use new stream segment reader in clustering compaction (#34232) 
#32939

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-06-30 20:26:07 +08:00
wayblink
fbe3231b1f
fix: fix error ignore in compactor (#34169) 
#34170

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-06-26 10:24:03 +08:00
jaime
9630974fbb
enhance: move rocksmq from internal to pkg module (#33881) 
issue: #33956

Signed-off-by: jaime <yun.zhang@zilliz.com>
 2024-06-25 21:18:15 +08:00
cai.zhang
c65f41dc60
fix: Only sync flushed segments to datanode (#34156) 
issue: #33540

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-25 20:46:07 +08:00
congqixia
962a5446f8
enhance: Add ctx in SyncTask.Run to be cancellable (#34042) 
Related to #33716

This PR add context param in SyncTask.Run execution functions to make it
cancellable from the caller.

This make it possible to cancel task when datanode/data sync service is
beeing shut down.

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-25 14:22:04 +08:00
congqixia
506a915272
fix: Deep copy ImportTask.segmentsInfo to prevent data race (#34090) 
See also #34089

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-25 10:06:02 +08:00
congqixia
b5c9a7364b
fix: Prevent remove new growing L1 segment when SyncSegments (#34056) 
Related to #34018

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2024-06-24 10:18:01 +08:00
yihao.dai
6c1d815894
enhance: Remove the unused compaction logic from shard (#33932) 
1. Remove the `compactTo` field in `SegmentInfo`.
2. Remove the target segment not match and its retry logic in
`SyncManager`.

issue: https://github.com/milvus-io/milvus/issues/32809

---------

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-23 21:12:01 +08:00
wayblink
380d3f4469
fix: Fix memory buffer error & some renaming (#33850) 
#30633

---------

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-06-21 17:30:01 +08:00
XuanYang-cn
04edb07d82
enhance: Add deltaRowCount in l0 compaction (#33997) 
See also: #33998

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2024-06-20 17:46:01 +08:00
cqy123456
dc4437ff82
enhance: use segment id and type to register in MmapChunkManager and opt malloc in variableChunk (#33993) 
issue: https://github.com/milvus-io/milvus/issues/32984

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-20 17:42:02 +08:00
wei liu
31ef0a1fe8
enhance: Add trace for bf cost in l0 compactor (#33860) 
Signed-off-by: Wei Liu <wei.liu@zilliz.com>
 2024-06-20 10:10:05 +08:00
smellthemoon
2a1356985d
enhance: support null in go payload (#32296) 
#31728

---------

Signed-off-by: lixinguo <xinguo.li@zilliz.com>
Co-authored-by: lixinguo <xinguo.li@zilliz.com>
 2024-06-19 17:08:00 +08:00
wayblink
5cb0760187
fix: Small fixs of major compaction (#33929) 
#30633

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2024-06-18 16:53:58 +08:00
cqy123456
32f685ff12
enhance: growing segment support mmap (#32633) 
issue: https://github.com/milvus-io/milvus/issues/32984

Signed-off-by: cqy123456 <qianya.cheng@zilliz.com>
 2024-06-18 14:42:00 +08:00
cai.zhang
95148866ed
fix: Don't remove growing L0 segment in datanode metacache (#33829) 
issue: #33540 
1. gorwing L0 segments is invisible to datacoord.
2. flushed L0 segments need to clean by datacoord.

Signed-off-by: Cai Zhang <cai.zhang@zilliz.com>
 2024-06-17 10:09:57 +08:00
yihao.dai
1a9ab52f66
enhance: Ensure the idempotency of compaction task (#33872) 
/kind enhancement

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2024-06-16 22:09:57 +08:00