test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-04 21:09:06 +08:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity
18,728 Commits 96 Branches 105 Tags 186 MiB
5474bce9d2
Commit Graph

60 Commits

Author SHA1 Message Date
XuanYang-cn
ae180d1628
enhance: Change ChannelManager to interface (#29300) 
Rewrite cluster test
issue: #28854

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-12-25 19:24:46 +08:00
SimFG
67ab0e424b
fix: Clean the compaction plan info to avoid the object leak (#29365) 
issue: #29296

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-12-22 12:00:43 +08:00
XuanYang-cn
2c62bcbbf6
fix: Fix some left comments (#29027) 
See also: #28814

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-12-13 18:48:39 +08:00
aoiasd
3c32ba2407
enhance: pack datacoord Cluster and SessionManager with interface and mock them (#28869) 
relate: https://github.com/milvus-io/milvus/issues/28861
https://github.com/milvus-io/milvus/issues/28854

---------

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-12-11 17:52:37 +08:00
XuanYang-cn
5bac7f7897
fix: Fix L0 compaction in datacoord (#28814) 
See also: #27606

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-12-05 18:44:37 +08:00
congqixia
3f9da2b8a0
fix: Check whether compaction plan exist before update (#28826) 
Fix: #28825

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-29 18:54:27 +08:00
XuanYang-cn
321c5c32e3
fix: Separate schedule and check results loop (#28692) 
This PR:

- Separates compaction scheduler and check results loop So that slow in
check-loop doesn't influence execution.

- Cleans compaction tasks when drop a vchannel so dropped-channel's
compaction tasks won't be checked over and over again.

  - Skips meta change when meta's already changed, avoid panic
  - Remove not inuse injectDone(bool) parameter

See also: #28628, #28209

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-29 10:50:29 +08:00
XuanYang-cn
9b371067d2
feat: Add Compaction views and triggers (#27906) 
- Add Compaction l0 views
- Add Compaction scheduler
- Add Compaction triggerv2
- Add Compaction view manager

See also: #27606

---------

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-23 17:30:25 +08:00
XuanYang-cn
0be22ccacd
Fix compaction unable to notify datanode (#28409) 
See also: #28214, #28368

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-14 16:38:18 +08:00
XuanYang-cn
40d5c902b6
Enable getting multiple segments in plan result (#28350) 
Compaction plan result contained one segment for one plan. For l0
compaction would write to multiple segments, this PR expand the segments
number in plan results and refactor some names for readibility.

- Name refactory: - CompactionStateResult -> CompactionPlanResult -
CompactionResult -> CompactionSegment

See also: #27606

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-14 15:56:19 +08:00
congqixia
c41df18b6d
Add compaction meta in SyncSegmentsRequest (#28199) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-11-07 10:06:17 +08:00
XuanYang-cn
22c089894e
Add scheduler (#27938) 
See also: #27606

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-11-07 03:18:18 +08:00
yah01
90e2c63d9e
Fix getting incorrect CPU num (#28146) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
 2023-11-06 06:02:16 +08:00
SimFG
26f06dd732
Format the code (#27275) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-09-21 09:45:27 +08:00
cai.zhang
d7cd1f2a6d
Fix bug for get index state with compaction segment (#26822) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-09-04 17:29:48 +08:00
XuanYang-cn
9598a8b236
Fix timeout task never release queue (#26593) 
See also: #26413, #26566

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-30 11:12:27 +08:00
XuanYang-cn
35f3e263cd
Fix lock usage in DataCoord compaction (#26032) 
See also: #26009

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2023-08-01 08:55:04 +08:00
congqixia
37636b2f4d
Add compaction worker max parallel config item (#25834) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-24 10:17:00 +08:00
congqixia
8d343bf75a
Make compaction rpc timeout and parallel maxium configurable (#25672) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-07-18 14:25:20 +08:00
Enwei Jiao
66fdc71479
Refactor logs in DataCoord & DataNode (#25574) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-07-14 15:56:31 +08:00
xige-16
33c2012675
Add more metrics (#25081) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2023-06-26 17:52:44 +08:00
Xiaofan
f57fe6d70b
Add compaction log (#24976) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2023-06-19 14:18:41 +08:00
congqixia
41af0a98fa
Use go-api/v2 for milvus-proto (#24770) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-06-09 01:28:37 +08:00
MrPresent-Han
b517bc9e6a
refine balance mechanism including:(#23454) (#23763) (#23791) 
1. balance granuity to replica to avoid influence unrelated replicas
2. avoid balance back and forth

Signed-off-by: MrPresent-Han <jamesharden11122@gmail.com>
 2023-05-04 12:22:40 +08:00
jaime
c9d0c157ec
Move some modules from internal to public package (#22572) 
Signed-off-by: jaime <yun.zhang@zilliz.com>
 2023-04-06 19:14:32 +08:00
SimFG
4a90490a67
Fix the segment not found error (#22772) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2023-03-17 17:27:56 +08:00
Enwei Jiao
697dedac7e
Use cockroachdb/errors to replace other error pkg (#22390) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2023-02-26 11:31:49 +08:00
zhenshan.cao
e768437681
Correct usage of Timer and Ticker (#22228) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2023-02-23 18:59:45 +08:00
aoiasd
1894766235
Update datacoord compaction plan after datanode update plan to ensure consistency (#22143) 
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
 2023-02-15 16:00:33 +08:00
congqixia
7adabe09f1
Fix Alloc timstamp failure blocks compaction queue forever (#22039) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2023-02-07 19:02:31 +08:00
cai.zhang
e5f408dceb
Merge IndexCoord and DataCoord (#21267) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2023-01-04 19:37:36 +08:00
Enwei Jiao
89b810a4db
Refactor all params into ParamItem (#20987) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-12-07 18:01:19 +08:00
congqixia
5e49b095f5
Fix compaction handler may double release and block forever (#21015) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-12-06 20:09:18 +08:00
SimFG
6d9d24b4ca
Make memory and etcd data as consistent as possible (#20683) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-11-24 10:19:12 +08:00
Ten Thousand Leaves
a8c7199a52
Fix segment_num, stored_rows_num and stored_rows_count metrics (#20704) 
TODO: better track segment with state other than flushed and dropped.

/kind bug

issue: #20146, #20145
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-11-22 19:21:13 +08:00
Bingyi Sun
2390095232
Fix load uses compacted segments' binlogs (#20655) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Signed-off-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-11-17 20:37:10 +08:00
Ten Thousand Leaves
480a2b8632
Improve compaction logs (#20490) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-11-15 11:13:07 +08:00
Ten Thousand Leaves
5708352a80
Improve DataCoord and DataNode logs (#20265) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-11-03 12:15:35 +08:00
Xiaofan
11efa0bb5f
Fix compacted segment not dropped (#20128) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-10-27 17:15:32 +08:00
jaime
d8ca56e084
Fix empty segment handoff fails (#20073) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-10-26 16:49:31 +08:00
Xiaofan
192151bc11
Refine Data Log (#19996) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-10-25 19:31:30 +08:00
yah01
e8ca754dca
Fix compaction produces empty segment (#19936) 
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
 2022-10-20 22:49:28 +08:00
SimFG
a55f739608
Separate public proto files (#19782) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-10-16 20:49:27 +08:00
XuanYang-cn
a0ce533727
Add a SyncSegments to sync meta between DN and DC (#19298) 
See also: #19072

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-09-27 16:02:53 +08:00
SimFG
d7f38a803d
Separate some proto files (#19218) 
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-09-16 16:56:49 +08:00
jaime
362c663b22
Remove deprecated inner compaction type (#19024) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-09-07 19:16:34 +08:00
Enwei Jiao
096149e33f
fix GetCompactionState hang (#18829) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-08-26 14:22:55 +08:00
cai.zhang
c924f73105
Refactor for IndexCoord to support cloud (#18643) 
Co-authored-by: Zach41 <zongmei.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Zach41 <zongmei.zhang@zilliz.com>
 2022-08-25 15:48:54 +08:00
Enwei Jiao
d9c7519107
refactor compaction concurency logic (#18660) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-08-23 15:50:52 +08:00
cai.zhang
ed94ecf847
Segments can't be compacted when they have reference lock (#17649) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-20 21:56:12 +08:00