Commit Graph

143 Commits

Author SHA1 Message Date
wayblink
6a722396bd
Integration test framework (#21283)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2023-01-12 19:49:40 +08:00
jaime
73d8509600
Improve ListSegments performance for datacoord catalog (#21477)
Signed-off-by: jaime <yun.zhang@zilliz.com>
2023-01-06 14:33:36 +08:00
cai.zhang
aa203acfb3
Low IndexCoord weight (#21548)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-06 14:21:37 +08:00
cai.zhang
e5f408dceb
Merge IndexCoord and DataCoord (#21267)
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2023-01-04 19:37:36 +08:00
Jiquan Long
ff2a68e65a
Fix collection not exist when tried to do recovery (#21471)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2023-01-04 16:37:35 +08:00
bigsheeper
fc10c74005
Use channel-cp as seekPosition when FromDmlCPLoadDelete (#21110)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-12-13 16:17:22 +08:00
aoiasd
de0ab9e2cf
Refactor showConfigurations to allow return global config rather than only return config of this component (#21063)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>

Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2022-12-09 14:31:21 +08:00
Enwei Jiao
89b810a4db
Refactor all params into ParamItem (#20987)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-12-07 18:01:19 +08:00
jaime
039e9ce4bb
Refine msgstream interface (#20832)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-12-05 20:55:17 +08:00
Ten Thousand Leaves
c373b44c2c
Verify and adjust segment # of rows before saving and passing SegmentInfo (#20837)
issue: #18120

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-12-01 16:31:16 +08:00
Enwei Jiao
2ecdb4ba4a
Etcd config source support TLS (#20874)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-30 18:23:15 +08:00
SimFG
dffcd974ff
Make sure that the segment path matches the segment id (#20677)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-11-18 15:35:09 +08:00
Enwei Jiao
c05b9ad539
Add event dispatcher for config (#20393)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-17 18:59:09 +08:00
bigsheeper
fc15789da9
Ensure compatibility of channel seek position and move syncPeriod to config (#20504)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-11-12 21:09:04 +08:00
congqixia
c21497fbb4
Filter fake segments for recovery info (#20493)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2022-11-11 18:03:05 +08:00
bigsheeper
cd19d99ad7
Add channel level checkpoint (#20350)
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2022-11-10 22:13:04 +08:00
Enwei Jiao
956c5e1b9d
Make Params singleton (#20088)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-11-04 14:25:38 +08:00
XuanYang-cn
22fb1d3b93
Remove dup mock_kv.go and mock_txn_kv.go (#20267)
Replace them with mockery generated mocks in kv/mocks

See also: #19289

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-11-03 14:41:35 +08:00
Xiaofan
11efa0bb5f
Fix compacted segment not dropped (#20128)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-10-27 17:15:32 +08:00
jaime
59bcbf0cf6
Support check health for the service level (#19779)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-18 13:39:26 +08:00
SimFG
a55f739608
Separate public proto files (#19782)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-10-16 20:49:27 +08:00
Enwei Jiao
667982a239
Fix static-check (#19707)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-10-11 17:59:22 +08:00
XuanYang-cn
28e949cf31
Add GetDataVChanPositions (#19657)
The original GetVChanPositions are specialized for QueryCoord,
the recovery info needed by DataNode shouldn't take `indexed` state
into account.

This PR splits GetVChanPositions into GetDataVChanPositions and
GetQueryVChanPositions.

See also: #19653

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2022-10-11 16:51:22 +08:00
jaime
31db01b6ae
Support TTL for collection level (#19383)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-10-10 20:31:22 +08:00
wayblink
85e04d845c
Move GetComponentStates to milvus.proto, make it an external interface (#19594)
Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-10-10 15:55:22 +08:00
wayblink
e7c2500c71
Implement GetPersistentSegmentInfo through datacoord (#19205)
Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-10-08 11:51:02 +08:00
wayblink
54e2c079c4
MEP-30:Support basic coordinators active-standby mechanism #16901 (#19166)
Signed-off-by: wayblink <anyang.wang@zilliz.com>

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-09-29 18:35:02 +08:00
yah01
2dca3688a0
Fix data race in DataCoord unit test (#19469)
Refine the DataCoord initialization

Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-09-27 18:04:53 +08:00
Ten Thousand Leaves
b30c9d4f60
Support Bulk Load in Milvus 2.2 (#18982)
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-09-26 18:06:54 +08:00
jaime
91df8f2d6a
Use log id instead of log path of the binlog in metastore (#19123)
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
2022-09-25 15:56:51 +08:00
yah01
ee9d634a43
Compaction requires the source segments indexed (#19368)
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-09-22 18:34:50 +08:00
SimFG
d7f38a803d
Separate some proto files (#19218)
Signed-off-by: SimFG <bang.fu@zilliz.com>

Signed-off-by: SimFG <bang.fu@zilliz.com>
2022-09-16 16:56:49 +08:00
yah01
11b352c3ae
Guarantee flushed segments are all indexed (#19062)
Signed-off-by: yah01 <yang.cen@zilliz.com>

Signed-off-by: yah01 <yang.cen@zilliz.com>
2022-09-16 11:32:48 +08:00
cai.zhang
c924f73105
Refactor for IndexCoord to support cloud (#18643)
Co-authored-by: Zach41 <zongmei.zhang@zilliz.com>
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>

Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
Co-authored-by: Zach41 <zongmei.zhang@zilliz.com>
2022-08-25 15:48:54 +08:00
Enwei Jiao
d9c7519107
refactor compaction concurency logic (#18660)
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>

Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
2022-08-23 15:50:52 +08:00
Jiquan Long
641afae22f
Watch channels with start positions (#18744)
Signed-off-by: longjiquan <jiquan.long@zilliz.com>

Signed-off-by: longjiquan <jiquan.long@zilliz.com>
2022-08-22 14:42:52 +08:00
codeman
70254c3ec5
Unified catalog interface for segment (#18289) (#18290)
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Signed-off-by: kejiang <ke.jiang@zilliz.com>
Co-authored-by: kejiang <ke.jiang@zilliz.com>
2022-08-20 10:24:51 +08:00
aoiasd
8a70f2badb
Add ShowConfigurations for all remaining session (#18594)
Signed-off-by: aoiasd <zhicheng.yue@zilliz.com>
2022-08-12 13:20:39 +08:00
xing.zhao
d90313a1a8
Use chunkManager rather than minio.Client in datanode garbage collection (#18092)
Signed-off-by: xingzhao <xing.zhao@zilliz.com>

Co-authored-by: xingzhao <xing.zhao@zilliz.com>
2022-07-22 22:10:28 +08:00
Jeng.Gwan
638f6c36e9
Support to get real row count of segment (#18115)
Signed-off-by: xaxys <zheng.guan@zilliz.com>
2022-07-18 09:58:28 +08:00
zhenshan.cao
72910d65e0
Fix inconsistency in DataCoord (#18170) (#18171)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2022-07-08 13:48:21 +08:00
cai.zhang
0ad15a22c8
IndexCoord handle events correctly (#17878)
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
2022-07-07 14:44:21 +08:00
Ten Thousand Leaves
f0b036a35a
Move bulk load segment lock happen early (#17612)
issue: #17600
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-06-27 13:56:17 +08:00
wayblink
074ec3060a
Support return dropped segments info in GetSegmentInfo rpc (#17617)
Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-06-17 18:24:12 +08:00
wayblink
eb5b0b7fc8
Move SegmentInfo out of VchannelInfo, leave Id only to decrease kv size. Get complete SegmentInfo through RPC (#17441)
Resolves: #17233 #16047

Signed-off-by: wayblink <anyang.wang@zilliz.com>
2022-06-16 12:00:10 +08:00
Xiaofan
54aeb077e9
Refine compaction selection poliy (#17486)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2022-06-15 23:14:10 +08:00
cai.zhang
ea5041aec2
Acquiring the segment reference lock on task level (#17544)
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
2022-06-15 21:38:10 +08:00
Ten Thousand Leaves
65ab847815
Add segment reference lock for bulk load (#17457)
So we can support DataQueryable and DataIndexed states

issue: #16607
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
2022-06-15 12:20:10 +08:00
groot
f31be941bb
Fix bulkload row count bug for multi datanodes (#17384)
Signed-off-by: yhmo <yihua.mo@zilliz.com>
2022-06-07 16:56:07 +08:00
shaoyue
76eaa3fc50
Support access s3 through AWS IAM role (#17292)
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
2022-06-02 19:42:03 +08:00