test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-01 11:29:48 +08:00
Code Issues Actions 11 Packages Projects Releases Wiki Activity
15,706 Commits 94 Branches 105 Tags 184 MiB
3a5aaeb7ad
Commit Graph

561 Commits

Author SHA1 Message Date
jaime
0e95f8a690
Refine indexcoord logs (#18589) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>

Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-08-11 10:00:38 +08:00
jaime
efa5dfaa7b
Fix Parse binlog path failure (#18583) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-08-10 17:04:37 +08:00
Ten Thousand Leaves
12023e8569
Fix a potential nullptr issue with metric (#18540) 
/kind improvement

related to: #18530

but not the root cause

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-08-09 21:12:42 +08:00
SimFG
ff0200210a
Support Role-Based Access Control (#18425) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-08-04 11:04:34 +08:00
congqixia
28aadc988d
Use vchannel name in datanode subName (#18519) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-08-03 17:10:36 +08:00
jaime
c9174d55ba
Refine merge operation during compacting phase (#18399) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-07-28 14:52:31 +08:00
congqixia
e9d777b336
Fix ParseSegmentIDBinlog panicks with bad input (#18413) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-07-26 19:32:30 +08:00
xing.zhao
d90313a1a8
Use chunkManager rather than minio.Client in datanode garbage collection (#18092) 
Signed-off-by: xingzhao <xing.zhao@zilliz.com>

Co-authored-by: xingzhao <xing.zhao@zilliz.com>
 2022-07-22 22:10:28 +08:00
Ten Thousand Leaves
cd2f64d30c
Fix a nil pointer issue (#18343) 
/kind bug

issue: #18270
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-07-20 17:56:30 +08:00
Ten Thousand Leaves
0ebe407221
Make DataNode not panic if stale segment is not found during flush (#18322) 
/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-07-20 15:06:30 +08:00
Jeng.Gwan
638f6c36e9
Support to get real row count of segment (#18115) 
Signed-off-by: xaxys <zheng.guan@zilliz.com>
 2022-07-18 09:58:28 +08:00
Xiaofan
6d82ef8c20
fix garbage collector err handling (#18277) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-07-15 13:54:26 +08:00
congqixia
e5fe46124b
Fix isCompacting flag lost after segment clone (#18248) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-07-13 10:08:28 +08:00
zhenshan.cao
72910d65e0
Fix inconsistency in DataCoord (#18170) (#18171) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-07-08 13:48:21 +08:00
wayblink
43ae6791e6
Make VChannelInfo read compatible with 2.0.2 (#17995) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-07-07 21:58:22 +08:00
cai.zhang
0ad15a22c8
IndexCoord handle events correctly (#17878) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-07-07 14:44:21 +08:00
Ten Thousand Leaves
d4a1e94f32
Fix flush failure caused by empty stale segments (#18097) 
/bug

issue: #18096
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-07-06 18:10:21 +08:00
zhenshan.cao
9bd35dd84e
Refine logs to reduce print (#17759) (#18028) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-07-04 11:10:20 +08:00
zhenshan.cao
0baeb609dc
Fix bugs in monitoring metrics (#17894) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-06-30 17:26:19 +08:00
yah01
ef6859a4d6
Refactor handoff handler (#17806) 
Signed-off-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: xiaofan-luan <xiaofan.luan@zilliz.com>

Co-authored-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-29 23:26:19 +08:00
XuanYang-cn
ff3c4ab2a8
Fix compaction panic (#17840) 
See also: #17823

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-29 16:54:17 +08:00
Ten Thousand Leaves
f0b036a35a
Move bulk load segment lock happen early (#17612) 
issue: #17600
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-06-27 13:56:17 +08:00
bigsheeper
230404499d
Check duplication before adding channel info in DataCoord (#17742) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-06-24 16:30:15 +08:00
Bingyi Sun
05273309e8
Drop empty sealed segment (#17708) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-06-24 15:16:14 +08:00
XuanYang-cn
21b52bfcf2
Enhance channel manager ut (#17682) 
Resolves: #17676

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-23 18:08:14 +08:00
SimFG
50d5d04552
Refine some codes about the datanode (#17720) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-06-23 09:58:14 +08:00
Xiaofan
6fdf88f452
Fix compaction selection policy (#17690) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-22 19:00:14 +08:00
Enwei Jiao
746aeea35b
fix typo (#17668) 
Signed-off-by: Enwei Jiao <enwei.jiao@zilliz.com>
 2022-06-21 18:54:13 +08:00
cai.zhang
ed94ecf847
Segments can't be compacted when they have reference lock (#17649) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-20 21:56:12 +08:00
Xiaofan
516d91470f
Fix Delta log loss after compaction (#17634) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-19 20:12:11 +08:00
Letian Jiang
b15c24a554
Acquire reference lock of compacted from segments during index building (#17619) 
- Accquire reference lock during index building
- Keep dropped segment record

Co-authored-by: Letian Jiang <letian.jiang@zilliz.com>
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-06-18 01:24:11 +08:00
wayblink
074ec3060a
Support return dropped segments info in GetSegmentInfo rpc (#17617) 
Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-06-17 18:24:12 +08:00
cai.zhang
53a49a9255
GetIndexState only get the state generated call CreateIndex (#17549) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-16 20:12:11 +08:00
wayblink
eb5b0b7fc8
Move SegmentInfo out of VchannelInfo, leave Id only to decrease kv size. Get complete SegmentInfo through RPC (#17441) 
Resolves: #17233 #16047

Signed-off-by: wayblink <anyang.wang@zilliz.com>
 2022-06-16 12:00:10 +08:00
Xiaofan
54aeb077e9
Refine compaction selection poliy (#17486) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-15 23:14:10 +08:00
cai.zhang
ea5041aec2
Acquiring the segment reference lock on task level (#17544) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-06-15 21:38:10 +08:00
Ten Thousand Leaves
51a9e54b21
Fix a bug that causes infinite loop (#17550) 
This will cause AddNode() to keep its lock forever so DataCoord would just hang

issue: #17366
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-06-15 20:50:10 +08:00
Bingyi Sun
d4fdfab818
Fix VDC-510 (#17554) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: sunby <bingyi.sun@zilliz.com>
 2022-06-15 19:58:10 +08:00
Ten Thousand Leaves
65ab847815
Add segment reference lock for bulk load (#17457) 
So we can support DataQueryable and DataIndexed states

issue: #16607
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-06-15 12:20:10 +08:00
XuanYang-cn
e6225d923b
Cancel timers while adding new timer (#17511) 
See also: #17335

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-13 14:46:08 +08:00
XuanYang-cn
2b5340581c
Skip remove if reassigns to the original node (#17450) 
Fix ut race

See also: #15966, #17432

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-10 10:54:07 +08:00
Xiaofan
6426e3d037
Split Etcd save operation into small txns (#17386) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-08 15:08:07 +08:00
XuanYang-cn
1dbfebf827
Add debug log in removing assignment (#17427) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-08 10:44:06 +08:00
groot
f31be941bb
Fix bulkload row count bug for multi datanodes (#17384) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-06-07 16:56:07 +08:00
xige-16
b5c11a216d
Alter varChar type params's name to max_length (#17409) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-06-07 15:58:06 +08:00
Xiaofan
af994b5e1c
Fix etcd watch error (#17364) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-06-06 16:26:06 +08:00
XuanYang-cn
c70af73005
Fix datacoord set wrong state for node registering (#17376) 
Fix datacoord datarace

See also: #17335

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-06 15:10:05 +08:00
shaoyue
76eaa3fc50
Support access s3 through AWS IAM role (#17292) 
Signed-off-by: shaoyue.chen <shaoyue.chen@zilliz.com>
 2022-06-02 19:42:03 +08:00
Ten Thousand Leaves
f5e63177d2
Fix an issue with bulk load with multiple DataNodes (#17324) 
issue: #17300
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-06-02 18:54:04 +08:00
XuanYang-cn
42f643e727
Make DataNode release rather than delete when reassign (#17293) 
1. Reassgin now will assign to the original Node if no other nodes
   avaliable
2. Make AddNode balance async: ToRealse + Reassign

See also: #16114, #17270

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-06-02 13:56:04 +08:00
cai.zhang
bcf3b7426a
Add distributed lock for segment refgerence (#16782) 
Signed-off-by: Cai.Zhang <cai.zhang@zilliz.com>
 2022-05-31 16:36:03 +08:00
groot
affcc2acad
Fix bulkload entity number bug (#17278) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-05-31 15:40:04 +08:00
congqixia
55f1e03ca7
Allow DataNode clean vchannel with meta error without panicking (#17247) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-05-27 16:20:00 +08:00
Ten Thousand Leaves
5e1e7a6896
Fix an issue that causes flush hanging (#17169) 
issue: #16628
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-05-25 14:34:00 +08:00
bigsheeper
9eeec4a2d5
Add collection load cache and InvalidateCollMetaCache by collID (#16882) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2022-05-19 10:13:56 +08:00
SimFG
cf08b5aa11
Fix 'TestChannelManager_StateTransfer' ut when etcd is slow (#17082) 
Signed-off-by: SimFG <bang.fu@zilliz.com>
 2022-05-18 17:19:57 +08:00
XuanYang-cn
127dd34b37
Make proxy use roundrobin to choose replica (#17063) 
Fixes: #17055

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-05-17 22:35:57 +08:00
Xiaofan
62658dcda6
Fix pulsar unscubsribe fail because of consumer not found (#16839) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-05-09 12:07:52 +08:00
Ten Thousand Leaves
6b2d3c3ec1
Make each dataNode only accept one bulk load task simultaneously (#16820) 
issue: #16674

/kind improvement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-05-07 14:05:52 +08:00
Enwei Jiao
67ce206cf0
make SegmentMaxLifetime configurable (#16783) 
Signed-off-by: Enwei Jiao <jiaoew2011@gmail.com>
 2022-05-06 23:41:51 +08:00
Ten Thousand Leaves
a8e1c8fa9d
Make duplicate flush recalls success and other logic update (#16755) 
Latest logic:
1) Duplicate flush calls on same segments will not result in errors (same as the original design)
2) `FlushSegments` now still flushes stale segments even if non-stale segments failed to get flushed

issue: #16749

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-05-06 17:49:51 +08:00
cai.zhang
2be46a01c0
Reduce the number of retries and add error log (#16754) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2022-05-05 09:31:51 +08:00
zhenshan.cao
97757405de
Fix bug: wrong metrics name and forget to register metrics (#16714) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-04-29 15:35:47 +08:00
zhenshan.cao
0a953948af
Simplify monitoring metrics (#16687) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-04-27 23:03:47 +08:00
Xiaofan
be5d1f1243
Fix Datacoord ut when etcd is slow (#16689) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-04-27 21:49:47 +08:00
groot
bc5e9ec929
Add list import tasks api (#16605) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-04-25 17:37:46 +08:00
Ten Thousand Leaves
e66ac6a77c
Resolve an issue with bulk load where segments in flow graph replica are not updated (#16609) 
issue: #15604

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-25 11:07:47 +08:00
Xiaofan
89b4a34892
Fix Data race in NodeID (#16603) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-04-24 22:03:44 +08:00
congqixia
69252f812d
Implement memory replica in Proxy, QueryNode and QueryCoord (#16470) 
Related to #16298 #16291 #16154
Co-authored-by: sunby <bingyi.sun@zilliz.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: yah01 <yang.cen@zilliz.com>
Co-authored-by: Letian Jiang <letian.jiang@zilliz.com>

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-04-20 16:15:41 +08:00
Ten Thousand Leaves
289e468a7a
Make bulk load fully work (#16512) 
issue: #15604

/kind enhancement

Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-20 14:03:40 +08:00
godchen
4781db8a2a
Add datanode import (#16414) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-04-12 22:19:34 +08:00
jaime
ed3c29eae7
Support Kafka (#16149) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-04-12 19:47:33 +08:00
Ten Thousand Leaves
442c73a544
Support passing channel names to DataNode (#16424) 
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-12 13:25:34 +08:00
codeman
4e12271d79
Support login with username and password (#15656) (#16341) 
Signed-off-by: kejiang <ke.jiang@zilliz.com>

Co-authored-by: kejiang <ke.jiang@zilliz.com>
 2022-04-11 19:49:34 +08:00
godchen
bb7a0766fe
Add dependency factory (#16204) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2022-04-07 22:05:32 +08:00
Ten Thousand Leaves
1e6f6bd072
Implement set segment state call from RootCoord to DataCoord. (#16374) 
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-06 15:33:32 +08:00
XuanYang-cn
5494f3c318
Remove offLineNodes from watch states check (#16347) 
This PR also
- adds more information in log
- makes Session able to logged by zap.Any/zap.String

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-04-01 22:33:29 +08:00
Ten Thousand Leaves
45be3deb3f
Implement RC, DC, DN calling path for import. (#16321) 
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-04-01 11:33:28 +08:00
congqixia
376a7f6567
Fix Datacoord unsubAttempt using pchannel (#16337) 
Fix the channel name used when Datacoord unsub for DataNode
This channel shall always be pChannel instead of vChannel

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-04-01 11:25:28 +08:00
Ten Thousand Leaves
0d54697d42
Implement ReportImport related logic. (#16312) 
/kind feature

issue: #15604
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-03-31 13:51:28 +08:00
Jiquan Long
ba37531456
Add support for loading multiple indexes (#16138) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2022-03-30 21:11:28 +08:00
XuanYang-cn
a77dd1034c
Adopt the extended state in DataCoord (#16200) 
ChannelWatchInfo with ToWatch, ToRelease will trigger a timer.
ChannelManager now reacts to different ChannelWatch states.

- WatchSuccess > log this info

- WatchFailure/WatchTimeout > ToRelease

- ReleaseSuccess > Delete, reassign if not from DropCollection

- ReleaseFailure/ReleaseTimeout > Cleanup subscription and Delete,
  reassgin if not from DropCollection.

Some Notes:
1. Reassignment will add this channel to buffer if there's only one node.

See also: #15846

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-03-28 22:33:27 +08:00
XuanYang-cn
a00968fdfe
Fix unregister in datacoord cluster (#16182) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2022-03-25 12:09:25 +08:00
xiyichan
ca129d4308
Support configurable msgstream (#16131) 
Signed-off-by: xiyichan <2863768433@qq.com>
 2022-03-24 10:15:25 +08:00
groot
1c4b949a1d
modify import rpc interface (#16129) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-03-22 15:11:24 +08:00
groot
2078b24dca
Prepare for import (#16083) 
Signed-off-by: groot <yihua.mo@zilliz.com>
 2022-03-21 15:47:23 +08:00
Ji Bin
3cd28420f1
Support compile under windows (#15786) 
This patch makes compile milvus under windows(MSYS), including:
- some cpp adaptation for compile under msys/gcc-10.3
- install toolchain scripts for setup from MinGW/MSYS `scripts/install_deps_msys.sh`
- adaptation for POSIX API use in golang
  * using gofrs/flock instead of syscall.Flock
  * using x/exp/mmap instead of syscall.Mmap
- introducing github actions for build milvus.exe under windows/MSYS
- rocksdb's patch for MSYS
- adaptation for compile knowhere under windows
- a windows package script for pack zip file, `scripts/package_windows.sh`

issue #7706

Signed-off-by: Ji Bin <matrixji@live.com>
 2022-03-17 17:17:22 +08:00
zhenshan.cao
f3eeecf146
Reduce the number of metrics (#16030) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2022-03-15 21:51:21 +08:00
xige-16
09b2c46b61
Support create collection with VarChar field (#15904) 
Signed-off-by: xige-16 <xi.ge@zilliz.com>
 2022-03-14 23:20:02 +08:00
jaime
29975a7a26
Remove Consume() API from mqstream interface (#15886) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-03-11 20:09:59 +08:00
groot
89356ebcb9
Modify import rpc interfaces (#16006) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-03-11 17:13:59 +08:00
groot
5fdef607d5
Add rpc interfaces for import (#15930) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2022-03-09 18:33:59 +08:00
congqixia
6513be5c44
Extend watch/release channel logic in DataNode (#15925) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-03-09 15:39:58 +08:00
Cai Yudong
3e862ecdfb
Merge msgChannelConfig and knowhereConfig into commonConfig (#15843) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2022-03-04 11:17:56 +08:00
jaime
7a3a721380
Reconstruct mqstream module (#15784) 
Signed-off-by: yun.zhang <yun.zhang@zilliz.com>
 2022-03-03 21:57:56 +08:00
Xiaofan
bd7d294c6d
Refine log level in data (#15821) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2022-03-02 15:35:55 +08:00
yah01
0cfb6a85be
Make compaction not generate empty segment (#15707) 
Signed-off-by: yah01 <yah2er0ne@outlook.com>
 2022-02-28 10:17:54 +08:00
congqixia
b0923f1299
Fix DataCoord panics if message queue service quits before it (#15702) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2022-02-23 11:09:52 +08:00
Nemo
3eb074a9b0
Update some comments for data coord with some minor code refactoring. (#15686) 
Signed-off-by: Yuchen Gao <yuchen.gao@zilliz.com>
 2022-02-22 13:15:51 +08:00
Jenny Li
83f1358408
Update codecov.yml (#15639) 
Signed-off-by: Jenny Li <jing.li@zilliz.com>
 2022-02-18 18:29:50 +08:00