test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-04 21:09:06 +08:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity
7,120 Commits 96 Branches 105 Tags 186 MiB
5c69c6902d
Commit Graph

171 Commits

Author SHA1 Message Date
zhenshan.cao
5c69c6902d Refactor proxy channelTimeTick and fix search timeout (#5760) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-15 16:36:36 +08:00
Cai Yudong
0f4bd50ce3 Support ETCD endpoints array (#5755) 2021-06-15 16:36:36 +08:00
Cai Yudong
17b19233dc add BroadcastMany, send DdMsg to collection channels only (#5734) 
* add BroadcastMany, send DdMsg to collection channels only

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix master unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix time tick of masterservice

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix time tick unittest of masterservice

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* skip dataservice ut

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix format issue

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update timeticksync

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update timeticksync

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update SendDdXXXReq interface

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update master unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* roll back unecessary change

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* enable ut

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 16:36:36 +08:00
zhenshan.cao
f74ea3beee [skip ci] Remove ProxyService (#5738) 
* [skip ci] Remove ProxyService

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Change helm branch

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fix bug: paramstable crashed

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-15 16:36:35 +08:00
sunby
189ac881f3 Fix bugs (#5676) 
* Remove redundant session startup

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Register datanode after start success

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix meta snap shot

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix datanode message stream channel

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix bugs when drop empty collection

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Fix bug of getting pchan statistics from task scheduler

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix i/dist/dataservice test code

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix epoch lifetime not applied

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* fix datanode flowgraph dd node

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix handle datanode timetick bug

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Remove repack function of dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix proxynode

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Apply extended seal policy

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add check for time tick

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix check

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix the repack function of dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix the bug when send statistics of pchan

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Fix the repack function when craete dml stream

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix bugs

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix describe collection

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix bug when send timestamp statistics

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* fix data node

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Add length check before flush request

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add log for data node

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix SaveBinlog bugs

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Add more log in datanode

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Put SegmentState.Flushing as the last one in enum to fit the client

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Fix params in GetInsertBinlogPaths

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Rename policy

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Remove unused ddl functions and fields

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Remove pchan when drop collection

Signed-off-by: dragondriver <jiquan.long@zilliz.com>

* Add balanced assignment policy

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* fix master ut

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Add lock in session manager

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* add log for debug

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* Fix some logic bug and typo

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix recover bugs

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Get collection scheme of a specific timestamp

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Change CheckPoint to SegmentInfo in VchannelInfo

Signed-off-by: sunby <bingyi.sun@zilliz.com>

* Recover Unflushed segment numOfRows

Signed-off-by: yangxuan <xuan.yang@zilliz.com>

* Fix dataservice unit tests

Signed-off-by: sunby <bingyi.sun@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
Co-authored-by: yangxuan <xuan.yang@zilliz.com>
Co-authored-by: dragondriver <jiquan.long@zilliz.com>
Co-authored-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:06:11 +08:00
congqixia
f794ceb0cc Add segment info rand logic in test case (#5639) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-15 16:04:48 +08:00
Cai Yudong
a948b71e06 Support to send dd msg to all dml channels (#5597) 
* send dd msg to dml channel

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add dml_channels.go

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix race

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-15 16:04:48 +08:00
sunby
e7521afed5 Change SaveBinlogPath (#5576) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-15 16:04:48 +08:00
godchen
275881dbdb Remove address param (#5556) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-06-15 16:04:46 +08:00
neza2017
cb6ae3678a fix master service ut (#5555) 
* fix master unitttest

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* tt

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* fix master ut

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 15:59:04 +08:00
Cai Yudong
2cc330b117 Remove proxy service from master (#5510) 
* remove proxy service

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* remove time sync

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* remove proxy service

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* remove proxy service

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* remove proxy service

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* remove proxy service from distributed/masterservice

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix static-check

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-15 15:59:04 +08:00
zhenshan.cao
bc46e4780d
Fixbug:index is not loaded in some cases (#5633) 
* Fixbug:index is not created in some cases

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Delete unecessary print

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* fix create index

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-06 09:41:35 +08:00
zhenshan.cao
03e3b4beb4
Add log to startup process (#5594) 
* Add log to startup process

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fixbug: avoid ctx exceed deadline

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fmt code

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fix bug: wrong time unit

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>

* Fixbug: datanservice client reconnect use time ctx

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-06-04 16:29:35 +08:00
Cai Yudong
2ba93deb5e
Support master get segment info channel name from config file (#5606) 
* optimize msgstream unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* get data service segment info channel name from config file directly

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-04 16:28:34 +08:00
Xiangyu Wang
f791ccb246 Update OWNERS file 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-06-03 20:01:24 +08:00
dragondriver
99699b2021
Support to show collections which are loaded into memory (#5567) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-06-03 19:09:33 +08:00
godchen
0610ceb3dc
Fix ut error (#5573) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-06-03 18:44:33 +08:00
Cai Yudong
4018dfc5b7
Support data segment channel seek to the break point when system start (#5540) 
* add DecodeDdOperation

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* reload MsgPosition from etcd and seek when system start

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* update testcase

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename const string

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* optimize master unittest

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add msg into dataSegmentChannel to test seek

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix seek test

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* roll back distributed/masterservice/masterservice_test.go

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-02 22:36:41 +08:00
Cai Yudong
d5cd561449
Add metrics for master (#5515) 
* metric

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* metric

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* add MasterDDChannelTimeTick for metrics

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add metrics for all master grpc

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* register metrics

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

* add metrics register

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* add comments for codacy check

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* fix format issue

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

Co-authored-by: yefu.chen <yefu.chen@zilliz.com>
 2021-06-01 11:04:31 +08:00
Cai Yudong
5ac726b7b9
use NewTicker instead of NewTimer (#5503) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-31 14:44:30 +08:00
Xiangyu Wang
ca0993c274 Update OWNERS 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-05-30 18:58:47 +08:00
neza2017
f80dbab6ec
watch proxynode (#5424) 
watch proxy node and,
send InvalidataCollectionMetaCache to each proxynode

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>

Resolves: #5423
 2021-05-26 12:14:30 +00:00
neza2017
c28c34e852
let master use session.ServerID as nodeID (#5385) 
let master use session.ServerID as nodeID

also see #5386 

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-05-25 07:06:05 +00:00
Cai Yudong
dab536a6cf
Rename some field names in master for better code readability (#5389) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-25 06:03:06 +00:00
Cai Yudong
4058350e30
update AddIndex not return error when fail (#5372) 
Resolves: #5332 

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-25 03:42:23 +00:00
Cai Yudong
39f8c9dd56
Save msg start position and end position when AddSegment and AddIndex (#5358) 
Resolves: #5332 

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-24 06:19:52 +00:00
neza2017
e326eaad1e
master client,reconnect if grpc failed (#5360) 
master client,reconnect if grpc failed

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-05-24 01:44:49 +00:00
godchen
0eee7e5252
Change session (#5329) 
Change session.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-21 11:28:52 +00:00
Cai Yudong
761350e44d
Refactor timetick msg handling between proxy node and master (#5321) 
Proxy node send channel timetick msg to master, master reduce timetick
msg for each channel and send the min timetick to msg stream.

Resolves: #5276 

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-21 08:08:12 +00:00
sunby
c1efa53779
Remove redundant fields in SegmentInfo and related structure (#5280) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-05-21 06:51:25 +00:00
neza2017
c0daf8e41d
set timestamp on dd requests (#5298) 
- set dd request's time tick on master
- send time tick into dd channel

See also: #5291

Signed-off-by: yefu.chen yefu.chen@zilliz.com
 2021-05-20 06:14:14 +00:00
godchen
e224d1e725
Add SessionManager (#5288) 
Add SessionManager.
Resolves: #5174 

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-19 10:36:05 +00:00
neza2017
d68ee3fbc2
query base on timeline (#5279) 
query base on timeline, add timestamp on these queries:
- ShowCollections
- HasCollection
- DescribeCollection

Resolves: #5219

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-05-18 09:12:17 +00:00
neza2017
49f6542b1d
query master service's with timestamp (#5275) 
let meta support snapshot
so collection could query meta with timestamp

Resolves: #5219

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-05-18 06:18:02 +00:00
Cai Yudong
e8ceb37f81
Add ShardsNum when create collection (#5245) 
Updata PhysicalChannelNames and VirtualChannelNames based
on ShardsNum when create collection, and update collVChan2Chan
map.

Resolves: #5240 

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-17 11:15:01 +00:00
Cai Yudong
001795ee70
Optimize CreateIndex routine in master service (#5238) 
Remove CreateIndexTask struct and indexTaskQueue

Resolves: #5228 

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-15 10:08:08 +00:00
Cai Yudong
08bb1b2ec3
Resend unsuccessful ddMsg when master start (#5214) 
Resend unsuccessful ddMsg when master start

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-14 21:26:06 +08:00
godchen
b74afd7a0e
Add service registration (#5189) 
Add service registration.
Part of Issue #5174.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-05-14 02:05:18 +00:00
Cai Yudong
7b83b11e95
Save ddmsg info and type into ETCD (#5173) 
Save ddmsg info and type into ETCD to support re-send ddmsg 
when system restart or recover from failure (#5172)

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-05-12 07:33:53 +00:00
neza2017
262c484cec
the return struct of "DescribeIndex" should contain "filed_name" (#5055) 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-04-27 02:30:55 +00:00
neza2017
d7c0a72611
index name is unique in collection scope (#5046) 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-04-26 15:53:47 +08:00
Xiangyu Wang
82ccd4cec0
Rename module (#4988) 
* Rename module

Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-04-22 14:45:57 +08:00
neza2017
83647e8503 Add copyright in master service module 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-04-19 11:12:56 +08:00
godchen
deba964590 Change name of kvBase 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-04-12 18:09:28 +08:00
neza2017
e760605187 Add unit test for master service, the coverage of meta table reachs 100% 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-04-12 15:03:23 +08:00
neza2017
15f26f653e Add unit test for master service 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-04-10 10:53:58 +08:00
neza2017
56d367b1e9 Add unit test for master service 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-04-09 16:10:12 +08:00
Cai Yudong
952f70af2a Clean makefile 
Signed-off-by: Cai Yudong <yudong.cai@zilliz.com>
 2021-04-09 14:07:03 +08:00
sunby
751c9d4e81 Refactor meta.go 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-04-09 09:55:04 +08:00
neza2017
8dd3e1e28f Add unit test for masterservice 
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
 2021-04-08 17:31:39 +08:00