test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-11 09:46:26 +08:00
Code Issues Actions 16 Packages Projects Releases Wiki Activity
11,750 Commits 97 Branches 105 Tags 189 MiB
4056b414f5
Commit Graph

245 Commits

Author SHA1 Message Date
zhenshan.cao
d1d5f9e409
Replace proto.MarshalTextString with proto.Marshal (#8542) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-09-29 20:26:00 +08:00
sunby
2ea7c0af25
Check error in datacoord server test (#8802) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-28 22:04:06 +08:00
sunby
eee075b10e
Check error in server test (#8801) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-28 21:54:14 +08:00
congqixia
d374ea71ba
Unify liveness check in datacoord and datanode (#8480) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-28 16:10:04 +08:00
sunby
745221e87d
Check error in segment_manager.go (#8705) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-27 22:46:12 +08:00
sunby
e5bc8e7050
Check error in MoveSegmentBinlogs (#8704) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-27 22:44:18 +08:00
sunby
c152cf2b6b
Check error in segment_manager_test.go (#8706) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-27 22:42:02 +08:00
congqixia
ffbfaf7f76
Apply collection start position when open segment (#8697) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-27 20:34:03 +08:00
sunby
d1dead500c
Check error in allocator test (#8621) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-27 10:46:08 +08:00
sunby
75767d58b3
Remove useless codes in datacoord (#8619) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-26 21:54:07 +08:00
Xiaofan
d38adda8c2
Rename log file based on node id (#8517) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-09-26 21:18:11 +08:00
dragondriver
019f18ddf9
Expose created time and updated time to datacoord metrics (#8172) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-26 17:50:07 +08:00
zhenshan.cao
5390136cd6
[skip ci] Fix golint error of services.go in datacoord (#8524) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-09-24 21:33:55 +08:00
congqixia
ca62b1e149
Use clone instead of ShadowClone when update pb data (#8461) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-24 12:53:54 +08:00
Xiaofan
f5173b595f
Add a channel prefix for all channels (#8166) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-09-23 10:53:53 +08:00
sunby
3fc1b9cf9f
Change copyright (#8318) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-22 20:12:08 +08:00
sunby
82dd762ef0
Change copyright (#8317) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-22 19:59:59 +08:00
cai.zhang
79b7dcc75a
Add trace log for insert (#8301) 
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
 2021-09-22 19:33:54 +08:00
Xiaofan
0c35cd57b0
Unify param table to the same naming format (#8254) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-09-20 23:27:52 +08:00
congqixia
21f15d66bb
Add option for DataNode creation (#8240) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-18 19:10:07 +08:00
Xiangyu Wang
c314b2b4a8
Remove unused label rules (#7929) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-09-18 18:19:51 +08:00
Yusup
ccb57ba4b2
add collection alias (#7255) 
Signed-off-by: Yusup <yusup@lsgrep.com>
 2021-09-18 11:13:51 +08:00
congqixia
266db4c883
Add readme for datacoord (#8141) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-17 23:57:55 +08:00
congqixia
01c8341a2e
[skip ci]Add comments for SegmentManager (#8087) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-16 19:55:49 +08:00
congqixia
2bca69596e
Add some comment for SegmentsInfo (#7995) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-15 21:07:53 +08:00
congqixia
c76f91233f
Put errors together (#7958) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-15 21:05:52 +08:00
sunby
eed856d626
Change error message to lower case (#7890) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-14 17:33:48 +08:00
sunby
9c2fb0bccb
Recorrect comment for Allocation (#7887) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-14 17:27:55 +08:00
congqixia
85d73358cc
Remove ctx from LongTermChecker (#7792) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-13 16:50:57 +08:00
congqixia
e2c15d5b01
[skip ci]Add comment for ClusterStore (#7785) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-13 15:59:55 +08:00
congqixia
d3be316bc2
Add Comment for datacoord cluster (#7676) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-10 11:26:00 +08:00
Xiangyu Wang
42aafdc407
Remove unused msg from msg.go (#7612) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-09-09 19:56:00 +08:00
sunby
49b6c3101d
Remove datacoord useless code (#7673) 
Issue: #7624
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-09 19:08:03 +08:00
congqixia
c96b19a640
Add datacoord metricsinfo unit test (#7595) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-09 10:16:00 +08:00
dragondriver
9b21b3b5fd
Initialize test framework of proxy (#7563) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-09 10:06:29 +08:00
congqixia
664aeb09c6
Add ut and comments for datacoord meta (#7560) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-08 11:35:59 +08:00
congqixia
5b22697ec4
Add segment manager unit tests (#7525) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-07 13:59:58 +08:00
bigsheeper
b7103357fd
Load failed if collection beyonds the memory limit (#7434) 
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-09-07 11:35:18 +08:00
congqixia
fb4e23bc79
Add datacoord server unit tests (#7499) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-06 17:02:41 +08:00
congqixia
1f590d6c26
Add datacoord services related unit tests (#7467) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-06 11:12:42 +08:00
dragondriver
960741ee16
Support more detailed metrics information (#7472) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-03 19:35:47 +08:00
dragondriver
1db573731b
Cache the system information metrics (#7436) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-03 17:15:26 +08:00
dragondriver
3721cbfea1
Proxy metrics integrates with other coordinators (#7400) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-01 17:35:00 +08:00
dragondriver
89748ac639
Expose metrics of data cluster (#7177) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-01 10:13:15 +08:00
congqixia
0281bd8d22
Change context to correct package (#7331) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-08-27 15:37:56 +08:00
Xiangyu Wang
c62f23e201
[skip ci]Remove neza2017 from reviewers (#7324) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-08-27 00:27:36 +08:00
sunby
4d56d53c0b
Add a tool to move binlog path to SegmentInfo (#7243) 
issue: #7214
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-08-24 15:51:51 +08:00
Xiaofan
58b2089692
Import Embedded Etcd Kv Part 1(#7112) (#7174) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-08-24 09:45:51 +08:00
congqixia
f5451d12f3
Refactor datacoord allocator (#7226) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-08-23 17:59:51 +08:00
congqixia
be5c492e5a
Add datacoord unit tests (#7197) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-08-20 17:50:12 +08:00
sunby
6e34f4c7f1
Add a seal policy which restrict the lifetime of a segment (#7172) 
issue: #7164
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-08-20 15:42:12 +08:00
sunby
3cc210da26
Add flush interval (#6996) 
Time tick message is sent about per 0.2 seconds. DataCoord will call
`FlushSegment` too frequently. We add a min interval for segment flush. A
segment will called `FlushSegment` only once between the interval.

issue: #6995
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-08-19 14:08:10 +08:00
sunby
4f7c3e7d3a
Save binlog path in SegmentInfo (#6590) 
issue: #6589
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-08-19 13:00:12 +08:00
sunby
9551dc3b7e
Fix segment auto flush bug (#7085) 
issue: #7084
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-08-16 11:00:09 +08:00
XuanYang-cn
6c5be83ea1
Refactor NewEtcdKV API (#7075) 
This PR mainly did the following 3 things:

- Refactor NewEtcdKV API
- Add unittests for `kv/etcd/`, raise the coverage
up to 94%
- Correct some APIs from version to revision

The old NewEtcdKV() API took `clientv3.Client` as
one of the input, making everyone who using this API
had to create a new `clientv3.Client`. So there're a lot
of repeated codes everywhere.

Meanwhile, someplace didn't even check whether the
`clientv3.New()` returned error or not, and the old API
also didn't check whether the clientv3.Client was Nil.
This caused issue#6955.

Ectd's version and revision have nothing alike, they
represent different concepts, but our APIs kind of
mixed them all up. So I correct those APIs' names.

Resolves: #6955

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-08-13 11:04:09 +08:00
sunby
5c383d1d02
Add a checker to listen for the timetick. (#7061) 
issue: #7060
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-08-12 19:58:08 +08:00
Xiangyu Wang
5e78fd6a97
[skip ci]Update approvers (#7063) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-08-12 19:36:09 +08:00
sunby
31182e52a9
Fix expire allocation bug (#7053) 
issue: #7052
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-08-12 10:48:08 +08:00
sunby
454620dfaa
Add unit test in datacoord (#6985) 
issue: #6984
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-08-04 16:59:25 +08:00
sunby
23bffe399d
Enhance log print in data coordinator (#6681) 
issue: #6680
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-07-28 11:43:22 +08:00
zhenshan.cao
8a1a841011
FixBug: Incorrect handling of time synchronization (#6814) 
* FixBug: proxy send wrong sync timetick and DataCoord did not filter channel

Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-07-27 14:34:48 +08:00
sunby
aa20d88f6f
Fix datacoord stop bug (#6642) 
issue: #6641
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-07-24 09:25:14 +08:00
sunby
fc74a0f578
Auto create new segments when allocating rows more than (#6665) 
max number of rows per segment

If user insert too much rows in a request. Now we will return a failed
response. Maybe auto creating new segments to hold that much rows is a
better way.

issue: #6664
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-07-23 21:58:33 +08:00
sunby
63387c83ef
Remove collection check when saving binlog (#6658) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-07-23 21:23:22 +08:00
sunby
b98b30b96f
Fix datacoord_test.go failed (#6558) 
Create a mock rootcoord when testing datacoord to hide the network layer
issue: #6557
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-07-19 10:41:15 +08:00
sunby
ee38d14f5d
Fix channel lost after data node crash (#6545) 
If we start up 2 data nodes and one of them crashes. We expect that all
channels of crashed node will be allcoated to the alive node. But now we
discover that these channels are lost after data node crash. The reason
is we pass a NodeInfo with empty channel info. We fix it and improve log
print.

issue: #6501
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-07-15 16:38:31 +08:00
groot
6b001fd76a
fix debug error in IDE (#6539) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-07-15 14:27:54 +08:00
sunby
83be910baa
Refactor segment manager (#6441) 
issue: #6440
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-07-12 17:24:25 +08:00
sunby
a8e5fd2024
Refactor cluster in dataservice (#6356) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-07-12 11:03:52 +08:00
sunby
017484e093
Refactor meta in data coord (#6300) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-07-07 14:02:01 +08:00
sunby
836a45ec26
Remove sending newSegmentMsg to rootcoord (#6301) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-07-06 09:24:05 +08:00
congqixia
76fd9ad339
Fix restart node logic (#6265) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-07-02 16:48:30 +08:00
sunby
07b0989628
Add GetFlushedSegments in data coordinator (#6253) 
Send segment flush message by grpc and add a new interface
`GetFlushedSegments`

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-07-02 11:16:20 +08:00
neza2017
6019c193fc
add grpc interface to process NewSegment And FlushSegmentCompleted (#6247) 
Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
 2021-07-01 14:58:17 +08:00
congqixia
580e3a57cf
stop add new node into cluster before validated (#6241) 
* stop add new new into cluster

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix startup & test cases

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-07-01 14:32:16 +08:00
congqixia
3119ce2a6c
Stop create session in watch op (#6222) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-30 16:50:20 +08:00
congqixia
ae072b4f1d
Add candidate management for datacoord dn cluster (#6196) 
* Add candidate management for datacoord dn cluster

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Fix test cases

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-30 10:20:15 +08:00
sunby
dd3ba9b2b3
Add rev by one when start watch event (#6152) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-29 14:52:14 +08:00
congqixia
c49ce4ddc2
Add datacoord Server comment and improve para (#6182) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-29 10:46:13 +08:00
congqixia
b27e6b52bf
Force cluster refresh for each dn change event (#6161) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-28 13:28:14 +08:00
yukun
39614aa8eb
Add rocksmq_path in config and paramtable (#6099) 
* Add rocksmq_path in config and paramtable

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add rocksdbPath in NewRmsFactory

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Change rdb default path to /vat/lib/milvus/rdb_data

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-25 19:44:11 +08:00
sunby
25d3fead9f
Change error type to error functions in data coord (#6076) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-24 19:05:42 +08:00
godchen
3785bb6330
Change retry times (#6083) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-06-24 19:05:06 +08:00
congqixia
f6761bbc33
Refactor datacoord segment status (#6039) 
* Refactor datacoord segment status

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>

* Remove lastExpireTime

Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-24 14:20:10 +08:00
yukun
c8bf9078ae
Change pchannels to vchannel in retrieveResults (#6054) 
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-24 13:40:03 +08:00
sunby
46f86a0354
Return sealed segments when calling flush (#6011) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-23 16:56:11 +08:00
congqixia
d6259d2dd6
Refactor policy (#6003) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-06-23 14:28:08 +08:00
sunby
a529410b38
Fix data race in internal/datacoord/meta.go (#6005) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-23 12:20:06 +08:00
sunby
fe8432016d
Refactor data coordinator (#6008) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-23 12:10:12 +08:00
godchen
fd07a44dba
Fix error in retry refactor (#6000) 
* Fix error in retry refactor

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-06-23 11:48:06 +08:00
godchen
99be4c09a5
change retry (#5996) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-06-23 09:24:10 +08:00
sunby
0978b93a21
Refactor data coordinator (#5982) 
Rename variable name and make error msg more clear

Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-22 18:24:08 +08:00
sunby
d2c8462dac
Rename params in data_coord.yaml (#5968) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-22 16:18:21 +08:00
Cai Yudong
d5f4ee6f44
Rename proto master/index_service/proxy_service (#5973) 
* rename proto proxy_service to proxy

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename proto index_service to index_coord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>

* rename proto master to root_coord

Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-22 16:14:09 +08:00
sunby
c863a193de
Rename filename from dataservice to datacoordinator (#5946) 
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-06-22 10:42:07 +08:00