Commit Graph

136 Commits

Author SHA1 Message Date
Xiaofan
2f92f78e7b
Remove meaningless Etcd kv logs (#8019)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-09-22 20:27:54 +08:00
yukun
06c9f03c59
Refactor rocksmq close logic and improve codecov (#8278)
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-09-22 17:21:00 +08:00
godchen
c8c10bb3d6
Remove performanceTest (#8164)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-09-18 10:13:52 +08:00
godchen
74bb712527
[skip ci]Add minio comment (#8165)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-09-17 23:59:47 +08:00
yukun
5a303e7672
Fix rocksmq load with prefix (#7678)
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-09-10 10:22:01 +08:00
sunby
62f4cb5e1b
Remove stats watcher (#7501)
issue: #7500
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-09-07 16:11:07 +08:00
dragondriver
960741ee16
Support more detailed metrics information (#7472)
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2021-09-03 19:35:47 +08:00
zhenshan.cao
7c60e6aa00
Avoid port conflicts between embedded etcd and external etcd (#7320)
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-08-27 10:13:57 +08:00
Xiangyu Wang
c62f23e201
[skip ci]Remove neza2017 from reviewers (#7324)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-08-27 00:27:36 +08:00
sunby
4d56d53c0b
Add a tool to move binlog path to SegmentInfo (#7243)
issue: #7214
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-08-24 15:51:51 +08:00
Xiaofan
58b2089692
Import Embedded Etcd Kv Part 1(#7112) (#7174)
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
2021-08-24 09:45:51 +08:00
XuanYang-cn
9e71daa759
Fix rocksdb not installed problem (#7200) 2021-08-20 17:46:48 +08:00
Cai Yudong
305fa9c17e
Fix timestamp go back issue for timeticksync (#7123)
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
2021-08-18 14:36:10 +08:00
yukun
847586eb95
Add rocksmq retention (#6617)
Signed-off-by: fishpenguin <kun.yu@zilliz.com>
2021-08-16 18:46:10 +08:00
XuanYang-cn
6c5be83ea1
Refactor NewEtcdKV API (#7075)
This PR mainly did the following 3 things:

- Refactor NewEtcdKV API
- Add unittests for `kv/etcd/`, raise the coverage
up to 94%
- Correct some APIs from version to revision

The old NewEtcdKV() API took `clientv3.Client` as
one of the input, making everyone who using this API
had to create a new `clientv3.Client`. So there're a lot
of repeated codes everywhere.

Meanwhile, someplace didn't even check whether the
`clientv3.New()` returned error or not, and the old API
also didn't check whether the clientv3.Client was Nil.
This caused issue#6955.

Ectd's version and revision have nothing alike, they
represent different concepts, but our APIs kind of
mixed them all up. So I correct those APIs' names.

Resolves: #6955

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
2021-08-13 11:04:09 +08:00
Xiangyu Wang
5e78fd6a97
[skip ci]Update approvers (#7063)
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-08-12 19:36:09 +08:00
congqixia
4d5eeb2890
Fix minio object leaking goroutine (#6944)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-08-02 15:25:24 +08:00
neza2017
5adb69161f
Set CollectionInfo.CreateTime to the timestamp when save the metadata into etcd, (#6763)
not the timestamp when RootCoord receive the CreateCollection Request

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-07-23 14:36:12 +08:00
cai.zhang
a3c662c7bf
Add watchWithVersion interface for etcdKV (#6663)
* Add watchWithVersion interface for etcdKV

Signed-off-by: xiaocai2333 <cai.zhang@zilliz.com>

* Add unittest

Signed-off-by: xiaocai2333 <cai.zhang@zilliz.com>

* Add LoadWithVersion interface

Signed-off-by: xiaocai2333 <cai.zhang@zilliz.com>
2021-07-21 18:20:11 +08:00
godchen
b9f5949680
Add vector chunk manager (#6613)
* add file manager

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* add file manager

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* add vector file manager

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* rename

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* go fmt

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix ut error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix action error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix memory error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-07-20 15:06:09 +08:00
cai.zhang
f3a5b46288
Fix bug for repeatedly create bucket in minio (#6477)
* Fix bug for repeatedly create bucket in minio

Signed-off-by: xiaocai2333 <cai.zhang@zilliz.com>

* Simplify code

Signed-off-by: xiaocai2333 <cai.zhang@zilliz.com>

* Add log

Signed-off-by: xiaocai2333 <cai.zhang@zilliz.com>
2021-07-14 09:55:46 +08:00
godchen
346e9cba4a
Add minio FGet method (#6386)
* Add minio FGet method

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* go fmt

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* change el append

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-07-09 11:44:40 +08:00
quicksilver
7d920e9c9c
Fix compile error on CentOS (#6359)
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
2021-07-08 15:43:45 +08:00
godchen
99be4c09a5
change retry (#5996)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-06-23 09:24:10 +08:00
Cai Yudong
0f4bd50ce3 Support ETCD endpoints array (#5755) 2021-06-15 16:36:36 +08:00
congqixia
7beab6eddc Fix new miniokv retry logic (#5749)
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
2021-06-15 16:36:36 +08:00
godchen
0610ceb3dc
Fix ut error (#5573)
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-06-03 18:44:33 +08:00
Xiangyu Wang
ca0993c274 Update OWNERS
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-05-30 18:58:47 +08:00
cai.zhang
a606ab9c67
Add power-off restart logic for IndexService and IndexNode (#5395)
Add power-off restart logic for IndexService and IndexNode


Signed-off-by: <xiaocai2333> <cai.zhang@zilliz.com>
2021-05-27 14:24:29 +00:00
neza2017
c0daf8e41d
set timestamp on dd requests (#5298)
- set dd request's time tick on master
- send time tick into dd channel

See also: #5291

Signed-off-by: yefu.chen yefu.chen@zilliz.com
2021-05-20 06:14:14 +00:00
godchen
ccc0130bb4
Add exclusive parameter (#5278)
Add exclusive parameter.
Issue #5174 

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-05-18 11:07:27 +00:00
neza2017
49f6542b1d
query master service's with timestamp (#5275)
let meta support snapshot
so collection could query meta with timestamp

Resolves: #5219

Signed-off-by: yefu.chen <yefu.chen@zilliz.com>
2021-05-18 06:18:02 +00:00
godchen
cca75583a6
Add session util (#5269)
Add session util.
Issue #5174 

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-05-18 03:39:21 +00:00
godchen
b74afd7a0e
Add service registration (#5189)
Add service registration.
Part of Issue #5174.

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-05-14 02:05:18 +00:00
Xiangyu Wang
82ccd4cec0
Rename module (#4988)
* Rename module

Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
2021-04-22 14:45:57 +08:00
godchen
139b334732
Add kv etcd unittest and remove useless code (#4974)
* Add kv etcd unittest and remove useless code

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-04-21 18:45:35 +08:00
yukun
89c4bf82a0 Add copyright in kv module
Signed-off-by: kun yu <kun.yu@zilliz.com>
2021-04-19 11:13:52 +08:00
godchen
deba964590 Change name of kvBase
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-04-12 18:09:28 +08:00
sunby
751c9d4e81 Refactor meta.go
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-04-09 09:55:04 +08:00
sunby
1d7195e036 Add minio and etcd stats watcher
Our system needs to watch the performance of etcd and minio. For now I just add the creation stats.

Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-04-08 14:14:43 +08:00
zhenshan.cao
9c574b7c7b Update sdk version to 0.0.55
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-03-23 18:40:08 +08:00
紫晴
3be6dde459 Add smoke tag
Signed-off-by: 紫晴 <ting.wang@zilliz.com>
2021-03-23 14:35:30 +08:00
zhenshan.cao
c2734fa55f Fix bug and enchance system
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-03-22 16:36:10 +08:00
sunby
6287ffa1b4 Replace log with zap
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-03-11 14:15:18 +08:00
neza2017
efa7306ef6 Remove with prefix
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-03-06 16:00:41 +08:00
sunby
ddddd65d10 Delete internal/errors package
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-03-05 10:15:27 +08:00
cai.zhang
f5977a1302 Change retry times and add retry logic
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2021-02-26 15:17:47 +08:00
groot
833f1d49ab Let singlenode use rocksmq
Signed-off-by: groot <yihua.mo@zilliz.com>
2021-02-09 15:57:10 +08:00
sunby
2f7319cdbb Add interfaces with other modules
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-01-22 19:43:27 +08:00
XuanYang-cn
fb63983eb2 Modify rocksdb compilation
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2021-01-14 15:12:08 +08:00
yukun
6e36f10a26 Fix rocksdb illegal instruction in some machine
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-01-14 14:24:14 +08:00
bigsheeper
2d2af1505a Improve mutex usage in query node
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-01-12 18:03:24 +08:00
GuoRentong
f3d2afa4f4 Update doc:detailed interface definition
Signed-off-by: GuoRentong <rentong.guo@zilliz.com>
2021-01-11 18:35:54 +08:00
quicksilver
a6690dbc99 Fix syntax error: Bad fd number
Signed-off-by: quicksilver <zhifeng.zhang@zilliz.com>
2021-01-11 15:25:19 +08:00
yukun
18cfd31d40 Fix Load and Remove with prefix
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-01-11 15:17:06 +08:00
sunby
5dfe9448ae Modify test_index
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-01-08 16:00:16 +08:00
yukun
d11f00cd9f Add rocksdb in kv
Signed-off-by: yukun <kun.yu@zilliz.com>
2021-01-08 19:07:57 +08:00
neza2017
4015d7245d Merge operation
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2021-01-06 14:45:50 +08:00
cai.zhang
1cfc2ff0a5 Add make clean in Makefile
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2021-01-06 13:50:48 +08:00
zhenshan.cao
b06e01f523 Initmate querynode collection manager
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2021-01-06 11:31:36 +08:00
godchen
1c87faf8b3 Add insert request opentracing
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2021-01-06 11:17:35 +08:00
FluorineDog
e16a7478da Add verifier
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
2021-01-05 20:13:54 +08:00
sunby
95b162ccfd Refactor flush scheduler
Signed-off-by: sunby <bingyi.sun@zilliz.com>
2021-01-05 10:57:59 +08:00
bigsheeper
d5d9fa03ea Disable the dropCollection filtering
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-01-04 20:46:00 +08:00
bigsheeper
33e1e17045 Fix writer client DescribeSegment api
Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
2021-01-04 20:16:15 +08:00
XuanYang-cn
32977e270c Add impl cgo of parquet
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2020-12-08 14:41:04 +08:00
zhenshan.cao
7bbbc14637 Fix bug: address already used
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2020-12-07 15:22:20 +08:00
xige-16
37d7526d31 Fix search test failure
Signed-off-by: xige-16 <xi.ge@zilliz.com>
2020-12-07 14:37:42 +08:00
neza2017
aeca8f85a5 Run cwrapper unittest in ci
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2020-12-05 17:39:58 +08:00
cai.zhang
806a97505c Add query node for CI progress
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2020-12-05 16:12:45 +08:00
neza2017
70710dee47 Add parquet payload
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2020-12-05 16:11:03 +08:00
FluorineDog
6412ebc0d4 Add support of metric type in schema, enable binary vector, fix segfault
Signed-off-by: FluorineDog <guilin.gou@zilliz.com>
2020-12-05 06:46:01 +08:00
zhenshan.cao
0b6fe0a8cf Refactor paramtable
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2020-12-04 14:18:17 +08:00
godchen
7ab5b5d80d Add MinIO kv implements
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
2020-12-03 19:00:11 +08:00
cai.zhang
17c3a70d64 update ParameterTable
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2020-11-21 15:06:46 +08:00
cai.zhang
5bcabffdaf Replace conf struct with ParamsTable
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2020-11-16 21:10:43 +08:00
dragondriver
c2382adf04 Initialize the task scheduler of Proxy
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2020-11-14 11:24:49 +08:00
cai.zhang
b68322d469 Open golangci-lint check
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2020-11-12 17:58:05 +08:00
neza2017
14d4a400dd Load etcd port from etcd
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2020-11-12 17:15:01 +08:00
cai.zhang
1cd2fd9954 Add the standard of golangci-lint
Signed-off-by: cai.zhang <cai.zhang@zilliz.com>
2020-11-12 12:04:12 +08:00
dragondriver
b8ccbc1c97 Add InsertTask and SearchTask to Proxy
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
2020-11-07 16:18:23 +08:00
zhenshan.cao
86d2d36053 Add MemoryKV, mockkv and tso test
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2020-11-06 16:47:18 +08:00
XuanYang-cn
3de52a51c2 Add interface for EtcdKV
Signed-off-by: XuanYang-cn <xuan.yang@zilliz.com>
2020-11-03 15:08:50 +08:00
zhenshan.cao
e962b57fbb Add timestamp allocator
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2020-11-03 14:53:36 +08:00
neza2017
10217f9485 Add test for etcd kv
Signed-off-by: neza2017 <yefu.chen@zilliz.com>
2020-11-02 10:26:33 +08:00
zhenshan.cao
bbecf51f8c Move kv directory
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
2020-10-29 11:46:34 +08:00