test/milvus
1
0
Fork 0
mirror of https://gitee.com/milvus-io/milvus.git synced 2024-12-04 04:49:08 +08:00
Code Issues Actions 14 Packages Projects Releases Wiki Activity
9,874 Commits 96 Branches 105 Tags 185 MiB
c53f3b56fb
Commit Graph

289 Commits

Author SHA1 Message Date
XuanYang-cn
9526d22dc7
[skip ci]Add comment for normal2FlushedSegment (#9431) 
Signed-off-by: Yang Xuan <xuan.yang@zilliz.com>
 2021-10-07 23:46:37 +08:00
godchen
445f5426d1
Check data node error (#9303) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-07 22:16:56 +08:00
congqixia
908b7df41d
Add comment and implementation assertion for ddnode (#9333) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-07 19:54:56 +08:00
godchen
cd4205e447
[skip ci]Fix allocator comment typo (#9363) 
Signed-off-by: godchen0212 <qingxiang.chen@zilliz.com>
 2021-10-06 23:32:27 +08:00
XuanYang-cn
e7e37448d2
[skip ci]Add comment for flowgraph insertBufferNode (#9361) 
Signed-off-by: Yang Xuan <xuan.yang@zilliz.com>
 2021-10-06 23:30:21 +08:00
XuanYang-cn
5bb832e6cc
[skip ci]Add comment on flowgraph dmstream inputnode (#9359) 
Signed-off-by: Yang Xuan <xuan.yang@zilliz.com>
 2021-10-06 23:02:08 +08:00
XuanYang-cn
27143c6095
[skip ci]Add comment on flowgraph ddNode (#9358) 
Signed-off-by: Yang Xuan <xuan.yang@zilliz.com>
 2021-10-06 22:51:58 +08:00
XuanYang-cn
7b14b0e63d
[skip ci]Add comment on readBinary (#9355) 
Signed-off-by: Yang Xuan <xuan.yang@zilliz.com>
 2021-10-06 22:28:11 +08:00
XuanYang-cn
65a76b9bb5
[skip ci]Add comment in insertbuffernode (#9354) 
Signed-off-by: Yang Xuan <xuan.yang@zilliz.com>
 2021-10-06 22:20:05 +08:00
godchen
83228c35ad
Add flowgraph insert buffer node error check (#9310) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-06 21:06:23 +08:00
XuanYang-cn
e7aa5eebd6
[skip ci]Update comment of cache in datanode (#9313) 
Signed-off-by: Yang Xuan <xuan.yang@zilliz.com>
 2021-10-06 07:08:24 +08:00
XuanYang-cn
cb6d853b56
[skip ci]Add README.md in datanode (#9311) 
Signed-off-by: Yang Xuan <xuan.yang@zilliz.com>
 2021-10-05 23:12:03 +08:00
XuanYang-cn
f49e35a7fb
[skip ci]Update datanode param table comment (#9308) 
Signed-off-by: Yang Xuan <xuan.yang@zilliz.com>
 2021-10-05 22:54:03 +08:00
godchen
e6ee4f39bb
Add flow graph dmstream input node test error check (#9213) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-04 17:37:57 +08:00
godchen
3eee4e4259
Add segment replica test error check (#9212) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-04 17:36:06 +08:00
godchen
83f256c31b
Remove datanode mock test error (#9211) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-10-04 17:34:16 +08:00
congqixia
becf6c8f7d
Add DataNode implementation Assertion (#9144) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-03 19:46:05 +08:00
congqixia
109186db9c
[skip ci] Add comment for datanode meta_service.go (#9036) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-10-01 22:25:24 +08:00
Xiaofan
2c2b800ff4
Change Log Name with NodeId (#8721) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-10-01 08:52:50 +08:00
congqixia
6bbf0a8f8f
Read datanode insertbuf from env (#8942) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-30 17:45:07 +08:00
congqixia
10a6f7f146
Defer buf removal from insertBuf after SaveBinlog succeeds (#8930) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-30 17:42:11 +08:00
zhenshan.cao
d1d5f9e409
Replace proto.MarshalTextString with proto.Marshal (#8542) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-09-29 20:26:00 +08:00
congqixia
db944cd09c
Refactor flowgraph and related invocation (#8770) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-29 20:19:59 +08:00
Cai Yudong
d94361fdcc
Add parameter partitionID for API filterSegments (#8813) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-29 10:27:58 +08:00
XuanYang-cn
3be6672753
Add flush channel for delete_node (#8762) 
Resolves: #8761

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-28 18:22:16 +08:00
congqixia
d374ea71ba
Unify liveness check in datacoord and datanode (#8480) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-28 16:10:04 +08:00
XuanYang-cn
b150f57927
No buffer also means flushed (#8686) 
Resolves: #8684

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-28 11:18:03 +08:00
Xiaofan
585989935a
Revert the logger change (#8677) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-09-27 17:37:57 +08:00
godchen
51151b7b95
Fix delete node ut error (#8674) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-09-27 17:07:58 +08:00
godchen
77a773dff0
Change segment replica struct (#8608) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-09-27 14:38:00 +08:00
congqixia
e7117f67ee
Add BroadcastMark for Msgstream returning MessageIDs (#8654) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-27 14:10:09 +08:00
zhenshan.cao
b71bffe26f
Replace ChannelID with ShardName in DML Request (#8539) 
Signed-off-by: zhenshan.cao <zhenshan.cao@zilliz.com>
 2021-09-27 10:01:59 +08:00
Xiaofan
d38adda8c2
Rename log file based on node id (#8517) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-09-26 21:18:11 +08:00
XuanYang-cn
1307782093
[skip-ci]Fix golint in param_table in datanode (#8602) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-26 20:57:59 +08:00
XuanYang-cn
ebd894d8b7
Refactor buffer size calculation (#8565) 
This PR changes the datanode insertBufSize config
from 32000 number of rows to 16MB

Resolves: #7741

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-26 20:55:59 +08:00
XuanYang-cn
b376107ab3
[skip-ci]Fix golint in segmentRep (#8601) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-26 20:51:58 +08:00
XuanYang-cn
b13170f83e
[skip-ci]Fix golint in flowgraphnode.go (#8600) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-26 20:49:59 +08:00
groot
e4e762f2e8
Reduce go unittest time cost (#8503) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-09-26 12:50:24 +08:00
Cai Yudong
b2e5627061
Rename insertMsg to flowGraphMsg in datanode flow graph (#8546) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-26 10:43:57 +08:00
Cai Yudong
28d305148a
Update ddnode insert (#8536) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-26 09:43:57 +08:00
XuanYang-cn
3aabc1d3ee
[skip-ci]Fix golint in type_def.go (#8464) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-24 21:43:56 +08:00
sunby
816bac82ec
Add root path in minio and replace etcd.rootPath in datanode (#8272) 
issue: #7905
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-24 20:50:23 +08:00
XuanYang-cn
e4f0d4bab6
[skip-ci]Fix golint in flow_graph_message.go (#8463) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-24 20:46:04 +08:00
XuanYang-cn
2539dd4593
[skip-ci]Fix golint in data_node.go (#8462) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-24 20:44:05 +08:00
XuanYang-cn
faeb1bf50f
Add comment on exposed struct (#8348) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-23 20:15:54 +08:00
XuanYang-cn
2b82482679
Fix golint and ruleguard (#8347) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-23 19:14:01 +08:00
XuanYang-cn
a830c28bd8
Fix golint in datanode (#8345) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-23 19:05:55 +08:00
congqixia
2aa9deda1c
Improve logic when watch dm channel fails (#8386) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-23 18:31:55 +08:00
SS
58a178fdf5
[skip ci] Fix grammar in comment (#8382) 
Signed-off-by: sida shen <sida@sidas-MacBook-Pro.local>

Co-authored-by: sida shen <sida@sidas-MacBook-Pro.local>
 2021-09-23 16:50:16 +08:00
XuanYang-cn
d816a81ee3
Make FlushSegments async (#8159) 
Resolves: #8122

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-23 16:03:54 +08:00
congqixia
6469d83072
Add datanode etcd alive check (#8363) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-23 14:29:54 +08:00
Xiaofan
f5173b595f
Add a channel prefix for all channels (#8166) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-09-23 10:53:53 +08:00
godchen
050e27cbcf
[skip ci]Add delete node comment (#8323) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-09-22 20:02:09 +08:00
Xiangyu Wang
c314b2b4a8
Remove unused label rules (#7929) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-09-18 18:19:51 +08:00
groot
334297b49d
[skip ci] Update OWNERS files (#8211) 
Signed-off-by: yhmo <yihua.mo@zilliz.com>
 2021-09-18 15:30:37 +08:00
congqixia
bdfb8768d4
Enable gosec (#8202) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-18 14:45:50 +08:00
XuanYang-cn
46c72c57c2
Add BufferData in insertbufferNode (#8138) 
Optimize bufferInsertMsg param to avoid a second transfer
of endposition

See also: #8058, #7741

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-18 14:25:50 +08:00
Cai Yudong
d10c6abc9e
Apply for msgstream from pool when creating collection (#7738) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-18 09:13:50 +08:00
dragondriver
daf001f683
Expose created time and updated time to datanode metrics (#8173) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-17 21:32:47 +08:00
XuanYang-cn
6c434b4801
Improve codes in insertbuffernode (#8054) 
Add error handling in insertbuffernode

See also: #7624, #7684

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-17 16:27:56 +08:00
XuanYang-cn
82d4d19337
Remove not in use segmentCheckPoints (#8039) 
Resolves: #8037

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-17 16:25:55 +08:00
XuanYang-cn
3db76e8e59
[skip-ci]Add comment in data sync service (#8002) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-15 21:25:49 +08:00
XuanYang-cn
376d39fa02
[skip-ci]Add comment for Cache and allocator (#7924) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-15 10:33:37 +08:00
dragondriver
a872a4357c
Add more test cases for proxy implementation (#7685) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-11 11:36:22 +08:00
XuanYang-cn
ce33855652
[skip-ci]Fix datanode comment typo (#7689) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-10 14:46:00 +08:00
Cai Yudong
4e23ed9a18
Revert apply physical channels from pool (d5f7e358 and 3b960969) (#7687) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-09-10 13:00:00 +08:00
sunby
06cee85a39
Add data node unit tests (#7622) 
Issue: #6357
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-09-09 16:40:00 +08:00
XuanYang-cn
64aad49959
Add ut in datasync service to 88% (#7615) 
See also: #6357

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-09 15:36:01 +08:00
congqixia
69794fd32d
Refactor insertBufferNode and add unit tests (#7621) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-09-09 15:00:00 +08:00
XuanYang-cn
762060e670
Add ut for segmentReplica to 100% (#7589) 
See also: #6357
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-09 10:14:00 +08:00
XuanYang-cn
074b610811
Add ut for dmstream_input_node to 100% (#7593) 
See also: #6357

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-08 19:12:00 +08:00
XuanYang-cn
a5767e05c6
Add ut for flowgraph_delete_node to 100% (#7550) 
See also: #6357
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-08 10:41:59 +08:00
XuanYang-cn
29756c6ce8
Add ut for flow_graph_dd_node to 100% (#7511) 
This PR:
- Change mutex to sync.Map
- Add ut for flow_graph_dd_node.go to 100%

See also: #6357

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-09-07 15:41:59 +08:00
dragondriver
960741ee16
Support more detailed metrics information (#7472) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-03 19:35:47 +08:00
dragondriver
89748ac639
Expose metrics of data cluster (#7177) 
Signed-off-by: dragondriver <jiquan.long@zilliz.com>
 2021-09-01 10:13:15 +08:00
Cai Yudong
3b9609692b
Apply for msgstream from pool when creating collection (#7377) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-08-31 18:35:58 +08:00
Cai Yudong
f4b53df727
Revert "Apply for msgstream from pool when creating collection (#7323)" (#7376) 
This reverts commit e6de86a433.
 2021-08-31 11:07:58 +08:00
Cai Yudong
e6de86a433
Apply for msgstream from pool when creating collection (#7323) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-08-31 09:47:58 +08:00
congqixia
65f168eac1
Fix datanode corner cases (#7336) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-08-30 10:03:58 +08:00
godchen
c333af0dcf
Add query node segment filter (#7303) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-08-28 10:12:00 +08:00
godchen
b4c55f6b3c
Add delete node (#7261) 
Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-08-25 11:41:52 +08:00
congqixia
daae05641a
Construct bloom filter when inserting (#7268) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-08-25 10:29:52 +08:00
Xiaofan
58b2089692
Import Embedded Etcd Kv Part 1(#7112) (#7174) 
Signed-off-by: xiaofan-luan <xiaofan.luan@zilliz.com>
 2021-08-24 09:45:51 +08:00
congqixia
881d0af1fc
Add sync logic in TestWatchChannel (#7188) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-08-20 17:48:13 +08:00
sunby
4f7c3e7d3a
Save binlog path in SegmentInfo (#6590) 
issue: #6589
Signed-off-by: sunby <bingyi.sun@zilliz.com>
 2021-08-19 13:00:12 +08:00
XuanYang-cn
86ca96a02e
Fix master branch unittest error (#7083) 
Resolves: #7080

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-08-13 14:00:14 +08:00
XuanYang-cn
6c5be83ea1
Refactor NewEtcdKV API (#7075) 
This PR mainly did the following 3 things:

- Refactor NewEtcdKV API
- Add unittests for `kv/etcd/`, raise the coverage
up to 94%
- Correct some APIs from version to revision

The old NewEtcdKV() API took `clientv3.Client` as
one of the input, making everyone who using this API
had to create a new `clientv3.Client`. So there're a lot
of repeated codes everywhere.

Meanwhile, someplace didn't even check whether the
`clientv3.New()` returned error or not, and the old API
also didn't check whether the clientv3.Client was Nil.
This caused issue#6955.

Ectd's version and revision have nothing alike, they
represent different concepts, but our APIs kind of
mixed them all up. So I correct those APIs' names.

Resolves: #6955

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-08-13 11:04:09 +08:00
congqixia
e1b3273d1c
Add Datanode watch etcd channel (#6965) 
Signed-off-by: Congqi Xia <congqi.xia@zilliz.com>
 2021-08-13 10:50:09 +08:00
Xiangyu Wang
5e78fd6a97
[skip ci]Update approvers (#7063) 
Signed-off-by: Xiangyu Wang <xiangyu.wang@zilliz.com>
 2021-08-12 19:36:09 +08:00
XuanYang-cn
9d81a90402
Fix mem leak of flush cache in DataNode (#6862) 
Resolves: #6858

Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-08-11 14:24:09 +08:00
XuanYang-cn
47df4a2e57
Fix DataNode BackGroudGC problem (#6551) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-08-03 09:59:24 +08:00
bigsheeper
7307332fab
Add nil check (#6832) 
* add nil check

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* use grpc functions

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* use grpc functions

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>

* fix indexCoord crash

Signed-off-by: bigsheeper <yihao.dai@zilliz.com>
 2021-07-29 20:11:23 +08:00
godchen
db94d7771f
Read vector from disk (#6707) 
* Read vector from disk

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* go fmt

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix git action error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix test error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix action error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix caculate error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* change var name

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* remove unused method

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* remove unused method

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix len error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* remove unused code

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* change bytes to float method

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* change float to bytes method

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix action error

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-07-24 09:25:22 +08:00
XuanYang-cn
dca6c6afde
Add unittest for BackGroundGC in DataNode (#6522) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-07-15 10:05:55 +08:00
XuanYang-cn
f0a68704e2
Add unittests for getChannelName in DataNode (#6346) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-07-08 15:52:40 +08:00
godchen
49a0528d53
Fix trace (#6218) 
* Fix trace

Signed-off-by: godchen <qingxiang.chen@zilliz.com>

* fix go fmt

Signed-off-by: godchen <qingxiang.chen@zilliz.com>
 2021-06-30 16:18:13 +08:00
XuanYang-cn
357af89d55
Add unittest in segment_replica (#6197) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-30 10:26:12 +08:00
XuanYang-cn
769cbb292a
Update doc in datanode (#6194) 
Signed-off-by: yangxuan <xuan.yang@zilliz.com>
 2021-06-29 17:34:13 +08:00
yukun
39614aa8eb
Add rocksmq_path in config and paramtable (#6099) 
* Add rocksmq_path in config and paramtable

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Add rocksdbPath in NewRmsFactory

Signed-off-by: fishpenguin <kun.yu@zilliz.com>

* Change rdb default path to /vat/lib/milvus/rdb_data

Signed-off-by: fishpenguin <kun.yu@zilliz.com>
 2021-06-25 19:44:11 +08:00
Cai Yudong
a09a3a1905
rename service to coord (#6020) 
Signed-off-by: yudong.cai <yudong.cai@zilliz.com>
 2021-06-23 16:14:08 +08:00