mirror of
https://gitee.com/sjqzhang/go-fastdfs.git
synced 2024-11-29 17:57:41 +08:00
exchange chinese and english markdown
This commit is contained in:
jqzhang
parent
ae53b4acf2
commit
0843f48877
588
README-en.md
588
README-en.md
@ -1,100 +1,97 @@
|
||||
|
||||
# [中文](README-en.md) [English](README.md)
|
||||
# [中文](README.md) [English](README-en.md)
|
||||
|
||||
|
||||
|
||||
|
||||
# go-fastdfs是一个基于http协议的分布式文件系统，它基于大道至简的设计理念，一切从简设计，使得它的运维及扩展变得更加简单，它具有高性能、高可靠、无中心、免维护等优点。
|
||||
# go-fastdfs is a distributed file system based on http protocol. It is based on the design concept of avenue to simple. All the simple design makes its operation and expansion more simple. It has high performance, high reliability and no center. , maintenance-free and so on.
|
||||
|
||||
### 大家担心的是这么简单的文件系统，靠不靠谱，可不可以用于生产环境？答案是肯定的，正因为简单所以高效，因为简单所以稳定。如果你担心功能，那就跑单元测试，如果担心性能，那就跑压力测试，项目都自带了，跑一跑更放心^_^。
|
||||
### Everyone is worried about such a simple file system. Is it not reliable, can it be used in a production environment? The answer is yes, it is efficient because it is simple, and it is stable because it is simple. If you are worried about the function, then run the unit test, if you are worried about the performance, then run the stress test, the project comes with it, run more confident ^_^.
|
||||
|
||||
注意：使用前请认真阅读完本文，特别是[wiki](https://github.com/sjqzhang/go-fastdfs/wiki)
|
||||
Note: Please read this article carefully before using it, especially [wiki](https://github.com/sjqzhang/go-fastdfs/wiki)
|
||||
|
||||
- 支持curl命令上传
|
||||
- 支持浏览器上传
|
||||
- 支持HTTP下载
|
||||
- 支持多机自动同步
|
||||
- 支持断点下载
|
||||
- 支持配置自动生成
|
||||
- 支持小文件自动合并(减少inode占用)
|
||||
- 支持秒传
|
||||
- 支持跨域访问
|
||||
- 支持一键迁移
|
||||
- 支持并行体验
|
||||
- 支持断点续传([tus](https://tus.io/))
|
||||
- 支持docker部署
|
||||
- 支持自监控告警
|
||||
- 支持图片缩放
|
||||
- 支持google认证码
|
||||
- 支持自定义认证
|
||||
- 支持集群文件信息查看
|
||||
- 使用通用HTTP协议
|
||||
- 无需专用客户端（支持wget,curl等工具）
|
||||
- 类fastdfs
|
||||
- 高性能 （使用leveldb作为kv库）
|
||||
- 高可靠（设计极其简单，使用成熟组件）
|
||||
- 无中心设计(所有节点都可以同时读写)
|
||||
- Support curl command upload
|
||||
- Support browser upload
|
||||
- Support HTTP download
|
||||
- Support multi-machine automatic synchronization
|
||||
- Support breakpoint download
|
||||
- Support configuration automatic generation
|
||||
- Support small file automatic merge (reduce inode occupancy)
|
||||
- Support for second pass
|
||||
- Support for cross-domain access
|
||||
- Support one-click migration
|
||||
- Support for parallel experience
|
||||
- Support for breakpoint resuming ([tus](https://tus.io/))
|
||||
- Support for docker deployment
|
||||
- Support self-monitoring alarm
|
||||
- Support image zoom
|
||||
- Support google authentication code
|
||||
- Support for custom authentication
|
||||
- Support cluster file information viewing
|
||||
- Use the universal HTTP protocol
|
||||
- No need for a dedicated client (support wget, curl, etc.)
|
||||
- class fastdfs
|
||||
- High performance (using leveldb as a kv library)
|
||||
- High reliability (design is extremely simple, using mature components)
|
||||
- No center design (all nodes can read and write at the same time)
|
||||
|
||||
# 优点
|
||||
# advantage
|
||||
|
||||
- 无依赖(单一文件）
|
||||
- 自动同步
|
||||
- 失败自动修复
|
||||
- 按天分目录方便维护
|
||||
- 支持不同的场景
|
||||
- 文件自动去重
|
||||
- 支持目录自定义
|
||||
- 支持保留原文件名
|
||||
- 支持自动生成唯一文件名
|
||||
- 支持浏览器上传
|
||||
- 支持查看集群文件信息
|
||||
- 支持集群监控邮件告警
|
||||
- 支持小文件自动合并(减少inode占用)
|
||||
- 支持秒传
|
||||
- 支持图片缩放
|
||||
- 支持google认证码
|
||||
- 支持自定义认证
|
||||
- 支持跨域访问
|
||||
- 极低资源开销
|
||||
- 支持断点续传([tus](https://tus.io/))
|
||||
- 支持docker部署
|
||||
- 支持一键迁移（从其他系统文件系统迁移过来）
|
||||
- 支持并行体验（与现有的文件系统并行体验，确认OK再一键迁移）
|
||||
- 支持token下载 token=md5(file_md5+timestamp)
|
||||
- 运维简单，只有一个角色（不像fastdfs有三个角色Tracker Server,Storage Server,Client），配置自动生成
|
||||
- 每个节点对等（简化运维）
|
||||
- 所有节点都可以同时读写
|
||||
- No dependencies (single file)
|
||||
- Automatic synchronization
|
||||
- Failure automatic repair
|
||||
- Convenient maintenance by talent directory
|
||||
- Support different scenarios
|
||||
- Automatic file deduplication
|
||||
- Support for directory customization
|
||||
- Support to retain the original file name
|
||||
- Support for automatic generation of unique file names
|
||||
- Support browser upload
|
||||
- Support for viewing cluster file information
|
||||
- Support cluster monitoring email alarm
|
||||
- Support small file automatic merge (reduce inode occupancy)
|
||||
- Support for second pass
|
||||
- Support image zoom
|
||||
- Support google authentication code
|
||||
- Support for custom authentication
|
||||
- Support for cross-domain access
|
||||
- Very low resource overhead
|
||||
- Support for breakpoint resuming ([tus](https://tus.io/))
|
||||
- Support for docker deployment
|
||||
- Support for one-click migration (migrated from other system file systems)
|
||||
- Support for parallel experience (parallel experience with existing file system, confirm OK and then one-click migration)
|
||||
- Support token download token=md5(file_md5+timestamp)
|
||||
- Easy operation and maintenance, only one role (unlike fastdfs has three roles Tracker Server, Storage Server, Client), the configuration is automatically generated
|
||||
- Peer-to-peer (simplified operation and maintenance)
|
||||
- All nodes can read and write simultaneously
|
||||
|
||||
|
||||
|
||||
# 启动服务器（已编译，[下载](https://github.com/sjqzhang/fastdfs/releases)极速体验，只需一分钟）
|
||||
一键安装：（请将以下命令复制到linux console中执行）
|
||||
```shell
|
||||
wget --no-check-certificate https://github.com/sjqzhang/go-fastdfs/releases/download/v1.2.6/fileserver -O fileserver && chmod +x fileserver && ./fileserver
|
||||
#Start the server (compiled, [download](https://github.com/sjqzhang/fastdfs/releases) experience)
|
||||
```
|
||||
./fileserver
|
||||
```
|
||||
(注意：下载时要注意链接的版本号，windows下直接运行fileserver.exe，执行文件在这里[下载](https://github.com/sjqzhang/fastdfs/releases))
|
||||
|
||||
# 命令上传
|
||||
|
||||
#Command upload
|
||||
|
||||
`curl -F file=@http-index-fs http://10.1.xx.60:8080/upload`
|
||||
|
||||
|
||||
# WEB上传（浏览器打开）
|
||||
# WEB upload (browser open)
|
||||
|
||||
`http://yourserver ip:8080/upload.html` 注意：不要使用127.0.0.1上传
|
||||
`http://127.0.0.1:8080`
|
||||
|
||||
# 代码上传(选项参阅浏览器上传)
|
||||
|
||||
## python版本:
|
||||
#Code upload (options see browser upload)
|
||||
## python
|
||||
```python
|
||||
import requests
|
||||
url = 'http://10.1.5.9:8080/upload'
|
||||
url = 'http://127.0.0.1:8080/upload'
|
||||
files = {'file': open('report.xls', 'rb')}
|
||||
options={'output':'json','path':'','scene':''} #参阅浏览器上传的选项
|
||||
r = requests.post(url,data=options, files=files)
|
||||
options={'output':'json','path':'','scene':''} #See browser upload options
|
||||
r = requests.post(url, files=files)
|
||||
print(r.text)
|
||||
```
|
||||
## golang版本
|
||||
## golang
|
||||
```go
|
||||
package main
|
||||
|
||||
@ -106,16 +103,17 @@ import (
|
||||
func main() {
|
||||
var obj interface{}
|
||||
req:=httplib.Post("http://10.1.5.9:8080/upload")
|
||||
req.PostFile("file","filename")//注意不是全路径
|
||||
req.PostFile("file","path/to/file")
|
||||
req.Param("output","json")
|
||||
req.Param("scene","")
|
||||
req.Param("path","")
|
||||
req.ToJSON(&obj)
|
||||
fmt.Print(obj)
|
||||
}
|
||||
```
|
||||
## java版本
|
||||
依赖(这里使用了hutool工具包,更简便)
|
||||
````
|
||||
|
||||
## java
|
||||
|
||||
```xml
|
||||
<dependency>
|
||||
<groupId>cn.hutool</groupId>
|
||||
@ -123,32 +121,22 @@ func main() {
|
||||
<version>4.4.3</version>
|
||||
</dependency>
|
||||
```
|
||||
上传代码
|
||||
|
||||
```java
|
||||
public static void main(String[] args) {
|
||||
//文件地址
|
||||
File file = new File("D:\\git\\2.jpg");
|
||||
//声明参数集合
|
||||
HashMap<String, Object> paramMap = new HashMap<>();
|
||||
//文件
|
||||
paramMap.put("file", file);
|
||||
//输出
|
||||
paramMap.put("output","json");
|
||||
//自定义路径
|
||||
paramMap.put("path","image");
|
||||
//场景
|
||||
paramMap.put("scene","image");
|
||||
//上传
|
||||
String result= HttpUtil.post("http://xxxxx:xxxx/upload", paramMap);
|
||||
//输出json结果
|
||||
System.out.println(result);
|
||||
}
|
||||
```
|
||||
[更多语言请参考](doc/upload.md)
|
||||
|
||||
# 断点续传示例
|
||||
|
||||
## golang版本
|
||||
# Breakpoint resume example
|
||||
## golang
|
||||
```go
|
||||
package main
|
||||
|
||||
@ -177,364 +165,350 @@ func main() {
|
||||
fmt.Println( uploader.Upload())
|
||||
|
||||
}
|
||||
|
||||
````
|
||||
[更多客户端请参考](https://github.com/tus)
|
||||
[more langue](doc/upload.md)
|
||||
|
||||
|
||||
|
||||
Universal file authentication timing diagram
|
||||
|
||||
|
||||
部署图
|
||||
|
||||
File google authentication timing diagram
|
||||
|
||||
|
||||
通用文件认证时序图
|
||||
|
||||
# Please click [Feedback](https://github.com/sjqzhang/go-fastdfs/issues/new)
|
||||
|
||||
文件google认证时序图
|
||||
|
||||
|
||||
# 有问题请[点击反馈](https://github.com/sjqzhang/go-fastdfs/issues/new)
|
||||
|
||||
# 重要说明
|
||||
## 在issue中有很多实际使用的问题及回答（很多已关闭，请查看已关闭的issue）
|
||||
|
||||
## 项目从v1.1.8开始进入稳定状态
|
||||
|
||||
# 更新说明
|
||||
## 从低版升给到高版本，可能存在配置项变动的情况，一定要注意使用新的版本时的配置项。如何获得新版本的配置项及说明？先备份旧的配置项（文件名不能为cfg.json），再运行新的版本，配置项就自动生成。然后再修改相应的配置项。
|
||||
|
||||
- v1.1.9 增加文件自动迁移功能，支持同名文件重复覆盖选项。
|
||||
|
||||
# <span id="qa">Q&A</span>
|
||||
|
||||
- 最佳实战？
|
||||
- Best practice?
|
||||
```
|
||||
一、如果是海量存储，不要开启文件token认证功能，减少性能开消。
|
||||
二、尽量用标准上传，上传后业务保存path，在业务用的时候再并接上域名（方便迁移扩展等）。
|
||||
三、如果使用断点续传，上传后一定要用文件id置换成path存储（如何置换看ＱＡ/API文档），为后面访问减少性能开消。
|
||||
四、尽量使用物理服务器部署，因为主要压力或性能来自于IO
|
||||
五、线上业务尽量使用nginx+gofastdfs部署架构(均衡算法使用ip_hash)，以满足后面的功能扩展性(nginx+lua)。
|
||||
六、线上环境最好不要使用容器部署，容器适用于测试和功能验证。
|
||||
总结：业务保存的文件的path,减少后期访问路径转换带来开消,文件访问权限由业务来完成，这样性能最好，通用性强（可直接其它web服务器）。
|
||||
First, if it is mass storage, do not open the file token authentication function to reduce performance.
|
||||
Second, try to use the standard upload, upload the business to save the path, and then connect the domain name when the business is used (convenient migration extension, etc.).
|
||||
Third, if you use breakpoints to continue transmission, you must use the file id to replace the path storage after uploading (how to replace the QA/API document), to reduce performance for subsequent access.
|
||||
Fourth, try to use physical server deployment, because the main pressure or performance comes from IO
|
||||
Fifth, the online business should use the nginx+gofastdfs deployment architecture (the equalization algorithm uses ip_hash) to meet the later functional scalability (nginx+lua).
|
||||
Sixth, the online environment is best not to use container deployment, the container is suitable for testing and functional verification.
|
||||
Summary: The path of the file saved by the business reduces the conversion of the later access path, and the file access permission is completed by the service, so that the performance is the best and the versatility is strong (can be directly connected to other web servers).
|
||||
|
||||
重要提醒：如果开启小文件合并功能，后期是无法删除小文件的。
|
||||
上传结果说明
|
||||
请使用md5,path,scene字段，其它是为了兼容老的线上系统添加的，以后有可能去掉。
|
||||
Important reminder: If the small file merge function is enabled, it is impossible to delete small files later.
|
||||
Upload result description
|
||||
Please use md5, path, scene field, others are added to be compatible with the old online system, and may be removed in the future.
|
||||
|
||||
```
|
||||
|
||||
- 有API文档么？
|
||||
[API文档](doc/api.md)
|
||||
- Is there an API document?
|
||||
[API documentation] (doc/api.md)
|
||||
|
||||
- 有管理后台么？
|
||||
- Is there a management background?
|
||||
```
|
||||
https://github.com/perfree/go-fastdfs-web
|
||||
Https://github.com/perfree/go-fastdfs-web
|
||||
```
|
||||
- 断点上传有使用说明么？
|
||||
- Is there any instructions for using the breakpoint upload?
|
||||
```
|
||||
https://github.com/tus
|
||||
Https://github.com/tus
|
||||
```
|
||||
|
||||
- 在微信讨论群中大家都问到go-fastdfs性能怎样？
|
||||
- In the WeChat discussion group, everyone asked about the performance of go-fastdfs?
|
||||
```
|
||||
由于问的人太多，在这里统一回答。
|
||||
go-fastdfs的文件定位与其它分布式系统不同，它的寻址是直接定位，不经过任何组件，所以可以近似时间复杂度为o(1)[文件路径定位]
|
||||
基本没有性能损耗，项目中也附有压测脚本，大家可以自已进行压测，群里就不要太多讨论问题了，人多每次回复同样的问题
|
||||
大家也会觉得这群无聊。
|
||||
Because there are too many people asking, answer here in unison.
|
||||
The file location of go-fastdfs is different from other distributed systems. Its addressing is directly located without any components, so the approximate time complexity is o(1)[file path location]
|
||||
There is basically no performance loss. The project also has a pressure test script. You can carry out the pressure test yourself. Don’t discuss the problem too much in the group. People reply to the same question every time.
|
||||
Everyone will also feel that this group is boring.
|
||||
```
|
||||
|
||||
|
||||
|
||||
- 已经使用fastdfs存储的文件可以迁移到go fastdfs下么（其它迁移情况可按类似的方式处理，极速体验也用类似的方式）？
|
||||
- Files that have been stored using fastdfs can be migrated to go fastdfs (other migrations can be handled in a similar manner, and the speed experience is similar)?
|
||||
```
|
||||
答案是可以的，你担心的问题是路径改变,go fastdfs为你考虑了这一点
|
||||
步骤：
|
||||
一、下载最新版的go-fastdfs
|
||||
二、将原来的fastdfs文件目录复制到go-fastdfs的 files目录下(如果文件很多，可以逆向过来，将fileserver复制过去，但要保留fileserver的目录结构）
|
||||
三、将配置enable_migrate设为true
|
||||
注意：迁移过程中会扫描整下files目录下的所有文件，
|
||||
速度较慢，迁移完成后请将enable_migrate设为false
|
||||
The answer is yes, the problem you are worried about is the path change, go fastdfs considers this for you.
|
||||
step:
|
||||
First, download the latest version of go-fastdfs
|
||||
Second, copy the original fastdfs file directory to the files directory of go-fastdfs (if there are a lot of files, you can reverse it, copy the fileserver, but keep the fileserver directory structure)
|
||||
Third, set the enable_migrate to true
|
||||
Note: All files in the files directory will be scanned during the migration process.
|
||||
Slower, set enable_migrate to false after migration is complete
|
||||
|
||||
说明：go-fastdfs的目录是不能变动的，与同步机制相关，很多同学在群里问题，我的files目录能不能自定义，答案是否定的。
|
||||
至于能不能用软链接的方式本人没有测试过，可以自行测试。
|
||||
Note: The directory of go-fastdfs can not be changed, related to the synchronization mechanism, many students in the group, my files directory can not be customized, the answer is no.
|
||||
As for whether or not I can use the soft link, I have not tested it and can test it myself.
|
||||
|
||||
```
|
||||
|
||||
- 什么是集群，如何用Nginx管理多集群？
|
||||
- What is a cluster, how to manage multiple clusters with Nginx?
|
||||
```
|
||||
1、在go-fastdfs中，一个集群就是一个group。
|
||||
2、请参阅部署图
|
||||
注意：配置中的 support_group_manage 参数设为true时，所有的url中都自动添加组信息。
|
||||
例如：http://10.1.5.9:8080/group/status
|
||||
默认：http://10.1.5.9:8080/status
|
||||
区别：多了group,对应配置中的 group 参数,这样主要是为了解决一个Nginx反向代理多个group(集群)
|
||||
具体请参阅部署图
|
||||
1. In go-fastdfs, a cluster is a group.
|
||||
2, please refer to the deployment diagram
|
||||
Note: When the support_group_manage parameter in the configuration is set to true, group information is automatically added to all urls.
|
||||
For example: http://10.1.5.9:8080/group/status
|
||||
Default: http://10.1.5.9:8080/status
|
||||
The difference: more group, corresponding to the group parameter in the configuration, so mainly to solve a Nginx reverse proxy multiple groups (cluster)
|
||||
Please refer to the deployment diagram for details.
|
||||
|
||||
```
|
||||
|
||||
|
||||
- 如何搭建集群？
|
||||
- How to build a cluster?
|
||||
```
|
||||
一、先下载已编译的可执行文件（用最新版本）
|
||||
二、运行可执行文件（生成配置）
|
||||
三、修改配置
|
||||
peers：增加对端的http地址
|
||||
检查:
|
||||
host:自动生成是否正确
|
||||
peer_id:集群内是否唯一
|
||||
四、重新运行服器
|
||||
五、验证服务是否OK
|
||||
First, download the compiled executable file (with the latest version)
|
||||
Second, run the executable file (generate configuration)
|
||||
Third, modify the configuration
|
||||
Peers: increase the peer's http address
|
||||
an examination:
|
||||
Host: Is the automatic generation correct?
|
||||
Peer_id: Is it unique within the cluster?
|
||||
Fourth, re-run the server
|
||||
5. Is the verification service OK?
|
||||
```
|
||||
|
||||
|
||||
- 适合海量存储吗？
|
||||
- Is it suitable for mass storage?
|
||||
```
|
||||
答案：适合海量存储
|
||||
Answer: Suitable for mass storage
|
||||
```
|
||||
|
||||
- 如何上传文件夹？
|
||||
- How to upload a folder?
|
||||
```
|
||||
DIR=log && ls $DIR |xargs -n 1 -I {} curl -s -F path=$DIR -F file=@$DIR/{} http://10.1.50.90:8080/upload
|
||||
上面命令的log为要上传的目录，如果代码上传就是简单的循环上传就ＯＫ。
|
||||
The log of the above command is the directory to be uploaded. If the code upload is a simple loop upload, it will be OK.
|
||||
```
|
||||
|
||||
- 如何缩放图片？
|
||||
- How to scale the picture?
|
||||
```
|
||||
在下载url中加入width各height参数
|
||||
例如：http://127.0.0.1:8080/group1/haystack/5/124,0,27344,.jpg?download=0&width=100&height=100
|
||||
特明说明是：如果要等比例缩放，请将width或height设为０
|
||||
Add width parameter in the download url
|
||||
For example: http://127.0.0.1:8080/group1/haystack/5/124,0,27344,.jpg?download=0&width=100&height=100
|
||||
The special explanation is: if you want to scale, please set the width or height to 0.
|
||||
```
|
||||
|
||||
- 如何在浏览器中直接显示图片？
|
||||
- How to display images directly in the browser?
|
||||
```
|
||||
在下载url中加入download=0参数
|
||||
例如：http://127.0.0.1:8080/group1/haystack/5/124,0,27344,.jpg?download=0
|
||||
Add the download=0 parameter to the download url.
|
||||
For example: http://127.0.0.1:8080/group1/haystack/5/124,0,27344,.jpg?download=0
|
||||
```
|
||||
|
||||
|
||||
- 如何实现自定义认证上传下载？
|
||||
- How to implement custom authentication upload and download?
|
||||
```
|
||||
一、使用1.2.6版本以后的go-fastdfs
|
||||
二、设auth_url参数（应用提供）
|
||||
三、应用实现验证权限接口（即第二步的url）,参数为 auth_toke 返回 ok 表示认证通过，其它为不通过
|
||||
四、认证通过后，可以上传或下载
|
||||
First, the use of the 1.2.6 version of the go-fastdfs
|
||||
Second, set the auth_url parameter (provided by the application)
|
||||
Third, the application implements the authentication permission interface (that is, the url of the second step), the parameter is auth_toke, ok, the authentication is passed, and the others are not passed.
|
||||
Fourth, after the certification is passed, you can upload or download
|
||||
```
|
||||
|
||||
|
||||
- 还需要安装nginx么？
|
||||
- Need to install nginx yet?
|
||||
```
|
||||
go-fastdfs本身是一个高性能的web服务器，在开发或测试时，可以不用安装nginx，
|
||||
但go-fastdfs的功能单一，如需要缓存或重定向或其它扩展，nginx都有成熟的组件
|
||||
所以建议线上还是加一层nginx，再借助nginx+lua解决扩展性问题。
|
||||
Go-fastdfs itself is a high-performance web server, you don't need to install nginx when developing or testing.
|
||||
But go-fastdfs has a single function, such as caching or redirection or other extensions, nginx has mature components
|
||||
Therefore, it is recommended to add a layer of nginx on the line, and then use nginx+lua to solve the scalability problem.
|
||||
```
|
||||
|
||||
- 能动态加载配置么？
|
||||
- Can I dynamically load the configuration?
|
||||
```
|
||||
答案：是可以的，但要更新到最新版本
|
||||
步骤：
|
||||
1）修改 conf/cfg.json 文件
|
||||
2）访问 http://10.1.xx.60:8080/reload
|
||||
3） 注意：每个节点都需要进行同样的操作
|
||||
Answer: Yes, but update to the latest version
|
||||
step:
|
||||
1) Modify the conf/cfg.json file
|
||||
2) Visit http://10.1.xx.60:8080/reload
|
||||
3) Note: Every node needs to do the same thing
|
||||
```
|
||||
|
||||
|
||||
- 如何查看集群文件信息？
|
||||
- How to view cluster file information?
|
||||
```
|
||||
http://10.1.xx.60:8080/stat
|
||||
Http://10.1.xx.60:8080/stat
|
||||
|
||||
如果出现文件统计出错怎么办？
|
||||
请删除 data目录下的 stat.json文件 重启服务，请系统自动重新计算文件数。
|
||||
What should I do if there is a file error?
|
||||
Please delete the stat.json file in the data directory. Restart the service, please recalculate the number of files automatically.
|
||||
|
||||
或者调用
|
||||
http://10.1.xx.60:8080/repair_stat
|
||||
Or call
|
||||
Http://10.1.xx.60:8080/repair_stat
|
||||
|
||||
```
|
||||
- 可靠性怎样，能用于生产环境么？
|
||||
- How reliable can it be used in a production environment?
|
||||
```
|
||||
本项目已大规模用于生产环境，如担心不能满足
|
||||
可以在使用前对其各项特性进行压力测试，有任何
|
||||
问题可以直接提issue
|
||||
This project has been used on a large scale in the production environment, such as fear of not meeting
|
||||
You can stress test its features before use, any
|
||||
The problem can be directly mentioned
|
||||
```
|
||||
|
||||
- 如何后台运行程序？
|
||||
- How to run the program in the background?
|
||||
```
|
||||
请使用control 对程序进行后面运行，具体操作如下:
|
||||
一、 chmod +x control
|
||||
二、 确保control与fileserver在同一个目录
|
||||
三、 ./control start|stop|status #对和序进行启动，停止，查看状态等。
|
||||
Please use control to run the program later, as follows:
|
||||
First, chmod +x control
|
||||
Second, make sure the control and fileserver are in the same directory
|
||||
Third, ./control start|stop|status #Start and start the sequence, stop, view the status, and so on.
|
||||
|
||||
```
|
||||
|
||||
|
||||
- 能不能在一台机器部置多个服务端？
|
||||
- Can I have multiple servers on one machine?
|
||||
```
|
||||
不能，在设计之初就已考虑到集群的高可用问题，为了保证集群的真正可用，必须为不同的ip,ip 不能用 127.0.0.1
|
||||
错误 "peers": ["http://127.0.0.1:8080","http://127.0.0.1:8081","http://127.0.0.1:8082"]
|
||||
正确 "peers": ["http://10.0.0.3:8080","http://10.0.0.4:8080","http://10.0.0.5:8080"]
|
||||
No, the high availability of the cluster has been considered at the beginning of the design. In order to ensure the true availability of the cluster, it must be different for ip, ip can not use 127.0.0.1
|
||||
Error "peers": ["http://127.0.0.1:8080","http://127.0.0.1:8081","http://127.0.0.1:8082"]
|
||||
Correct "peers": ["http://10.0.0.3:8080","http://10.0.0.4:8080","http://10.0.0.5:8080"]
|
||||
```
|
||||
- 文件不同步了怎么办？
|
||||
- What should I do if the files are not synchronized?
|
||||
```
|
||||
正常情况下，集群会每小时自动同步修复文件。（性能较差，在海量情况下建议关闭自动修复）
|
||||
那异常情况下怎么？
|
||||
答案：手动同步（最好在低峰执行）
|
||||
http://172.16.70.123:7080/sync?date=20190117&force=1 (说明：要在文件多的服务器上执行，相关于推送到别的服务器)
|
||||
参数说明：date 表示同步那一天的数据 force 1.表示是否强制同步当天所有(性能差)，0.表示只同步失败的文件
|
||||
Under normal circumstances, the cluster automatically synchronizes the repair files every hour. (The performance is poor, it is recommended to turn off automatic repair in case of massive)
|
||||
What about the abnormal situation?
|
||||
Answer: Manual synchronization (preferably at low peaks)
|
||||
Http://172.16.70.123:7080/sync?date=20190117&force=1 (Note: To be executed on a server with many files, related to pushing to another server)
|
||||
Parameter description: date indicates the data of the day of synchronization. force 1. indicates whether to force synchronization of all the day (poor performance), 0. means that only failed files are synchronized.
|
||||
|
||||
不同步的情况：
|
||||
1) 原来运行N台，现在突然加入一台变成N+1台
|
||||
2）原来运行N台，某一台机器出现问题，变成N-1台
|
||||
Unsynchronized situation:
|
||||
1) Originally running N sets, now suddenly join one to become N+1
|
||||
2) Originally running N sets, one machine has a problem and becomes N-1
|
||||
|
||||
如果出现多天数据不一致怎么办？能一次同步所有吗？
|
||||
答案是可以：(最好在低峰执行)
|
||||
http://172.16.70.123:7080/repair?force=1
|
||||
What if there is a multi-day data inconsistency? Can I synchronize all at once?
|
||||
The answer is yes: (preferably at low peaks)
|
||||
Http://172.16.70.123:7080/repair?force=1
|
||||
|
||||
```
|
||||
|
||||
- 文件不同步会影响访问吗？
|
||||
- Does the file out of sync affect access?
|
||||
```
|
||||
答案：不会影响，会在访问不到时，自动修复不同步的文件。
|
||||
Answer: It will not affect, it will automatically repair the files that are not synchronized when the access is not available.
|
||||
```
|
||||
|
||||
- 如何查看系统状态及说明？
|
||||
- How do I check the system status and description?
|
||||
```
|
||||
http://172.16.70.123:7080/status
|
||||
注意:（Fs.Peers是不带本机的，如果带有可能出问题）
|
||||
本机为 Fs.Local
|
||||
sts["Fs.ErrorSetSize"] = this.errorset.Cardinality() 这个会导致内存增加
|
||||
Http://172.16.70.123:7080/status
|
||||
Note: (Fs.Peers is not equipped with this unit, if there is a problem with the problem)
|
||||
This machine is Fs.Local
|
||||
Sts["Fs.ErrorSetSize"] = this.errorset.Cardinality() This will cause memory to increase
|
||||
|
||||
```
|
||||
|
||||
|
||||
- 如何编译(go1.9.2+)？
|
||||
- How to compile (go1.9.2+)?
|
||||
```
|
||||
git clone https://github.com/sjqzhang/go-fastdfs.git
|
||||
cd go-fastdfs
|
||||
mv vendor src
|
||||
pwd=`pwd`
|
||||
Git clone https://github.com/sjqzhang/go-fastdfs.git
|
||||
Cd go-fastdfs
|
||||
Mv vendor src
|
||||
Pwd=`pwd`
|
||||
GOPATH=$pwd go build -o fileserver fileserver.go
|
||||
```
|
||||
|
||||
- 如何跑单元测试 (尽量在linux下进行)？
|
||||
- How to run a unit test (try to do it under linux)?
|
||||
```
|
||||
|
||||
git clone https://github.com/sjqzhang/go-fastdfs.git
|
||||
cd go-fastdfs
|
||||
mv vendor src
|
||||
pwd=`pwd`
|
||||
Git clone https://github.com/sjqzhang/go-fastdfs.git
|
||||
Cd go-fastdfs
|
||||
Mv vendor src
|
||||
Pwd=`pwd`
|
||||
GOPATH=$pwd go test -v fileserver.go fileserver_test.go
|
||||
|
||||
```
|
||||
|
||||
|
||||
|
||||
- 如何压测？
|
||||
- How to test?
|
||||
```
|
||||
步骤：
|
||||
一、创建files文件夹
|
||||
二、将gen_file.py复制到files文件夹中，通过python gen_file.py 生成大量文件
|
||||
三、将benchmark.py放到 files目录外（即与files目录同一级），通过python benchmark.py进行压测（注意对benchmark.py中的ip进行修改）
|
||||
先用gen_file.py产生大量文件（注意如果要生成大文件，自已在内容中乘上一个大的数即可）
|
||||
例如:
|
||||
step:
|
||||
First, create a files folder
|
||||
Second, copy gen_file.py to the files folder, generate a large number of files through python gen_file.py
|
||||
Third, put benchmark.py outside the files directory (that is, the same level as the files directory), press the python benchmark.py (note the ip in the benchmark.py)
|
||||
First use gen_file.py to generate a large number of files (note that if you want to generate large files, you can multiply the content by a large number)
|
||||
E.g:
|
||||
# -*- coding: utf-8 -*-
|
||||
import os
|
||||
Import os
|
||||
j=0
|
||||
for i in range(0,1000000):
|
||||
if i%1000==0:
|
||||
For i in range(0,1000000):
|
||||
If i%1000==0:
|
||||
j=i
|
||||
os.system('mkdir %s'%(i))
|
||||
with open('%s/%s.txt'%(j,i),'w+') as f:
|
||||
f.write(str(i)*1024)
|
||||
接着用benchmark.py进行压测
|
||||
也可以多机同时进行压测，所有节点都是可以同时读写的
|
||||
Os.system('mkdir %s'%(i))
|
||||
With open('%s/%s.txt'%(j,i),'w+') as f:
|
||||
F.write(str(i)*1024)
|
||||
Then use benchmark.py for pressure measurement
|
||||
It is also possible to perform pressure measurement simultaneously in multiple machines, and all nodes can be read and written simultaneously.
|
||||
```
|
||||
|
||||
- 代码为什么写在一个文件中？
|
||||
- Why is the code written in a file?
|
||||
```
|
||||
一、目前的代码还非常简单，没必要弄得太复杂。
|
||||
二、个人理解模块化不是分开多个文件就表示模块化，大家可以用IDE去看一下代码结构，其实是已经模块化的。
|
||||
First, the current code is still very simple, no need to make it too complicated.
|
||||
Second, the individual understands that modularity is not modular when multiple files are separated. You can use the IDE to look at the code structure, which is actually modular.
|
||||
```
|
||||
|
||||
- 支持断点下载？
|
||||
- Support for breakpoint downloads?
|
||||
```
|
||||
答案：支持
|
||||
curl wget 如何
|
||||
wget -c http://10.1.5.9:8080/group1/default/20190128/16/10/2G
|
||||
culr -C - http://10.1.5.9:8080/group1/default/20190128/16/10/2G
|
||||
Answer: Support
|
||||
Curl wget how
|
||||
Wget -c http://10.1.5.9:8080/group1/default/20190128/16/10/2G
|
||||
Culr -C - http://10.1.5.9:8080/group1/default/20190128/16/10/2G
|
||||
```
|
||||
|
||||
- Docker如何部署？
|
||||
- How is Docker deployed?
|
||||
```
|
||||
步骤：
|
||||
方式一、
|
||||
一、构建镜像
|
||||
docker build . -t fastdfs
|
||||
二、运行容器（使用环境变量 GO_FASTDFS_DIR 指向存储目录。）
|
||||
docker run --name fastdfs -v /data/fastdfs_data:/data -e GO_FASTDFS_DIR=/data fastdfs
|
||||
方式二、
|
||||
一、拉取镜像
|
||||
docker pull sjqzhang/go-fastdfs
|
||||
二、运行容器
|
||||
docker run --name fastdfs -v /data/fastdfs_data:/data -e GO_FASTDFS_DIR=/data fastdfs
|
||||
step:
|
||||
method one,
|
||||
First, build a mirror
|
||||
Docker build . -t fastdfs
|
||||
Second, run the container (use the environment variable GO_FASTDFS_DIR to point to the storage directory.)
|
||||
Docker run --name fastdfs -v /data/fastdfs_data:/data -e GO_FASTDFS_DIR=/data fastdfs
|
||||
Method 2,
|
||||
First, pull the mirror
|
||||
Docker pull sjqzhang/go-fastdfs
|
||||
Second, run the container
|
||||
Docker run --name fastdfs -v /data/fastdfs_data:/data -e GO_FASTDFS_DIR=/data fastdfs
|
||||
|
||||
```
|
||||
|
||||
- 大文件如何分块上传或断点续传？
|
||||
- How to upload large files or breakpoints?
|
||||
```
|
||||
一般的分块上传都要客户端支持，而语言的多样性，客户端难以维护，但分块上传的功能又有必要，为此提供一个简单的实现思路。
|
||||
方案一、
|
||||
借助linux split cat 实现分割与合并，具体查看 split 与 cat 帮助。
|
||||
分割： split -b 1M filename #按每个文1M
|
||||
合并： cat x* > filename #合并
|
||||
方案二、
|
||||
借助hjsplit
|
||||
http://www.hjsplit.org/
|
||||
具体自行实现
|
||||
方案三、
|
||||
建议用go实现hjsplit分割合并功，这样具有跨平台功能。（未实现，等你来....）
|
||||
方案四、
|
||||
使用内置的继点续传功能（使用protocol for resumable uploads协议，[详情](https://tus.io/)）
|
||||
注意：方案四、只能指定一个上传服务器，不支持同时写，并且上传的url有变化
|
||||
原上传url： http://10.1.5.9:8080/<group>/upload
|
||||
断点上传url： http://10.1.5.9:8080/<group>/big/upload/
|
||||
上传完成，再通过秒传接口，获取文件信息
|
||||
The general block uploading must be supported by the client, and the diversity of the language is difficult for the client to maintain, but the function of the block uploading is necessary, and a simple implementation idea is provided for this.
|
||||
Option One,
|
||||
Split and merge with linux split cat, see the split and cat help.
|
||||
Split: split -b 1M filename #1M per text
|
||||
Merge: cat x* > filename #merge
|
||||
Option II,
|
||||
With hjsplit
|
||||
Http://www.hjsplit.org/
|
||||
Specific self-realization
|
||||
third solution,
|
||||
It is recommended to implement the hjsplit split merge function with go, which has cross-platform capabilities. (Unrealized, waiting for you to come....)
|
||||
Option 4
|
||||
Use the built-in relay function (using the protocol for resumable uploads protocol, [Details] (https://tus.io/))
|
||||
Note: Option 4, you can only specify one upload server, do not support simultaneous writes, and the uploaded url has changed.
|
||||
Original upload url: http://10.1.5.9:8080/<group>/upload
|
||||
Breakpoint upload url: http://10.1.5.9:8080/<group>/big/upload/
|
||||
The upload is completed, and then the second pass interface is used to obtain the file information.
|
||||
```
|
||||
|
||||
- 如何秒传文件？
|
||||
- How to pass files in seconds?
|
||||
```
|
||||
通过http get的方式访问上传接口
|
||||
http://10.0.5.9:8080/upload?md5=filesum&output=json
|
||||
参数说明：
|
||||
md5=sum(file) 文件的摘要算法要与文件务器的算法一致（算法支持md5|sha1），如果是断点续传，可以使用文件的id，也就是urlolad后的id
|
||||
output=json|text 返回的格式
|
||||
Access the upload interface by means of http get
|
||||
Http://10.0.5.9:8080/upload?md5=filesum&output=json
|
||||
Parameter Description:
|
||||
Md5=sum(file) The digest algorithm of the file should be consistent with the algorithm of the server (the algorithm supports md5|sha1). If it is a breakpoint, you can use the id of the file, which is the id after urlolad.
|
||||
Output=json|text The format returned
|
||||
|
||||
```
|
||||
|
||||
- 集群如何规划及如何进行扩容？
|
||||
- How is the cluster planned and how is it expanded?
|
||||
```
|
||||
建议在前期规划时，尽量采购大容量的机器作为存储服务器，如果要两个副本就用两台组成一个集群，如果要三个副本
|
||||
就三台组成一个集群。（注意每台服务器最好配置保持一样，并且使用raid5磁盘阵列）
|
||||
It is recommended that in the early planning, try to purchase a large-capacity machine as a storage server. If you want two copies, use two to form a cluster. If you want three copies.
|
||||
Let the three units form a cluster. (Note that the best configuration for each server remains the same and uses raid5 disk arrays)
|
||||
|
||||
如果提高可用性，只要在现在的集群peers中加入新的机器，再对集群进行修复即可。
|
||||
修复办法 http://172.16.70.123:7080/repair?force=1 （建议低峰变更）
|
||||
If you increase the availability, just add a new machine to the current cluster peers and fix the cluster.
|
||||
Repair method http://172.16.70.123:7080/repair?force=1 (recommended low peak change)
|
||||
|
||||
如何扩容？
|
||||
为简单可靠起见，直接搭建一个新集群即可（搭建就是启动./fileserver进程，设置一下peers的IP地址，三五分钟的事）
|
||||
issue中chengyuansen同学向我提议使用增加扩容特性，我觉得对代码逻辑及运维都增加复杂度，暂时没有加入这特性。
|
||||
How to expand?
|
||||
For the sake of simplicity and reliability, you can directly build a new cluster (build is to start the ./fileserver process, set the IP address of the peers, three or five minutes)
|
||||
In the issue, chengyuansen suggested to me to use the increased capacity expansion feature. I feel that the complexity of the code logic and operation and maintenance is increased. I have not added this feature for the time being.
|
||||
|
||||
```
|
||||
|
||||
|
||||
- 访问限制问题
|
||||
- Access restriction issues
|
||||
```
|
||||
出于安全考虑,管理API只能在群集内部调用或者用127.0.0.1调用.
|
||||
For security reasons, the management API can only be called inside the cluster or with 127.0.0.1.
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
- 有问题请[点击反馈](https://github.com/sjqzhang/go-fastdfs/issues/new)
|
||||
## 有问题请加群
|
||||
|
||||
- If you have any questions, please click [Reply] (https://github.com/sjqzhang/go-fastdfs/issues/new)
|
||||
## Have questions?
|
||||
|
||||
|
||||
#### 进群请改昵称，昵称格式：城市－公司－昵称，如果你喜欢这项目，请关注(star)此项目，关注是对项目的肯定，也是作者创新的动力。
|
||||
|
||||
#### [捐赠](doc/pay.png)
|
||||
#### [donation] (doc/pay.png)
|
||||
|
||||
588
README.md
588
README.md
@ -1,97 +1,100 @@
|
||||
# [中文](README-en.md) [English](README.md)
|
||||
|
||||
# [中文](README.md) [English](README-en.md)
|
||||
|
||||
|
||||
|
||||
|
||||
# go-fastdfs is a distributed file system based on http protocol. It is based on the design concept of avenue to simple. All the simple design makes its operation and expansion more simple. It has high performance, high reliability and no center. , maintenance-free and so on.
|
||||
# go-fastdfs是一个基于http协议的分布式文件系统，它基于大道至简的设计理念，一切从简设计，使得它的运维及扩展变得更加简单，它具有高性能、高可靠、无中心、免维护等优点。
|
||||
|
||||
### Everyone is worried about such a simple file system. Is it not reliable, can it be used in a production environment? The answer is yes, it is efficient because it is simple, and it is stable because it is simple. If you are worried about the function, then run the unit test, if you are worried about the performance, then run the stress test, the project comes with it, run more confident ^_^.
|
||||
### 大家担心的是这么简单的文件系统，靠不靠谱，可不可以用于生产环境？答案是肯定的，正因为简单所以高效，因为简单所以稳定。如果你担心功能，那就跑单元测试，如果担心性能，那就跑压力测试，项目都自带了，跑一跑更放心^_^。
|
||||
|
||||
Note: Please read this article carefully before using it, especially [wiki](https://github.com/sjqzhang/go-fastdfs/wiki)
|
||||
注意：使用前请认真阅读完本文，特别是[wiki](https://github.com/sjqzhang/go-fastdfs/wiki)
|
||||
|
||||
- Support curl command upload
|
||||
- Support browser upload
|
||||
- Support HTTP download
|
||||
- Support multi-machine automatic synchronization
|
||||
- Support breakpoint download
|
||||
- Support configuration automatic generation
|
||||
- Support small file automatic merge (reduce inode occupancy)
|
||||
- Support for second pass
|
||||
- Support for cross-domain access
|
||||
- Support one-click migration
|
||||
- Support for parallel experience
|
||||
- Support for breakpoint resuming ([tus](https://tus.io/))
|
||||
- Support for docker deployment
|
||||
- Support self-monitoring alarm
|
||||
- Support image zoom
|
||||
- Support google authentication code
|
||||
- Support for custom authentication
|
||||
- Support cluster file information viewing
|
||||
- Use the universal HTTP protocol
|
||||
- No need for a dedicated client (support wget, curl, etc.)
|
||||
- class fastdfs
|
||||
- High performance (using leveldb as a kv library)
|
||||
- High reliability (design is extremely simple, using mature components)
|
||||
- No center design (all nodes can read and write at the same time)
|
||||
- 支持curl命令上传
|
||||
- 支持浏览器上传
|
||||
- 支持HTTP下载
|
||||
- 支持多机自动同步
|
||||
- 支持断点下载
|
||||
- 支持配置自动生成
|
||||
- 支持小文件自动合并(减少inode占用)
|
||||
- 支持秒传
|
||||
- 支持跨域访问
|
||||
- 支持一键迁移
|
||||
- 支持并行体验
|
||||
- 支持断点续传([tus](https://tus.io/))
|
||||
- 支持docker部署
|
||||
- 支持自监控告警
|
||||
- 支持图片缩放
|
||||
- 支持google认证码
|
||||
- 支持自定义认证
|
||||
- 支持集群文件信息查看
|
||||
- 使用通用HTTP协议
|
||||
- 无需专用客户端（支持wget,curl等工具）
|
||||
- 类fastdfs
|
||||
- 高性能 （使用leveldb作为kv库）
|
||||
- 高可靠（设计极其简单，使用成熟组件）
|
||||
- 无中心设计(所有节点都可以同时读写)
|
||||
|
||||
# advantage
|
||||
# 优点
|
||||
|
||||
- No dependencies (single file)
|
||||
- Automatic synchronization
|
||||
- Failure automatic repair
|
||||
- Convenient maintenance by talent directory
|
||||
- Support different scenarios
|
||||
- Automatic file deduplication
|
||||
- Support for directory customization
|
||||
- Support to retain the original file name
|
||||
- Support for automatic generation of unique file names
|
||||
- Support browser upload
|
||||
- Support for viewing cluster file information
|
||||
- Support cluster monitoring email alarm
|
||||
- Support small file automatic merge (reduce inode occupancy)
|
||||
- Support for second pass
|
||||
- Support image zoom
|
||||
- Support google authentication code
|
||||
- Support for custom authentication
|
||||
- Support for cross-domain access
|
||||
- Very low resource overhead
|
||||
- Support for breakpoint resuming ([tus](https://tus.io/))
|
||||
- Support for docker deployment
|
||||
- Support for one-click migration (migrated from other system file systems)
|
||||
- Support for parallel experience (parallel experience with existing file system, confirm OK and then one-click migration)
|
||||
- Support token download token=md5(file_md5+timestamp)
|
||||
- Easy operation and maintenance, only one role (unlike fastdfs has three roles Tracker Server, Storage Server, Client), the configuration is automatically generated
|
||||
- Peer-to-peer (simplified operation and maintenance)
|
||||
- All nodes can read and write simultaneously
|
||||
- 无依赖(单一文件）
|
||||
- 自动同步
|
||||
- 失败自动修复
|
||||
- 按天分目录方便维护
|
||||
- 支持不同的场景
|
||||
- 文件自动去重
|
||||
- 支持目录自定义
|
||||
- 支持保留原文件名
|
||||
- 支持自动生成唯一文件名
|
||||
- 支持浏览器上传
|
||||
- 支持查看集群文件信息
|
||||
- 支持集群监控邮件告警
|
||||
- 支持小文件自动合并(减少inode占用)
|
||||
- 支持秒传
|
||||
- 支持图片缩放
|
||||
- 支持google认证码
|
||||
- 支持自定义认证
|
||||
- 支持跨域访问
|
||||
- 极低资源开销
|
||||
- 支持断点续传([tus](https://tus.io/))
|
||||
- 支持docker部署
|
||||
- 支持一键迁移（从其他系统文件系统迁移过来）
|
||||
- 支持并行体验（与现有的文件系统并行体验，确认OK再一键迁移）
|
||||
- 支持token下载 token=md5(file_md5+timestamp)
|
||||
- 运维简单，只有一个角色（不像fastdfs有三个角色Tracker Server,Storage Server,Client），配置自动生成
|
||||
- 每个节点对等（简化运维）
|
||||
- 所有节点都可以同时读写
|
||||
|
||||
|
||||
|
||||
#Start the server (compiled, [download](https://github.com/sjqzhang/fastdfs/releases) experience)
|
||||
```
|
||||
./fileserver
|
||||
# 启动服务器（已编译，[下载](https://github.com/sjqzhang/fastdfs/releases)极速体验，只需一分钟）
|
||||
一键安装：（请将以下命令复制到linux console中执行）
|
||||
```shell
|
||||
wget --no-check-certificate https://github.com/sjqzhang/go-fastdfs/releases/download/v1.2.6/fileserver -O fileserver && chmod +x fileserver && ./fileserver
|
||||
```
|
||||
(注意：下载时要注意链接的版本号，windows下直接运行fileserver.exe，执行文件在这里[下载](https://github.com/sjqzhang/fastdfs/releases))
|
||||
|
||||
|
||||
#Command upload
|
||||
# 命令上传
|
||||
|
||||
`curl -F file=@http-index-fs http://10.1.xx.60:8080/upload`
|
||||
|
||||
|
||||
# WEB upload (browser open)
|
||||
# WEB上传（浏览器打开）
|
||||
|
||||
`http://127.0.0.1:8080`
|
||||
`http://yourserver ip:8080/upload.html` 注意：不要使用127.0.0.1上传
|
||||
|
||||
#Code upload (options see browser upload)
|
||||
## python
|
||||
# 代码上传(选项参阅浏览器上传)
|
||||
|
||||
## python版本:
|
||||
```python
|
||||
import requests
|
||||
url = 'http://127.0.0.1:8080/upload'
|
||||
url = 'http://10.1.5.9:8080/upload'
|
||||
files = {'file': open('report.xls', 'rb')}
|
||||
options={'output':'json','path':'','scene':''} #See browser upload options
|
||||
r = requests.post(url, files=files)
|
||||
options={'output':'json','path':'','scene':''} #参阅浏览器上传的选项
|
||||
r = requests.post(url,data=options, files=files)
|
||||
print(r.text)
|
||||
```
|
||||
## golang
|
||||
## golang版本
|
||||
```go
|
||||
package main
|
||||
|
||||
@ -103,17 +106,16 @@ import (
|
||||
func main() {
|
||||
var obj interface{}
|
||||
req:=httplib.Post("http://10.1.5.9:8080/upload")
|
||||
req.PostFile("file","path/to/file")
|
||||
req.PostFile("file","filename")//注意不是全路径
|
||||
req.Param("output","json")
|
||||
req.Param("scene","")
|
||||
req.Param("path","")
|
||||
req.ToJSON(&obj)
|
||||
fmt.Print(obj)
|
||||
}
|
||||
````
|
||||
|
||||
## java
|
||||
|
||||
```
|
||||
## java版本
|
||||
依赖(这里使用了hutool工具包,更简便)
|
||||
```xml
|
||||
<dependency>
|
||||
<groupId>cn.hutool</groupId>
|
||||
@ -121,22 +123,32 @@ func main() {
|
||||
<version>4.4.3</version>
|
||||
</dependency>
|
||||
```
|
||||
|
||||
上传代码
|
||||
```java
|
||||
public static void main(String[] args) {
|
||||
//文件地址
|
||||
File file = new File("D:\\git\\2.jpg");
|
||||
//声明参数集合
|
||||
HashMap<String, Object> paramMap = new HashMap<>();
|
||||
//文件
|
||||
paramMap.put("file", file);
|
||||
//输出
|
||||
paramMap.put("output","json");
|
||||
//自定义路径
|
||||
paramMap.put("path","image");
|
||||
//场景
|
||||
paramMap.put("scene","image");
|
||||
//上传
|
||||
String result= HttpUtil.post("http://xxxxx:xxxx/upload", paramMap);
|
||||
//输出json结果
|
||||
System.out.println(result);
|
||||
}
|
||||
```
|
||||
[更多语言请参考](doc/upload.md)
|
||||
|
||||
# Breakpoint resume example
|
||||
## golang
|
||||
# 断点续传示例
|
||||
|
||||
## golang版本
|
||||
```go
|
||||
package main
|
||||
|
||||
@ -165,350 +177,364 @@ func main() {
|
||||
fmt.Println( uploader.Upload())
|
||||
|
||||
}
|
||||
|
||||
````
|
||||
[more langue](doc/upload.md)
|
||||
[更多客户端请参考](https://github.com/tus)
|
||||
|
||||
|
||||
|
||||
Universal file authentication timing diagram
|
||||
|
||||
|
||||
File google authentication timing diagram
|
||||
|
||||
部署图
|
||||
|
||||
|
||||
# Please click [Feedback](https://github.com/sjqzhang/go-fastdfs/issues/new)
|
||||
通用文件认证时序图
|
||||
|
||||
|
||||
文件google认证时序图
|
||||
|
||||
|
||||
# 有问题请[点击反馈](https://github.com/sjqzhang/go-fastdfs/issues/new)
|
||||
|
||||
# 重要说明
|
||||
## 在issue中有很多实际使用的问题及回答（很多已关闭，请查看已关闭的issue）
|
||||
|
||||
## 项目从v1.1.8开始进入稳定状态
|
||||
|
||||
# 更新说明
|
||||
## 从低版升给到高版本，可能存在配置项变动的情况，一定要注意使用新的版本时的配置项。如何获得新版本的配置项及说明？先备份旧的配置项（文件名不能为cfg.json），再运行新的版本，配置项就自动生成。然后再修改相应的配置项。
|
||||
|
||||
- v1.1.9 增加文件自动迁移功能，支持同名文件重复覆盖选项。
|
||||
|
||||
# <span id="qa">Q&A</span>
|
||||
|
||||
- Best practice?
|
||||
- 最佳实战？
|
||||
```
|
||||
First, if it is mass storage, do not open the file token authentication function to reduce performance.
|
||||
Second, try to use the standard upload, upload the business to save the path, and then connect the domain name when the business is used (convenient migration extension, etc.).
|
||||
Third, if you use breakpoints to continue transmission, you must use the file id to replace the path storage after uploading (how to replace the QA/API document), to reduce performance for subsequent access.
|
||||
Fourth, try to use physical server deployment, because the main pressure or performance comes from IO
|
||||
Fifth, the online business should use the nginx+gofastdfs deployment architecture (the equalization algorithm uses ip_hash) to meet the later functional scalability (nginx+lua).
|
||||
Sixth, the online environment is best not to use container deployment, the container is suitable for testing and functional verification.
|
||||
Summary: The path of the file saved by the business reduces the conversion of the later access path, and the file access permission is completed by the service, so that the performance is the best and the versatility is strong (can be directly connected to other web servers).
|
||||
一、如果是海量存储，不要开启文件token认证功能，减少性能开消。
|
||||
二、尽量用标准上传，上传后业务保存path，在业务用的时候再并接上域名（方便迁移扩展等）。
|
||||
三、如果使用断点续传，上传后一定要用文件id置换成path存储（如何置换看ＱＡ/API文档），为后面访问减少性能开消。
|
||||
四、尽量使用物理服务器部署，因为主要压力或性能来自于IO
|
||||
五、线上业务尽量使用nginx+gofastdfs部署架构(均衡算法使用ip_hash)，以满足后面的功能扩展性(nginx+lua)。
|
||||
六、线上环境最好不要使用容器部署，容器适用于测试和功能验证。
|
||||
总结：业务保存的文件的path,减少后期访问路径转换带来开消,文件访问权限由业务来完成，这样性能最好，通用性强（可直接其它web服务器）。
|
||||
|
||||
Important reminder: If the small file merge function is enabled, it is impossible to delete small files later.
|
||||
Upload result description
|
||||
Please use md5, path, scene field, others are added to be compatible with the old online system, and may be removed in the future.
|
||||
重要提醒：如果开启小文件合并功能，后期是无法删除小文件的。
|
||||
上传结果说明
|
||||
请使用md5,path,scene字段，其它是为了兼容老的线上系统添加的，以后有可能去掉。
|
||||
|
||||
```
|
||||
|
||||
- Is there an API document?
|
||||
[API documentation] (doc/api.md)
|
||||
- 有API文档么？
|
||||
[API文档](doc/api.md)
|
||||
|
||||
- Is there a management background?
|
||||
- 有管理后台么？
|
||||
```
|
||||
Https://github.com/perfree/go-fastdfs-web
|
||||
https://github.com/perfree/go-fastdfs-web
|
||||
```
|
||||
- Is there any instructions for using the breakpoint upload?
|
||||
- 断点上传有使用说明么？
|
||||
```
|
||||
Https://github.com/tus
|
||||
https://github.com/tus
|
||||
```
|
||||
|
||||
- In the WeChat discussion group, everyone asked about the performance of go-fastdfs?
|
||||
- 在微信讨论群中大家都问到go-fastdfs性能怎样？
|
||||
```
|
||||
Because there are too many people asking, answer here in unison.
|
||||
The file location of go-fastdfs is different from other distributed systems. Its addressing is directly located without any components, so the approximate time complexity is o(1)[file path location]
|
||||
There is basically no performance loss. The project also has a pressure test script. You can carry out the pressure test yourself. Don’t discuss the problem too much in the group. People reply to the same question every time.
|
||||
Everyone will also feel that this group is boring.
|
||||
由于问的人太多，在这里统一回答。
|
||||
go-fastdfs的文件定位与其它分布式系统不同，它的寻址是直接定位，不经过任何组件，所以可以近似时间复杂度为o(1)[文件路径定位]
|
||||
基本没有性能损耗，项目中也附有压测脚本，大家可以自已进行压测，群里就不要太多讨论问题了，人多每次回复同样的问题
|
||||
大家也会觉得这群无聊。
|
||||
```
|
||||
|
||||
|
||||
|
||||
- Files that have been stored using fastdfs can be migrated to go fastdfs (other migrations can be handled in a similar manner, and the speed experience is similar)?
|
||||
- 已经使用fastdfs存储的文件可以迁移到go fastdfs下么（其它迁移情况可按类似的方式处理，极速体验也用类似的方式）？
|
||||
```
|
||||
The answer is yes, the problem you are worried about is the path change, go fastdfs considers this for you.
|
||||
step:
|
||||
First, download the latest version of go-fastdfs
|
||||
Second, copy the original fastdfs file directory to the files directory of go-fastdfs (if there are a lot of files, you can reverse it, copy the fileserver, but keep the fileserver directory structure)
|
||||
Third, set the enable_migrate to true
|
||||
Note: All files in the files directory will be scanned during the migration process.
|
||||
Slower, set enable_migrate to false after migration is complete
|
||||
答案是可以的，你担心的问题是路径改变,go fastdfs为你考虑了这一点
|
||||
步骤：
|
||||
一、下载最新版的go-fastdfs
|
||||
二、将原来的fastdfs文件目录复制到go-fastdfs的 files目录下(如果文件很多，可以逆向过来，将fileserver复制过去，但要保留fileserver的目录结构）
|
||||
三、将配置enable_migrate设为true
|
||||
注意：迁移过程中会扫描整下files目录下的所有文件，
|
||||
速度较慢，迁移完成后请将enable_migrate设为false
|
||||
|
||||
Note: The directory of go-fastdfs can not be changed, related to the synchronization mechanism, many students in the group, my files directory can not be customized, the answer is no.
|
||||
As for whether or not I can use the soft link, I have not tested it and can test it myself.
|
||||
说明：go-fastdfs的目录是不能变动的，与同步机制相关，很多同学在群里问题，我的files目录能不能自定义，答案是否定的。
|
||||
至于能不能用软链接的方式本人没有测试过，可以自行测试。
|
||||
|
||||
```
|
||||
|
||||
- What is a cluster, how to manage multiple clusters with Nginx?
|
||||
- 什么是集群，如何用Nginx管理多集群？
|
||||
```
|
||||
1. In go-fastdfs, a cluster is a group.
|
||||
2, please refer to the deployment diagram
|
||||
Note: When the support_group_manage parameter in the configuration is set to true, group information is automatically added to all urls.
|
||||
For example: http://10.1.5.9:8080/group/status
|
||||
Default: http://10.1.5.9:8080/status
|
||||
The difference: more group, corresponding to the group parameter in the configuration, so mainly to solve a Nginx reverse proxy multiple groups (cluster)
|
||||
Please refer to the deployment diagram for details.
|
||||
1、在go-fastdfs中，一个集群就是一个group。
|
||||
2、请参阅部署图
|
||||
注意：配置中的 support_group_manage 参数设为true时，所有的url中都自动添加组信息。
|
||||
例如：http://10.1.5.9:8080/group/status
|
||||
默认：http://10.1.5.9:8080/status
|
||||
区别：多了group,对应配置中的 group 参数,这样主要是为了解决一个Nginx反向代理多个group(集群)
|
||||
具体请参阅部署图
|
||||
|
||||
```
|
||||
|
||||
|
||||
- How to build a cluster?
|
||||
- 如何搭建集群？
|
||||
```
|
||||
First, download the compiled executable file (with the latest version)
|
||||
Second, run the executable file (generate configuration)
|
||||
Third, modify the configuration
|
||||
Peers: increase the peer's http address
|
||||
an examination:
|
||||
Host: Is the automatic generation correct?
|
||||
Peer_id: Is it unique within the cluster?
|
||||
Fourth, re-run the server
|
||||
5. Is the verification service OK?
|
||||
一、先下载已编译的可执行文件（用最新版本）
|
||||
二、运行可执行文件（生成配置）
|
||||
三、修改配置
|
||||
peers：增加对端的http地址
|
||||
检查:
|
||||
host:自动生成是否正确
|
||||
peer_id:集群内是否唯一
|
||||
四、重新运行服器
|
||||
五、验证服务是否OK
|
||||
```
|
||||
|
||||
|
||||
- Is it suitable for mass storage?
|
||||
- 适合海量存储吗？
|
||||
```
|
||||
Answer: Suitable for mass storage
|
||||
答案：适合海量存储
|
||||
```
|
||||
|
||||
- How to upload a folder?
|
||||
- 如何上传文件夹？
|
||||
```
|
||||
DIR=log && ls $DIR |xargs -n 1 -I {} curl -s -F path=$DIR -F file=@$DIR/{} http://10.1.50.90:8080/upload
|
||||
The log of the above command is the directory to be uploaded. If the code upload is a simple loop upload, it will be OK.
|
||||
上面命令的log为要上传的目录，如果代码上传就是简单的循环上传就ＯＫ。
|
||||
```
|
||||
|
||||
- How to scale the picture?
|
||||
- 如何缩放图片？
|
||||
```
|
||||
Add width parameter in the download url
|
||||
For example: http://127.0.0.1:8080/group1/haystack/5/124,0,27344,.jpg?download=0&width=100&height=100
|
||||
The special explanation is: if you want to scale, please set the width or height to 0.
|
||||
在下载url中加入width各height参数
|
||||
例如：http://127.0.0.1:8080/group1/haystack/5/124,0,27344,.jpg?download=0&width=100&height=100
|
||||
特明说明是：如果要等比例缩放，请将width或height设为０
|
||||
```
|
||||
|
||||
- How to display images directly in the browser?
|
||||
- 如何在浏览器中直接显示图片？
|
||||
```
|
||||
Add the download=0 parameter to the download url.
|
||||
For example: http://127.0.0.1:8080/group1/haystack/5/124,0,27344,.jpg?download=0
|
||||
在下载url中加入download=0参数
|
||||
例如：http://127.0.0.1:8080/group1/haystack/5/124,0,27344,.jpg?download=0
|
||||
```
|
||||
|
||||
|
||||
- How to implement custom authentication upload and download?
|
||||
- 如何实现自定义认证上传下载？
|
||||
```
|
||||
First, the use of the 1.2.6 version of the go-fastdfs
|
||||
Second, set the auth_url parameter (provided by the application)
|
||||
Third, the application implements the authentication permission interface (that is, the url of the second step), the parameter is auth_toke, ok, the authentication is passed, and the others are not passed.
|
||||
Fourth, after the certification is passed, you can upload or download
|
||||
一、使用1.2.6版本以后的go-fastdfs
|
||||
二、设auth_url参数（应用提供）
|
||||
三、应用实现验证权限接口（即第二步的url）,参数为 auth_toke 返回 ok 表示认证通过，其它为不通过
|
||||
四、认证通过后，可以上传或下载
|
||||
```
|
||||
|
||||
|
||||
- Need to install nginx yet?
|
||||
- 还需要安装nginx么？
|
||||
```
|
||||
Go-fastdfs itself is a high-performance web server, you don't need to install nginx when developing or testing.
|
||||
But go-fastdfs has a single function, such as caching or redirection or other extensions, nginx has mature components
|
||||
Therefore, it is recommended to add a layer of nginx on the line, and then use nginx+lua to solve the scalability problem.
|
||||
go-fastdfs本身是一个高性能的web服务器，在开发或测试时，可以不用安装nginx，
|
||||
但go-fastdfs的功能单一，如需要缓存或重定向或其它扩展，nginx都有成熟的组件
|
||||
所以建议线上还是加一层nginx，再借助nginx+lua解决扩展性问题。
|
||||
```
|
||||
|
||||
- Can I dynamically load the configuration?
|
||||
- 能动态加载配置么？
|
||||
```
|
||||
Answer: Yes, but update to the latest version
|
||||
step:
|
||||
1) Modify the conf/cfg.json file
|
||||
2) Visit http://10.1.xx.60:8080/reload
|
||||
3) Note: Every node needs to do the same thing
|
||||
答案：是可以的，但要更新到最新版本
|
||||
步骤：
|
||||
1）修改 conf/cfg.json 文件
|
||||
2）访问 http://10.1.xx.60:8080/reload
|
||||
3） 注意：每个节点都需要进行同样的操作
|
||||
```
|
||||
|
||||
|
||||
- How to view cluster file information?
|
||||
- 如何查看集群文件信息？
|
||||
```
|
||||
Http://10.1.xx.60:8080/stat
|
||||
http://10.1.xx.60:8080/stat
|
||||
|
||||
What should I do if there is a file error?
|
||||
Please delete the stat.json file in the data directory. Restart the service, please recalculate the number of files automatically.
|
||||
如果出现文件统计出错怎么办？
|
||||
请删除 data目录下的 stat.json文件 重启服务，请系统自动重新计算文件数。
|
||||
|
||||
Or call
|
||||
Http://10.1.xx.60:8080/repair_stat
|
||||
或者调用
|
||||
http://10.1.xx.60:8080/repair_stat
|
||||
|
||||
```
|
||||
- How reliable can it be used in a production environment?
|
||||
- 可靠性怎样，能用于生产环境么？
|
||||
```
|
||||
This project has been used on a large scale in the production environment, such as fear of not meeting
|
||||
You can stress test its features before use, any
|
||||
The problem can be directly mentioned
|
||||
本项目已大规模用于生产环境，如担心不能满足
|
||||
可以在使用前对其各项特性进行压力测试，有任何
|
||||
问题可以直接提issue
|
||||
```
|
||||
|
||||
- How to run the program in the background?
|
||||
- 如何后台运行程序？
|
||||
```
|
||||
Please use control to run the program later, as follows:
|
||||
First, chmod +x control
|
||||
Second, make sure the control and fileserver are in the same directory
|
||||
Third, ./control start|stop|status #Start and start the sequence, stop, view the status, and so on.
|
||||
请使用control 对程序进行后面运行，具体操作如下:
|
||||
一、 chmod +x control
|
||||
二、 确保control与fileserver在同一个目录
|
||||
三、 ./control start|stop|status #对和序进行启动，停止，查看状态等。
|
||||
|
||||
```
|
||||
|
||||
|
||||
- Can I have multiple servers on one machine?
|
||||
- 能不能在一台机器部置多个服务端？
|
||||
```
|
||||
No, the high availability of the cluster has been considered at the beginning of the design. In order to ensure the true availability of the cluster, it must be different for ip, ip can not use 127.0.0.1
|
||||
Error "peers": ["http://127.0.0.1:8080","http://127.0.0.1:8081","http://127.0.0.1:8082"]
|
||||
Correct "peers": ["http://10.0.0.3:8080","http://10.0.0.4:8080","http://10.0.0.5:8080"]
|
||||
不能，在设计之初就已考虑到集群的高可用问题，为了保证集群的真正可用，必须为不同的ip,ip 不能用 127.0.0.1
|
||||
错误 "peers": ["http://127.0.0.1:8080","http://127.0.0.1:8081","http://127.0.0.1:8082"]
|
||||
正确 "peers": ["http://10.0.0.3:8080","http://10.0.0.4:8080","http://10.0.0.5:8080"]
|
||||
```
|
||||
- What should I do if the files are not synchronized?
|
||||
- 文件不同步了怎么办？
|
||||
```
|
||||
Under normal circumstances, the cluster automatically synchronizes the repair files every hour. (The performance is poor, it is recommended to turn off automatic repair in case of massive)
|
||||
What about the abnormal situation?
|
||||
Answer: Manual synchronization (preferably at low peaks)
|
||||
Http://172.16.70.123:7080/sync?date=20190117&force=1 (Note: To be executed on a server with many files, related to pushing to another server)
|
||||
Parameter description: date indicates the data of the day of synchronization. force 1. indicates whether to force synchronization of all the day (poor performance), 0. means that only failed files are synchronized.
|
||||
正常情况下，集群会每小时自动同步修复文件。（性能较差，在海量情况下建议关闭自动修复）
|
||||
那异常情况下怎么？
|
||||
答案：手动同步（最好在低峰执行）
|
||||
http://172.16.70.123:7080/sync?date=20190117&force=1 (说明：要在文件多的服务器上执行，相关于推送到别的服务器)
|
||||
参数说明：date 表示同步那一天的数据 force 1.表示是否强制同步当天所有(性能差)，0.表示只同步失败的文件
|
||||
|
||||
Unsynchronized situation:
|
||||
1) Originally running N sets, now suddenly join one to become N+1
|
||||
2) Originally running N sets, one machine has a problem and becomes N-1
|
||||
不同步的情况：
|
||||
1) 原来运行N台，现在突然加入一台变成N+1台
|
||||
2）原来运行N台，某一台机器出现问题，变成N-1台
|
||||
|
||||
What if there is a multi-day data inconsistency? Can I synchronize all at once?
|
||||
The answer is yes: (preferably at low peaks)
|
||||
Http://172.16.70.123:7080/repair?force=1
|
||||
如果出现多天数据不一致怎么办？能一次同步所有吗？
|
||||
答案是可以：(最好在低峰执行)
|
||||
http://172.16.70.123:7080/repair?force=1
|
||||
|
||||
```
|
||||
|
||||
- Does the file out of sync affect access?
|
||||
- 文件不同步会影响访问吗？
|
||||
```
|
||||
Answer: It will not affect, it will automatically repair the files that are not synchronized when the access is not available.
|
||||
答案：不会影响，会在访问不到时，自动修复不同步的文件。
|
||||
```
|
||||
|
||||
- How do I check the system status and description?
|
||||
- 如何查看系统状态及说明？
|
||||
```
|
||||
Http://172.16.70.123:7080/status
|
||||
Note: (Fs.Peers is not equipped with this unit, if there is a problem with the problem)
|
||||
This machine is Fs.Local
|
||||
Sts["Fs.ErrorSetSize"] = this.errorset.Cardinality() This will cause memory to increase
|
||||
http://172.16.70.123:7080/status
|
||||
注意:（Fs.Peers是不带本机的，如果带有可能出问题）
|
||||
本机为 Fs.Local
|
||||
sts["Fs.ErrorSetSize"] = this.errorset.Cardinality() 这个会导致内存增加
|
||||
|
||||
```
|
||||
|
||||
|
||||
- How to compile (go1.9.2+)?
|
||||
- 如何编译(go1.9.2+)？
|
||||
```
|
||||
Git clone https://github.com/sjqzhang/go-fastdfs.git
|
||||
Cd go-fastdfs
|
||||
Mv vendor src
|
||||
Pwd=`pwd`
|
||||
git clone https://github.com/sjqzhang/go-fastdfs.git
|
||||
cd go-fastdfs
|
||||
mv vendor src
|
||||
pwd=`pwd`
|
||||
GOPATH=$pwd go build -o fileserver fileserver.go
|
||||
```
|
||||
|
||||
- How to run a unit test (try to do it under linux)?
|
||||
- 如何跑单元测试 (尽量在linux下进行)？
|
||||
```
|
||||
|
||||
Git clone https://github.com/sjqzhang/go-fastdfs.git
|
||||
Cd go-fastdfs
|
||||
Mv vendor src
|
||||
Pwd=`pwd`
|
||||
git clone https://github.com/sjqzhang/go-fastdfs.git
|
||||
cd go-fastdfs
|
||||
mv vendor src
|
||||
pwd=`pwd`
|
||||
GOPATH=$pwd go test -v fileserver.go fileserver_test.go
|
||||
|
||||
```
|
||||
|
||||
|
||||
|
||||
- How to test?
|
||||
- 如何压测？
|
||||
```
|
||||
step:
|
||||
First, create a files folder
|
||||
Second, copy gen_file.py to the files folder, generate a large number of files through python gen_file.py
|
||||
Third, put benchmark.py outside the files directory (that is, the same level as the files directory), press the python benchmark.py (note the ip in the benchmark.py)
|
||||
First use gen_file.py to generate a large number of files (note that if you want to generate large files, you can multiply the content by a large number)
|
||||
E.g:
|
||||
步骤：
|
||||
一、创建files文件夹
|
||||
二、将gen_file.py复制到files文件夹中，通过python gen_file.py 生成大量文件
|
||||
三、将benchmark.py放到 files目录外（即与files目录同一级），通过python benchmark.py进行压测（注意对benchmark.py中的ip进行修改）
|
||||
先用gen_file.py产生大量文件（注意如果要生成大文件，自已在内容中乘上一个大的数即可）
|
||||
例如:
|
||||
# -*- coding: utf-8 -*-
|
||||
Import os
|
||||
import os
|
||||
j=0
|
||||
For i in range(0,1000000):
|
||||
If i%1000==0:
|
||||
for i in range(0,1000000):
|
||||
if i%1000==0:
|
||||
j=i
|
||||
Os.system('mkdir %s'%(i))
|
||||
With open('%s/%s.txt'%(j,i),'w+') as f:
|
||||
F.write(str(i)*1024)
|
||||
Then use benchmark.py for pressure measurement
|
||||
It is also possible to perform pressure measurement simultaneously in multiple machines, and all nodes can be read and written simultaneously.
|
||||
os.system('mkdir %s'%(i))
|
||||
with open('%s/%s.txt'%(j,i),'w+') as f:
|
||||
f.write(str(i)*1024)
|
||||
接着用benchmark.py进行压测
|
||||
也可以多机同时进行压测，所有节点都是可以同时读写的
|
||||
```
|
||||
|
||||
- Why is the code written in a file?
|
||||
- 代码为什么写在一个文件中？
|
||||
```
|
||||
First, the current code is still very simple, no need to make it too complicated.
|
||||
Second, the individual understands that modularity is not modular when multiple files are separated. You can use the IDE to look at the code structure, which is actually modular.
|
||||
一、目前的代码还非常简单，没必要弄得太复杂。
|
||||
二、个人理解模块化不是分开多个文件就表示模块化，大家可以用IDE去看一下代码结构，其实是已经模块化的。
|
||||
```
|
||||
|
||||
- Support for breakpoint downloads?
|
||||
- 支持断点下载？
|
||||
```
|
||||
Answer: Support
|
||||
Curl wget how
|
||||
Wget -c http://10.1.5.9:8080/group1/default/20190128/16/10/2G
|
||||
Culr -C - http://10.1.5.9:8080/group1/default/20190128/16/10/2G
|
||||
答案：支持
|
||||
curl wget 如何
|
||||
wget -c http://10.1.5.9:8080/group1/default/20190128/16/10/2G
|
||||
culr -C - http://10.1.5.9:8080/group1/default/20190128/16/10/2G
|
||||
```
|
||||
|
||||
- How is Docker deployed?
|
||||
- Docker如何部署？
|
||||
```
|
||||
step:
|
||||
method one,
|
||||
First, build a mirror
|
||||
Docker build . -t fastdfs
|
||||
Second, run the container (use the environment variable GO_FASTDFS_DIR to point to the storage directory.)
|
||||
Docker run --name fastdfs -v /data/fastdfs_data:/data -e GO_FASTDFS_DIR=/data fastdfs
|
||||
Method 2,
|
||||
First, pull the mirror
|
||||
Docker pull sjqzhang/go-fastdfs
|
||||
Second, run the container
|
||||
Docker run --name fastdfs -v /data/fastdfs_data:/data -e GO_FASTDFS_DIR=/data fastdfs
|
||||
步骤：
|
||||
方式一、
|
||||
一、构建镜像
|
||||
docker build . -t fastdfs
|
||||
二、运行容器（使用环境变量 GO_FASTDFS_DIR 指向存储目录。）
|
||||
docker run --name fastdfs -v /data/fastdfs_data:/data -e GO_FASTDFS_DIR=/data fastdfs
|
||||
方式二、
|
||||
一、拉取镜像
|
||||
docker pull sjqzhang/go-fastdfs
|
||||
二、运行容器
|
||||
docker run --name fastdfs -v /data/fastdfs_data:/data -e GO_FASTDFS_DIR=/data fastdfs
|
||||
|
||||
```
|
||||
|
||||
- How to upload large files or breakpoints?
|
||||
- 大文件如何分块上传或断点续传？
|
||||
```
|
||||
The general block uploading must be supported by the client, and the diversity of the language is difficult for the client to maintain, but the function of the block uploading is necessary, and a simple implementation idea is provided for this.
|
||||
Option One,
|
||||
Split and merge with linux split cat, see the split and cat help.
|
||||
Split: split -b 1M filename #1M per text
|
||||
Merge: cat x* > filename #merge
|
||||
Option II,
|
||||
With hjsplit
|
||||
Http://www.hjsplit.org/
|
||||
Specific self-realization
|
||||
third solution,
|
||||
It is recommended to implement the hjsplit split merge function with go, which has cross-platform capabilities. (Unrealized, waiting for you to come....)
|
||||
Option 4
|
||||
Use the built-in relay function (using the protocol for resumable uploads protocol, [Details] (https://tus.io/))
|
||||
Note: Option 4, you can only specify one upload server, do not support simultaneous writes, and the uploaded url has changed.
|
||||
Original upload url: http://10.1.5.9:8080/<group>/upload
|
||||
Breakpoint upload url: http://10.1.5.9:8080/<group>/big/upload/
|
||||
The upload is completed, and then the second pass interface is used to obtain the file information.
|
||||
一般的分块上传都要客户端支持，而语言的多样性，客户端难以维护，但分块上传的功能又有必要，为此提供一个简单的实现思路。
|
||||
方案一、
|
||||
借助linux split cat 实现分割与合并，具体查看 split 与 cat 帮助。
|
||||
分割： split -b 1M filename #按每个文1M
|
||||
合并： cat x* > filename #合并
|
||||
方案二、
|
||||
借助hjsplit
|
||||
http://www.hjsplit.org/
|
||||
具体自行实现
|
||||
方案三、
|
||||
建议用go实现hjsplit分割合并功，这样具有跨平台功能。（未实现，等你来....）
|
||||
方案四、
|
||||
使用内置的继点续传功能（使用protocol for resumable uploads协议，[详情](https://tus.io/)）
|
||||
注意：方案四、只能指定一个上传服务器，不支持同时写，并且上传的url有变化
|
||||
原上传url： http://10.1.5.9:8080/<group>/upload
|
||||
断点上传url： http://10.1.5.9:8080/<group>/big/upload/
|
||||
上传完成，再通过秒传接口，获取文件信息
|
||||
```
|
||||
|
||||
- How to pass files in seconds?
|
||||
- 如何秒传文件？
|
||||
```
|
||||
Access the upload interface by means of http get
|
||||
Http://10.0.5.9:8080/upload?md5=filesum&output=json
|
||||
Parameter Description:
|
||||
Md5=sum(file) The digest algorithm of the file should be consistent with the algorithm of the server (the algorithm supports md5|sha1). If it is a breakpoint, you can use the id of the file, which is the id after urlolad.
|
||||
Output=json|text The format returned
|
||||
通过http get的方式访问上传接口
|
||||
http://10.0.5.9:8080/upload?md5=filesum&output=json
|
||||
参数说明：
|
||||
md5=sum(file) 文件的摘要算法要与文件务器的算法一致（算法支持md5|sha1），如果是断点续传，可以使用文件的id，也就是urlolad后的id
|
||||
output=json|text 返回的格式
|
||||
|
||||
```
|
||||
|
||||
- How is the cluster planned and how is it expanded?
|
||||
- 集群如何规划及如何进行扩容？
|
||||
```
|
||||
It is recommended that in the early planning, try to purchase a large-capacity machine as a storage server. If you want two copies, use two to form a cluster. If you want three copies.
|
||||
Let the three units form a cluster. (Note that the best configuration for each server remains the same and uses raid5 disk arrays)
|
||||
建议在前期规划时，尽量采购大容量的机器作为存储服务器，如果要两个副本就用两台组成一个集群，如果要三个副本
|
||||
就三台组成一个集群。（注意每台服务器最好配置保持一样，并且使用raid5磁盘阵列）
|
||||
|
||||
If you increase the availability, just add a new machine to the current cluster peers and fix the cluster.
|
||||
Repair method http://172.16.70.123:7080/repair?force=1 (recommended low peak change)
|
||||
如果提高可用性，只要在现在的集群peers中加入新的机器，再对集群进行修复即可。
|
||||
修复办法 http://172.16.70.123:7080/repair?force=1 （建议低峰变更）
|
||||
|
||||
How to expand?
|
||||
For the sake of simplicity and reliability, you can directly build a new cluster (build is to start the ./fileserver process, set the IP address of the peers, three or five minutes)
|
||||
In the issue, chengyuansen suggested to me to use the increased capacity expansion feature. I feel that the complexity of the code logic and operation and maintenance is increased. I have not added this feature for the time being.
|
||||
如何扩容？
|
||||
为简单可靠起见，直接搭建一个新集群即可（搭建就是启动./fileserver进程，设置一下peers的IP地址，三五分钟的事）
|
||||
issue中chengyuansen同学向我提议使用增加扩容特性，我觉得对代码逻辑及运维都增加复杂度，暂时没有加入这特性。
|
||||
|
||||
```
|
||||
|
||||
|
||||
- Access restriction issues
|
||||
- 访问限制问题
|
||||
```
|
||||
For security reasons, the management API can only be called inside the cluster or with 127.0.0.1.
|
||||
出于安全考虑,管理API只能在群集内部调用或者用127.0.0.1调用.
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
- If you have any questions, please click [Reply] (https://github.com/sjqzhang/go-fastdfs/issues/new)
|
||||
## Have questions?
|
||||
|
||||
- 有问题请[点击反馈](https://github.com/sjqzhang/go-fastdfs/issues/new)
|
||||
## 有问题请加群
|
||||
|
||||
|
||||
#### 进群请改昵称，昵称格式：城市－公司－昵称，如果你喜欢这项目，请关注(star)此项目，关注是对项目的肯定，也是作者创新的动力。
|
||||
|
||||
#### [donation] (doc/pay.png)
|
||||
#### [捐赠](doc/pay.png)
|
||||
|
||||
Loading…
Reference in New Issue
Block a user