mirror of
https://gitee.com/dolphinscheduler/DolphinScheduler.git
synced 2024-12-02 04:08:31 +08:00
[Fix][Docker] Support Multi-Architecture Docker Images (#10052)
Co-authored-by: kezhenxu94 <kezhenxu94@apache.org>
This commit is contained in:
旺阳
GitHub
parent
cd70ad584a
commit
537c206c2d
4
.github/workflows/publish-docker.yaml
vendored
4
.github/workflows/publish-docker.yaml
vendored
@ -48,9 +48,9 @@ jobs:
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
- name: Set up QEMU
|
||||
uses: docker/setup-qemu-action@v1
|
||||
uses: docker/setup-qemu-action@v2
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@v1
|
||||
uses: docker/setup-buildx-action@v2
|
||||
- name: Build and push docker images
|
||||
run: |
|
||||
./mvnw -B clean deploy \
|
||||
|
||||
@ -23,6 +23,58 @@ git clone git@github.com:apache/dolphinscheduler.git
|
||||
|
||||
ii. Run `mvn clean install -Prelease -Dmaven.test.skip=true`
|
||||
|
||||
## Docker image build
|
||||
|
||||
DolphinScheduler will release new Docker images after it released, you could find them in [Docker Hub](https://hub.docker.com/search?q=DolphinScheduler).
|
||||
|
||||
* If you want to modify DolphinScheduler source code, and build Docker images locally, you can run when finished the modification
|
||||
```shell
|
||||
cd dolphinscheduler
|
||||
./mvnw -B clean package \
|
||||
-Dmaven.test.skip \
|
||||
-Dmaven.javadoc.skip \
|
||||
-Dmaven.checkstyle.skip \
|
||||
-Ddocker.tag=<TAG> \
|
||||
-Pdocker,release
|
||||
```
|
||||
|
||||
When the command is finished you could find them by command `docker imaegs`.
|
||||
|
||||
* If you want to modify DolphinScheduler source code, build and push Docker images to your registry <HUB_URL>，you can run when finished the modification
|
||||
```shell
|
||||
cd dolphinscheduler
|
||||
./mvnw -B clean deploy \
|
||||
-Dmaven.test.skip \
|
||||
-Dmaven.javadoc.skip \
|
||||
-Dmaven.checkstyle.skip \
|
||||
-Dmaven.deploy.skip \
|
||||
-Ddocker.tag=<TAG> \
|
||||
-Ddocker.hub=<HUB_URL> \
|
||||
-Pdocker,release
|
||||
```
|
||||
|
||||
* If you want to modify DolphinScheduler source code, and also want to add customize dependencies of Docker image, you can modify the definition of Dockerfile after modifying the source code. You can run the following command to find all Dockerfile files.
|
||||
|
||||
```shell
|
||||
cd dolphinscheduler
|
||||
find . -iname 'Dockerfile'
|
||||
```
|
||||
|
||||
Then run the Docker build command above
|
||||
|
||||
* You could create custom Docker images base on those images if you want to change image like add some dependencies or upgrade package.
|
||||
|
||||
```Dockerfile
|
||||
FROM dolphinscheduler-standalone-server
|
||||
RUN apt update ; \
|
||||
apt install -y <YOUR-CUSTOM-DEPENDENCE> ; \
|
||||
```
|
||||
|
||||
> **_Note：_** Docker will build and push linux/amd64,linux/arm64 multi-architecture images by default
|
||||
>
|
||||
> Have to use version after Docker 19.03, because after 19.03 docker contains buildx
|
||||
|
||||
|
||||
## Notice
|
||||
|
||||
There are two ways to configure the DolphinScheduler development environment, standalone mode and normal mode
|
||||
@ -33,7 +85,9 @@ There are two ways to configure the DolphinScheduler development environment, st
|
||||
## DolphinScheduler Standalone Quick Start
|
||||
|
||||
> **_Note:_** Use standalone server only for development and debugging, because it uses H2 Database as default database and Zookeeper Testing Server which may not be stable in production.
|
||||
>
|
||||
> Standalone is only supported in DolphinScheduler 1.3.9 and later versions.
|
||||
>
|
||||
> Standalone server is able to connect to external databases like mysql and postgresql, see [Standalone Deployment](https://dolphinscheduler.apache.org/en-us/docs/dev/user_doc/guide/installation/standalone.html) for instructions.
|
||||
|
||||
### Git Branch Choose
|
||||
|
||||
@ -700,47 +700,14 @@ A：The repair can be completed by executing the following SQL in the database:
|
||||
update t_ds_version set version='2.0.1';
|
||||
```
|
||||
|
||||
## Can not find python-gateway-server in distribute package
|
||||
---
|
||||
|
||||
After version 3.0.0-alpha, Python gateway server integrate into API server, and Python gateway service will start when you
|
||||
## Q：Can not find python-gateway-server in distribute package
|
||||
|
||||
A：After version 3.0.0-alpha, Python gateway server integrate into API server, and Python gateway service will start when you
|
||||
start API server. If you want disabled when Python gateway service you could change API server configuration in path
|
||||
`api-server/conf/application.yaml` and change attribute `python-gateway.enabled : false`.
|
||||
|
||||
## How to Build Custom Docker Image
|
||||
|
||||
DolphinScheduler will release new Docker images after it released, you could find them in DockerHub. You could create
|
||||
custom Docker images base on those images if you want to change image like add some dependencies or upgrade package.
|
||||
|
||||
```Dockerfile
|
||||
FROM apache/dolphinscheduler-standalone-server
|
||||
RUN apt update ; \
|
||||
apt install -y <YOUR-CUSTOM-DEPENDENCE> ; \
|
||||
```
|
||||
|
||||
If you want to modify DolphinScheduler source code, then build and distribute your own images, you can run below command
|
||||
to build Docker images and install them locally, which you could find them by command `docker imaegs`.
|
||||
|
||||
```shell
|
||||
./mvnw -B clean install \
|
||||
-Dmaven.test.skip \
|
||||
-Dmaven.javadoc.skip \
|
||||
-Dmaven.checkstyle.skip \
|
||||
-Dmaven.deploy.skip \
|
||||
-Ddocker.tag=latest \
|
||||
-Pdocker,release
|
||||
```
|
||||
|
||||
If you want to modify DolphinScheduler source code, but also want to add customize dependencies of Docker image, you can
|
||||
modify the definition of Dockerfile after modifying the source code. You can run the following command in root source code
|
||||
directory to find all Dockerfile files.
|
||||
|
||||
```shell
|
||||
find . -iname 'Dockerfile'
|
||||
```
|
||||
|
||||
Then run the command above start with `./mvnw -B clean install`. You can see all docker images you just created with
|
||||
command `docker images` after finish commnand `mvnw`.
|
||||
|
||||
---
|
||||
|
||||
## We will collect more FAQ later
|
||||
We will collect more FAQ later
|
||||
|
||||
@ -23,6 +23,56 @@ git clone git@github.com:apache/dolphinscheduler.git
|
||||
|
||||
* 运行 `mvn clean install -Prelease -Dmaven.test.skip=true`
|
||||
|
||||
## Docker镜像构建
|
||||
|
||||
DolphinScheduler 每次发版都会同时发布 Docker 镜像，你可以在 [Docker Hub](https://hub.docker.com/search?q=DolphinScheduler) 中找到这些镜像
|
||||
|
||||
* 如果你想基于源码进行改造，然后在本地构建Docker镜像，可以在代码改造完成后运行
|
||||
```shell
|
||||
cd dolphinscheduler
|
||||
./mvnw -B clean package \
|
||||
-Dmaven.test.skip \
|
||||
-Dmaven.javadoc.skip \
|
||||
-Dmaven.checkstyle.skip \
|
||||
-Ddocker.tag=<TAG> \
|
||||
-Pdocker,release
|
||||
```
|
||||
当命令运行完了后你可以通过 `docker images` 命令查看刚刚创建的镜像
|
||||
|
||||
* 如果你想基于源码进行改造，然后构建Docker镜像并推送到 <HUB_URL>，可以在代码改造完成后运行
|
||||
```shell
|
||||
cd dolphinscheduler
|
||||
./mvnw -B clean deploy \
|
||||
-Dmaven.test.skip \
|
||||
-Dmaven.javadoc.skip \
|
||||
-Dmaven.checkstyle.skip \
|
||||
-Dmaven.deploy.skip \
|
||||
-Ddocker.tag=<TAG> \
|
||||
-Ddocker.hub=<HUB_URL> \
|
||||
-Pdocker,release
|
||||
```
|
||||
|
||||
* 如果你不仅需要改造源码，还想要自定义 Docker 镜像打包的依赖，可以在修改源码的同时修改 Dockerfile 的定义。你可以运行以下命令找到所有的 Dockerfile 文件
|
||||
|
||||
```shell
|
||||
cd dolphinscheduler
|
||||
find . -iname 'Dockerfile'
|
||||
```
|
||||
|
||||
之后再运行上面的构建镜像命令
|
||||
|
||||
* 如果你因为个性化需求想要自己打包 Docker 镜像，最佳实践是基于 DolphinScheduler 对应镜像编写 Dockerfile 文件
|
||||
|
||||
```Dockerfile
|
||||
FROM dolphinscheduler-standalone-server
|
||||
RUN apt update ; \
|
||||
apt install -y <YOUR-CUSTOM-DEPENDENCE> ; \
|
||||
```
|
||||
|
||||
> **_注意：_** Docker默认会构建并推送 linux/amd64,linux/arm64 多架构镜像
|
||||
>
|
||||
> 必须使用Docker 19.03及以后的版本，因为19.03及以后的版本包含 buildx
|
||||
|
||||
## 开发者须知
|
||||
|
||||
DolphinScheduler 开发环境配置有两个方式，分别是standalone模式，以及普通模式
|
||||
@ -33,6 +83,7 @@ DolphinScheduler 开发环境配置有两个方式，分别是standalone模式
|
||||
## DolphinScheduler Standalone快速开发模式
|
||||
|
||||
> **_注意：_** 仅供单机开发调试使用，默认使用 H2 Database,Zookeeper Testing Server
|
||||
>
|
||||
> Standalone 仅在 DolphinScheduler 1.3.9 及以后的版本支持
|
||||
|
||||
### 分支选择
|
||||
|
||||
@ -682,40 +682,12 @@ update t_ds_version set version='2.0.1';
|
||||
|
||||
---
|
||||
|
||||
## 在二进制分发包中找不到 python-gateway-server 文件夹
|
||||
## Q：在二进制分发包中找不到 python-gateway-server 文件夹
|
||||
|
||||
在 3.0.0-alpha 版本之后，Python gateway server 集成到 api server 中，当您启动 api server 后，Python gateway server 将启动。
|
||||
A：在 3.0.0-alpha 版本之后，Python gateway server 集成到 api server 中，当您启动 api server 后，Python gateway server 将启动。
|
||||
如果您不想在 api server 启动的时候启动 Python gateway server，您可以修改 api server 中的配置文件 `api-server/conf/application.yaml`
|
||||
并更改可选项 `python-gateway.enabled` 中的值设置为 `false`。
|
||||
|
||||
## 如何构建自定义的 Docker 镜像
|
||||
|
||||
DolphinScheduler 每次发版都会同时发布 Docker 镜像，你可以在 docker hub 中找到这些镜像，如果你因为个性化需求想要自己打包 docker 镜像，最佳实践是基于 DolphinScheduler 对应镜像编写 Dockerfile 文件
|
||||
|
||||
```Dockerfile
|
||||
FROM dolphinscheduler-standalone-server
|
||||
RUN apt update ; \
|
||||
apt install -y <YOUR-CUSTOM-DEPENDENCE> ; \
|
||||
```
|
||||
|
||||
如果你想基于源码进行改造，打包并分发你的镜像，可以在代码改造完成后运行。当命令运行完了后你可以通过 `docker images` 命令查看刚刚创建的镜像
|
||||
|
||||
```shell
|
||||
./mvnw -B clean install \
|
||||
-Dmaven.test.skip \
|
||||
-Dmaven.javadoc.skip \
|
||||
-Dmaven.checkstyle.skip \
|
||||
-Dmaven.deploy.skip \
|
||||
-Ddocker.tag=latest \
|
||||
-Pdocker,release
|
||||
```
|
||||
|
||||
如果你不仅需要改造源码，还想要自定义 Docker 镜像打包的依赖，可以在修改源码的同时修改 Dockerfile 的定义，你可以在源码项目根目录中运行以下命令找到所有的 Dockerfile 文件
|
||||
|
||||
```shell
|
||||
find . -iname 'Dockerfile'
|
||||
```
|
||||
|
||||
之后再运行上面的 `./mvnw -B clean install` 命令，当命令运行完成后，你可以通过命令 `docker images` 查看刚刚创建的 docker 镜像
|
||||
---
|
||||
|
||||
我们会持续收集更多的 FAQ。
|
||||
Loading…
Reference in New Issue
Block a user