DolphinScheduler/deploy/terraform/aws
Wenjun Ruan 41a8ba9aab
Support alert server HA (#13865)
* Support alert server cluster

* Remove WORKER_ALERT_LISTEN_HOST in statefulset-dolphinscheduler-worker.yaml
2023-04-05 17:31:50 +08:00
..
packer Add Terraform deploy manifests for quick setup in AWS (#13252) 2022-12-22 16:55:32 +08:00
templates Add Terraform deploy manifests for quick setup in AWS (#13252) 2022-12-22 16:55:32 +08:00
.terraform.lock.hcl Add Terraform deploy manifests for quick setup in AWS (#13252) 2022-12-22 16:55:32 +08:00
dolphinscheduler-alert.tf Add Terraform deploy manifests for quick setup in AWS (#13252) 2022-12-22 16:55:32 +08:00
dolphinscheduler-api.tf Add Terraform deploy manifests for quick setup in AWS (#13252) 2022-12-22 16:55:32 +08:00
dolphinscheduler-master.tf Add Terraform deploy manifests for quick setup in AWS (#13252) 2022-12-22 16:55:32 +08:00
dolphinscheduler-output.tf Add Terraform deploy manifests for quick setup in AWS (#13252) 2022-12-22 16:55:32 +08:00
dolphinscheduler-standalone.tf Add Terraform deploy manifests for quick setup in AWS (#13252) 2022-12-22 16:55:32 +08:00
dolphinscheduler-variables.tf Add Terraform deploy manifests for quick setup in AWS (#13252) 2022-12-22 16:55:32 +08:00
dolphinscheduler-worker.tf Support alert server HA (#13865) 2023-04-05 17:31:50 +08:00
key-pair-main.tf Add Terraform deploy manifests for quick setup in AWS (#13252) 2022-12-22 16:55:32 +08:00
network-main.tf Add Terraform deploy manifests for quick setup in AWS (#13252) 2022-12-22 16:55:32 +08:00
network-variables.tf Add Terraform deploy manifests for quick setup in AWS (#13252) 2022-12-22 16:55:32 +08:00
os-versions.tf Add Terraform deploy manifests for quick setup in AWS (#13252) 2022-12-22 16:55:32 +08:00
provider-main.tf Add Terraform deploy manifests for quick setup in AWS (#13252) 2022-12-22 16:55:32 +08:00
provider-variables.tf Add Terraform deploy manifests for quick setup in AWS (#13252) 2022-12-22 16:55:32 +08:00
rds-main.tf Add Terraform deploy manifests for quick setup in AWS (#13252) 2022-12-22 16:55:32 +08:00
rds-output.tf Add Terraform deploy manifests for quick setup in AWS (#13252) 2022-12-22 16:55:32 +08:00
rds-variables.tf Add Terraform deploy manifests for quick setup in AWS (#13252) 2022-12-22 16:55:32 +08:00
README.md Add Terraform deploy manifests for quick setup in AWS (#13252) 2022-12-22 16:55:32 +08:00
s3-main.tf Add Terraform deploy manifests for quick setup in AWS (#13252) 2022-12-22 16:55:32 +08:00
s3-outputs.tf Add Terraform deploy manifests for quick setup in AWS (#13252) 2022-12-22 16:55:32 +08:00
s3-variables.tf Add Terraform deploy manifests for quick setup in AWS (#13252) 2022-12-22 16:55:32 +08:00
zookeeper-main.tf Add Terraform deploy manifests for quick setup in AWS (#13252) 2022-12-22 16:55:32 +08:00
zookeeper-output.tf Add Terraform deploy manifests for quick setup in AWS (#13252) 2022-12-22 16:55:32 +08:00
zookeeper-variables.tf Add Terraform deploy manifests for quick setup in AWS (#13252) 2022-12-22 16:55:32 +08:00

Prerequisites

Build AMI

Set necessary variables by creating a file ds-ami.pkrvars.hcl and adding the following variables according to your own usage.

cat <<EOF > ds-ami.pkrvars.hcl
aws_access_key = ""
aws_secret_key = ""
aws_region     = "cn-north-1"


ds_ami_name = "my-test-ds-2"

# If you want to use the official distribution tar, just set the `ds_version` to the one you want.
ds_version  = 3.1.1

# If you want to use a locally built distribution tar, set the `ds_tar` to the tar file location.
ds_tar      = "~/workspace/dolphinscheduler/dolphinscheduler-dist/target/apache-dolphinscheduler-3.1.3-SNAPSHOT-bin.tar.gz"
EOF

Then run the following command to initialize and build a custom AMI.

  • If you want to use the official distribution tar.
packer init --var-file=ds-ami.pkrvars.hcl packer/ds-ami-official.pkr.hcl
packer build --var-file=ds-ami.pkrvars.hcl packer/ds-ami-official.pkr.hcl
  • If you want to use the locally built distribution tar.
packer init --var-file=ds-ami.pkrvars.hcl packer/ds-ami-local.pkr.hcl
packer build --var-file=ds-ami.pkrvars.hcl packer/ds-ami-local.pkr.hcl

Create resources

Set necessary variables by creating a file terraform.tfvars and adding the following variables according to your own usage.

Make sure ds_ami_name is the same as the one in ds-ami.pkrvars.hcl above.

cat <<EOF > terraform.tfvars
aws_access_key = ""
aws_secret_key = ""

name_prefix = "test-ds-terraform"
ds_ami_name = "my-test-ds"

ds_component_replicas = {
  master            = 1
  worker            = 1
  alert             = 1
  api               = 1
  standalone_server = 0
}
EOF

Then run the following commands to apply necessary resources.

terraform init -var-file=terraform.tfvars
terraform apply -var-file=terraform.tfvars -auto-approve

Open DolphinScheduler UI

open http://$(terraform output -json api_server_instance_public_dns | jq -r '.[0]'):12345/dolphinscheduler/ui

Inputs

Name Description Type Default Required
aws_access_key AWS access key string n/a yes
aws_region AWS region string "cn-north-1" no
aws_secret_key AWS secret key string n/a yes
db_instance_class Database instance class string "db.t3.micro" no
db_password Database password string n/a yes
db_username Database username string "dolphinscheduler" no
ds_ami_name Name of DolphinScheduler AMI string "dolphinscheduler-ami" no
ds_component_replicas Replicas of the DolphinScheduler Components map(number)
{
"alert": 1,
"api": 1,
"master": 1,
"standalone_server": 0,
"worker": 1
}
no
ds_version DolphinScheduler Version string "3.1.1" no
name_prefix Name prefix for all resources string "dolphinscheduler" no
private_subnet_cidr_blocks Available CIDR blocks for private subnets list(string)
[
"10.0.101.0/24",
"10.0.102.0/24",
"10.0.103.0/24",
"10.0.104.0/24"
]
no
public_subnet_cidr_blocks CIDR blocks for the public subnets list(string)
[
"10.0.1.0/24",
"10.0.2.0/24",
"10.0.3.0/24",
"10.0.4.0/24"
]
no
s3_bucket_prefix n/a string "dolphinscheduler-test-" no
subnet_count Number of subnets map(number)
{
"private": 2,
"public": 1
}
no
tags Tags to apply to all resources map(string)
{
"Deployment": "Test"
}
no
vm_associate_public_ip_address Associate a public IP address to the EC2 instance map(bool)
{
"alert": true,
"api": true,
"master": true,
"standalone_server": true,
"worker": true
}
no
vm_data_volume_size Data volume size of the EC2 Instance map(number)
{
"alert": 10,
"api": 10,
"master": 10,
"standalone_server": 10,
"worker": 10
}
no
vm_data_volume_type Data volume type of the EC2 Instance map(string)
{
"alert": "gp2",
"api": "gp2",
"master": "gp2",
"standalone_server": "gp2",
"worker": "gp2"
}
no
vm_instance_type EC2 instance type map(string)
{
"alert": "t2.micro",
"api": "t2.small",
"master": "t2.medium",
"standalone_server": "t2.small",
"worker": "t2.medium"
}
no
vm_root_volume_size Root Volume size of the EC2 Instance map(number)
{
"alert": 30,
"api": 30,
"master": 30,
"standalone_server": 30,
"worker": 30
}
no
vm_root_volume_type Root volume type of the EC2 Instance map(string)
{
"alert": "gp2",
"api": "gp2",
"master": "gp2",
"standalone_server": "gp2",
"worker": "gp2"
}
no
vpc_cidr CIDR for the VPC string "10.0.0.0/16" no
zookeeper_connect_string Zookeeper connect string, if empty, will create a single-node zookeeper for demonstration, don't use this in production string "" no

Outputs

Name Description
alert_server_instance_id Instance IDs of alert instances
alert_server_instance_private_ip Private IPs of alert instances
alert_server_instance_public_dns Public domain names of alert instances
alert_server_instance_public_ip Public IPs of alert instances
api_server_instance_id Instance IDs of api instances
api_server_instance_private_ip Private IPs of api instances
api_server_instance_public_dns Public domain names of api instances
api_server_instance_public_ip Public IPs of api instances
db_address Database address
db_name Database name
db_port Database port
master_server_instance_id Instance IDs of master instances
master_server_instance_private_ip Private IPs of master instances
master_server_instance_public_dns Public domain names of master instances
master_server_instance_public_ip Public IPs of master instances
s3_access_key S3 access key
s3_address S3 address
s3_bucket S3 bucket name
s3_regional_domain_name S3 regional domain name
s3_secret S3 access secret
vm_server_instance_id Instance IDs of standalone instances
vm_server_instance_private_ip Private IPs of standalone instances
vm_server_instance_public_dns Public domain names of standalone instances
vm_server_instance_public_ip Public IPs of standalone instances
worker_server_instance_id Instance IDs of worker instances
worker_server_instance_private_ip Private IPs of worker instances
worker_server_instance_public_dns Public domain names of worker instances
worker_server_instance_public_ip Public IPs of worker instances
zookeeper_server_instance_id Instance IDs of zookeeper instances
zookeeper_server_instance_private_ip Private IPs of zookeeper instances
zookeeper_server_instance_public_dns Public domain names of zookeeper instances
zookeeper_server_instance_public_ip Public IPs of zookeeper instances