Tagbangers Blog

YugabyteDB Anywhere Terraform Provider

Terraform を使用して AWS に YugabyteDB Anywhere と Universe をデプロイします

ogawa・July 2, 2024

Yugabyte Anywhere Terraform Provider は、YugabyteDB Anywhere (YBA) のリソースを管理するための Terraform Provider です。 この記事では Terraform を使用して、AWS の複数リージョンにまたがる YugabyteDB をデプロイする手順を紹介します。

必要なもの

この記事の手順で必要なものは次のとおりです。

  1. AWS Account
  2. Terraform 1.8.x
  3. YBA のライセンスファイル

YBA は有償プロダクトのため、ライセンスが必要です。

AWS リソースの準備

YBA をセットアップして Universe（≒ クラスター）を AWS の複数リージョンにまたがってデプロイするためには、事前に次のリソースを準備する必要があります。

  1. YBA をデプロイする VPC と Subnet
  2. ノードをデプロイする VPC と Subnet（リージョンごと）
  3. YBA と各リージョンのノード間の VPC Peering
  4. 各リージョンのノード間の VPC Peering
  5. YBA の Security Group
  6. ノードの Security Group（リージョンごと）
  7. YBA の EC2 にアタッチする IAM Role
  8. ノードの EC2 にアタッチする IAM Role
  9. YBA の Key Pair
  10. YBA をインストールする EC2

今回は 3 つのリージョンにまたがって Universe を作成するため、次のように Provider を定義します。 Tokyo (ap-northeast-1) と N. Virginia (us-east-1) と London (eu-west-2) を使ってみることにします。

provider "aws" {
  alias  = "ap-northeast-1"
  region = "ap-northeast-1"
}
 
provider "aws" {
  alias  = "us-east-1"
  region = "us-east-1"
}
 
provider "aws" {
  alias  = "eu-west-2"
  region = "eu-west-2"
}

YBA をデプロイするための VPC を作成します。

module "yba_vpc" {
  source = "terraform-aws-modules/vpc/aws"
 
  providers = {
    aws = aws.ap-northeast-1
  }
 
  name           = "yba"
  cidr           = "10.0.0.0/16"
  azs            = ["ap-northeast-1a", "ap-northeast-1c", "ap-northeast-1d"]
  public_subnets = ["10.1.0.0/24", "10.1.1.0/24", "10.1.2.0/24"]
}

Universe をデプロイするための VPC を作成します。

module "cluster_vpc_1" {
  source = "terraform-aws-modules/vpc/aws"
 
  providers = {
    aws = aws.ap-northeast-1
  }
 
  name           = "yb-cluster-1"
  cidr           = "10.1.0.0/16"
  azs            = ["ap-northeast-1a", "ap-northeast-1c", "ap-northeast-1d"]
  public_subnets = ["10.1.0.0/24", "10.1.1.0/24", "10.1.2.0/24"]
}

同じように us-east-1 と eu-west-2 にも VPC を作成します。あとで VPC Peering を作成するため、VPC の CIDR は被らないようにします。

次に YBA ↔ ノード間の VPC Peering を作成します。

resource "aws_vpc_peering_connection" "yba_cluster_1" {
  provider = aws.ap-northeast-1
 
  vpc_id      = module.yba_vpc.vpc_id
  peer_vpc_id = module.cluster_vpc_1.vpc_id
  peer_region = "ap-northeast-1"
  auto_accept = false
}
 
resource "aws_vpc_peering_connection_accepter" "yba_cluster_1" {
  provider = aws.ap-northeast-1
 
  vpc_peering_connection_id = aws_vpc_peering_connection.yba_cluster_1.id
  auto_accept               = true
}
 
resource "aws_vpc_peering_connection_options" "yba_cluster_1" {
  provider = aws.ap-northeast-1
 
  vpc_peering_connection_id = local.yba_cluster_1_vpc_peering_connection_id
  accepter {
    allow_remote_vpc_dns_resolution = true
  }
}
 
resource "aws_route" "yba_cluster_1" {
  provider = aws.ap-northeast-1
  for_each = toset(module.yba_vpc.public_route_table_ids)
 
  route_table_id            = each.key
  destination_cidr_block    = module.cluster_vpc_1.vpc_cidr_block
  vpc_peering_connection_id = aws_vpc_peering_connection.yba_cluster_1.id
}
 
resource "aws_route" "cluster_1_yba" {
  provider = aws.ap-northeast-1
  for_each = toset(module.cluster_vpc_1.public_route_table_ids)
 
  route_table_id            = each.key
  destination_cidr_block    = module.yba_vpc.vpc_cidr_block
  vpc_peering_connection_id = aws_vpc_peering_connection.yba_cluster_1.id
}

3 つのリージョンのノード間の VPC Peering を作成します。

resource "aws_vpc_peering_connection" "cluster_1_cluster_2" {
  provider = aws.ap-northeast-1
 
  vpc_id      = module.cluster_vpc_1.vpc_id
  peer_vpc_id = module.cluster_vpc_2.vpc_id
  peer_region = "us-east-1"
  auto_accept = false
}
 
resource "aws_vpc_peering_connection_accepter" "cluster_1_cluster_2" {
  provider = aws.us-east-1
 
  vpc_peering_connection_id = aws_vpc_peering_connection.cluster_1_cluster_2.id
  auto_accept               = true
}
 
resource "aws_vpc_peering_connection_options" "cluster_1_cluster_2" {
  provider = aws.us-east-1
 
  vpc_peering_connection_id = local.cluster_1_cluster_2_vpc_peering_connection_id
  accepter {
    allow_remote_vpc_dns_resolution = true
  }
}
 
resource "aws_route" "cluster_1_cluster_2" {
  provider = aws.ap-northeast-1
  for_each = toset(module.cluster_vpc_1.public_route_table_ids)
 
  route_table_id            = each.key
  destination_cidr_block    = module.cluster_vpc_2.vpc_cidr_block
  vpc_peering_connection_id = aws_vpc_peering_connection.cluster_1_cluster_2.id
}
 
resource "aws_route" "cluster_2_cluster_1" {
  provider = aws.us-east-1
  for_each = toset(module.cluster_vpc_2.public_route_table_ids)
 
  route_table_id            = each.key
  destination_cidr_block    = module.cluster_vpc_1.vpc_cidr_block
  vpc_peering_connection_id = aws_vpc_peering_connection.cluster_1_cluster_2.id
}

同じように us-east-1 ↔ eu-west-2 と eu-west-2 ↔ ap-northeast-1 の VPC Peering も作成します。

次に YBA の Security Group を作成します。VPC Peering を利用した通信なので、すべてのポートを許可していますが、細かく制御したい場合は、次の URL に YBA の通信要件の記載があります。

YugabyteDB Anywhere networking requirements | YugabyteDB Docs

module "yba_security_group" {
  source = "terraform-aws-modules/security-group/aws"
  providers = {
    aws = aws.ap-northeast-1
  }
 
  name        = "yba"
  description = "yba"
  vpc_id      = module.yba_vpc.vpc_id
 
  ingress_cidr_blocks = [
    module.yba_vpc.vpc_cidr_block,
    module.cluster_vpc_1.vpc_cidr_block,
    module.cluster_vpc_2.vpc_cidr_block,
    module.cluster_vpc_3.vpc_cidr_block
  ]
  ingress_rules = ["all-all"]
  egress_rules  = ["all-all"]
}

ノードの Security Group を作成します。

module "cluster_1_security_group" {
  source = "terraform-aws-modules/security-group/aws"
  providers = {
    aws = aws.ap-northeast-1
  }
 
  name        = "yb-cluster-1"
  description = "yb-cluster-1"
  vpc_id      = module.cluster_vpc_1.vpc_id
 
  ingress_cidr_blocks = [
    module.yba_vpc.vpc_cidr_block,
    module.cluster_vpc_1.vpc_cidr_block,
    module.cluster_vpc_2.vpc_cidr_block,
    module.cluster_vpc_3.vpc_cidr_block,
  ]
  ingress_rules = ["all-all"]
  egress_rules  = ["all-all"]
}

同じように us-east-1 と eu-west-2 にも Security Group を作成します。

次に YBA の EC2 にアタッチする IAM Policy を作成します。 YBA は EC2 を動的に作成するため、EC2 に関連する Permission が必要になります。

module "yba_policy" {
  source = "terraform-aws-modules/iam/aws//modules/iam-policy"
 
  name_prefix = "yba-"
  path        = "/"
  policy      = <<EOF
{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Sid": "VisualEditor0",
           "Effect": "Allow",
           "Action": [
               "iam:PassRole",
               "ec2:AttachVolume",
               "ec2:AuthorizeSecurityGroupIngress",
               "ec2:ImportVolume",
               "ec2:ModifyVolumeAttribute",
               "ec2:DescribeInstances",
               "ec2:DescribeInstanceAttribute",
               "ec2:CreateKeyPair",
               "ec2:DescribeVolumesModifications",
               "ec2:DeleteVolume",
               "ec2:DescribeVolumeStatus",
               "ec2:StartInstances",
               "ec2:DescribeAvailabilityZones",
               "ec2:DescribeVolumes",
               "ec2:ModifyInstanceAttribute",
               "ec2:DescribeKeyPairs",
               "ec2:DescribeInstanceStatus",
               "ec2:DetachVolume",
               "ec2:ModifyVolume",
               "ec2:TerminateInstances",
               "ec2:AssignIpv6Addresses",
               "ec2:ImportKeyPair",
               "ec2:DescribeTags",
               "ec2:CreateTags",
               "ec2:RunInstances",
               "ec2:AssignPrivateIpAddresses",
               "ec2:StopInstances",
               "ec2:AllocateAddress",
               "ec2:DescribeVolumeAttribute",
               "ec2:DescribeSecurityGroups",
               "ec2:CreateVolume",
               "ec2:EnableVolumeIO",
               "ec2:DescribeImages",
               "ec2:DescribeVpcs",
               "ec2:DeleteSecurityGroup",
               "ec2:DescribeSubnets",
               "ec2:DeleteKeyPair",
               "ec2:DescribeVpcPeeringConnections",
               "ec2:DescribeRouteTables",
               "ec2:DescribeInternetGateways",
               "ec2:GetConsoleOutput",
               "ec2:CreateSnapshot",
               "ec2:DeleteSnapshot",
               "ec2:DescribeInstanceTypes"
           ],
           "Resource": "*"
       }
   ]
}
EOF
}

ノードの EC2 にアタッチする IAM Role を作成します。

module "node_role" {
  source = "terraform-aws-modules/iam/aws//modules/iam-assumable-role"
 
  create_role             = true
  create_instance_profile = true
  role_name               = "yb-node"
  role_requires_mfa       = false
 
  custom_role_policy_arns = [
    "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore",
    "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
  ]
  trusted_role_services = [
    "ec2.amazonaws.com"
  ]
}

YBA の EC2 に設定する Key Pair を作成します。

module "yba_key_pair" {
  source = "terraform-aws-modules/key-pair/aws"
  providers = {
    aws = aws.ap-northeast-1
  }
 
  key_name   = "yba"
  public_key = file("path/to/.ssh/yba_rsa.pub")
}

YBA をインストールする EC2 を作成します。

data "aws_ssm_parameter" "amazonlinux_2023" {
  name = "/aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64"
}
 
# https://docs.yugabyte.com/preview/yugabyte-platform/prepare/server-yba/
# YugabyteDB Anywhere doesn't support ARM architectures (but can be used to deploy universes to ARM-based nodes).
module "yba_ec2" {
  source = "terraform-aws-modules/ec2-instance/aws"
  providers = {
    aws = aws.ap-northeast-1
  }
 
  name                        = "yba"
  instance_type               = "t3.xlarge" # YBA requires 4 CPUs
  ami                         = data.aws_ssm_parameter.amazonlinux_2023.value
  key_name                    = module.yba_key_pair.key_pair_name
  vpc_security_group_ids      = [module.yba_security_group.security_group_id]
  subnet_id                   = module.yba_vpc.public_subnets[0]
  associate_public_ip_address = true
 
  create_iam_instance_profile = true
  iam_role_policies = {
    YBAPolicy                    = module.yba_policy.arn
    AmazonSSMManagedInstanceCore = "arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore"
    CloudWatchAgentServerPolicy  = "arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy"
  }
 
  root_block_device = [
    {
      encrypted   = true
      volume_type = "gp3"
      volume_size = 250 # YBA requires 200 GiB
    }
  ]
}

YBA のインストール

さて、ようやくここで、YugabyteDB Anywhere Terraform Provider の登場です。 provider.yba に YBA の IP アドレスを設定します。

provider "yba" {
  alias = "unauthenticated"
  host  = "<YBA_IP_ADDRESS>"
}

resource.yba_installer を使用すると、EC2 に対して YBA のインストールを実行できます。

resource "yba_installer" "install" {
  provider                  = yba.unauthenticated
  ssh_host_ip               = "<YBA_IP_ADDRESS>"
  ssh_user                  = "ec2-user"
  ssh_private_key_file_path = "path/to/.ssh/yba_rsa"
  yba_license_file          = "path/to/yba_license.lic"
  yba_version               = "2.20.4.0-b50"
}

インストールが完了したら YBA の Web Console (https://<YBA IP ADDRESS>) にアクセスしてユーザーを登録します。

Universe の作成

新しく provider.yba を定義します。api_token は YBA Web Console で生成します。

provider "yba" {
  host      = "<YBA_IP_ADDRESS>"
  api_token = "<API_TOKEN>"
}

Cloud Provider を作成します。

resource "yba_cloud_provider" "aws" {
  code = "aws"
  name = "aws-provider"
  aws_config_settings {
    use_iam_instance_profile = true
  }
  regions {
    code              = "ap-northeast-1"
    name              = "ap-northeast-1"
    security_group_id = "<SECURITY_GROUP_ID>"
    vnet_name         = "<VPC_ID>"
    yb_image          = "<AMI_ID>"
    zones {
      code   = "ap-northeast-1a"
      name   = "ap-northeast-1a"
      subnet = "<SUBNET_ID>"
    }
    zones {
      code   = "ap-northeast-1c"
      name   = "ap-northeast-1c"
      subnet = "<SUBNET_ID>"
    }
    zones {
      code   = "ap-northeast-1d"
      name   = "ap-northeast-1d"
      subnet = "<SUBNET_ID>"
    }
  }
  regions {
    code              = "us-east-1"
    name              = "us-east-1"
    security_group_id = "<SECURITY_GROUP_ID>"
    vnet_name         = "<VPC_ID>"
    yb_image          = "<AMI_ID>"
    zones {
      code   = "us-east-1a"
      name   = "us-east-1a"
      subnet = "<SUBNET_ID>"
    }
    zones {
      code   = "us-east-1c"
      name   = "us-east-1c"
      subnet = "<SUBNET_ID>"
    }
    zones {
      code   = "us-east-1d"
      name   = "us-east-1d"
      subnet = "<SUBNET_ID>"
    }
  }
  regions {
    code              = "eu-west-2"
    name              = "eu-west-2"
    security_group_id = "<SECURITY_GROUP_ID>"
    vnet_name         = "<VPC_ID>"
    yb_image          = "<AMI_ID>"
    zones {
      code   = "eu-west-2a"
      name   = "eu-west-2a"
      subnet = "<SUBNET_ID>"
    }
    zones {
      code   = "eu-west-2b"
      name   = "eu-west-2b"
      subnet = "<SUBNET_ID>"
    }
    zones {
      code   = "eu-west-2c"
      name   = "eu-west-2c"
      subnet = "<SUBNET_ID>"
    }
  }
  ssh_port        = 22
  air_gap_install = false
}

resource.yba_universe を定義して Universe を作成します。

data "yba_provider_key" "aws" {
  provider_id = yba_cloud_provider.aws.id
}
 
data "yba_release_version" "release_version" {
  // To fetch default YBDB version
}
 
resource "yba_universe" "demo" {
  clusters {
    cluster_type = "PRIMARY"
    user_intent {
      universe_name      = "demo"
      provider_type      = yba_cloud_provider.aws.code
      provider           = yba_cloud_provider.aws.id
      region_list        = yba_cloud_provider.aws.regions[*].uuid
      num_nodes          = 3
      replication_factor = 3
      instance_type      = "c5.large"
      assign_public_ip   = true
      aws_arn_string     = "<INSTANCE_PROFILE_ARN>"
      device_info {
        num_volumes  = 1
        volume_size  = 40
        storage_type = "GP3"
        disk_iops    = 3000
        throughput   = 125
      }
      use_time_sync       = true
      enable_ysql         = true
      yb_software_version = data.yba_release_version.release_version.id
      access_key_code     = data.yba_provider_key.aws.id
    }
  }
}

YBA の Web Console で進捗を確認してみましょう。 最後のステップまで完了すれば Multi-region Universe の完成です 🎉🎉