Today we will look at how to setup EC2 instance with Terraform.
- Set up Terraform
- Spin up EC2
- Externalise secrets and other resources with terraform variables.
- Set up Vault as secret repo
1. Set up Terraform
So first thing first, quick installation guide, visit https://www.terraform.io/downloads.html , pick up right version and download:
➜ apps wget https://releases.hashicorp.com/terraform/0.11.1/terraform_0.11.1_darwin_amd64.zip\?_ga\=2.1738614.654909398.1512400028-228831855.1511115744 --2017-12-04 15:16:06-- https://releases.hashicorp.com/terraform/0.11.1/terraform_0.11.1_darwin_amd64.zip?_ga=2.1738614.654909398.1512400028-228831855.1511115744 Resolving releases.hashicorp.com... 151.101.17.183, 2a04:4e42:4::439 Connecting to releases.hashicorp.com|151.101.17.183|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 15750266 (15M) [application/zip] Saving to: ‘terraform_0.11.1_darwin_amd64.zip?_ga=2.1738614.654909398.1512400028-228831855.1511115744’ terraform_0.11.1_darwin_amd64.zip?_ga=2.17386 100%[=================================================================================================>] 15.02M 499KB/s in 30s 2017-12-04 15:16:36 (517 KB/s) - ‘terraform_0.11.1_darwin_amd64.zip?_ga=2.1738614.654909398.1512400028-228831855.1511115744’ saved [15750266/15750266]
Then unzip:
➜ apps unzip terraform_0.11.1_darwin_amd64.zip\?_ga=2.1738614.654909398.1512400028-228831855.1511115744 Archive: terraform_0.11.1_darwin_amd64.zip?_ga=2.1738614.654909398.1512400028-228831855.1511115744 inflating: terraform
Finally make sure location added to PATH:
➜ ~ export PATH=~/apps:$PATH
Check installation works:
➜ ~ terraform -v Terraform v0.11.1
2. Spin up EC2
The plan is to spin up latest Ubuntu.
Before terraform can do anything to AWS we need to create an access for terraform user in AWS IAM,
here is how:
Pick up EC2 AMI from https://cloud-images.ubuntu.com/locator/ec2/, you need to specify your region and then ubuntu release name, so I used ‘eu-west-2 zesty’.
So once AMI id is found and user is created we are ready to perform our first test.
Create a file instance.tf as below(use your own access key and secret of course):
➜ terraform_demo cat instance.tf
provider "aws" {
access_key = "AKIAJ3XZ5SA3OKEWGGIQ"
secret_key = "vU9d6MmQFX+RODfBaK5A1YrDVpOSIt2KZzE9FntC"
region = "eu-west-2"
}
resource "aws_instance" "ubuntu_zesty" {
ami = "ami-6b7f610f"
instance_type = "t2.micro"
}
➜ terraform_demo
We are ready to provision it now, let’s first create plan:
➜ terraform_demo terraform plan Plugin reinitialization required. Please run "terraform init". Reason: Could not satisfy plugin requirements. Plugins are external binaries that Terraform uses to access and manipulate resources. The configuration provided requires plugins which can't be located, don't satisfy the version constraints, or are otherwise incompatible. 1 error(s) occurred: * provider.aws: no suitable version installed version requirements: "(any version)" versions installed: none Terraform automatically discovers provider requirements from your configuration, including providers used in child modules. To see the requirements and constraints from each module, run "terraform providers". Error: error satisfying plugin requirements
Oops, we need to initialise a Terraform working directory, let’s fix it:
➜ terraform_demo terraform init Initializing provider plugins... - Checking for available provider plugins on https://releases.hashicorp.com... - Downloading plugin for provider "aws" (1.5.0)... The following providers do not have any version constraints in configuration, so the latest version was installed. To prevent automatic upgrades to new major versions that may contain breaking changes, it is recommended to add version = "..." constraints to the corresponding provider blocks in configuration, with the constraint strings suggested below. * provider.aws: version = "~> 1.5" Terraform has been successfully initialized! You may now begin working with Terraform. Try running "terraform plan" to see any changes that are required for your infrastructure. All Terraform commands should now work. If you ever set or change modules or backend configuration for Terraform, rerun this command to reinitialize your working directory. If you forget, other commands will detect it and remind you to do so if necessary.
Now we are ready to generate and show an execution plan:
➜ terraform_demo terraform plan
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
+ aws_instance.ubuntu_zesty
id: <computed>
ami: "ami-6b7f610f"
associate_public_ip_address: <computed>
availability_zone: <computed>
ebs_block_device.#: <computed>
ephemeral_block_device.#: <computed>
instance_state: <computed>
instance_type: "t2.micro"
ipv6_address_count: <computed>
ipv6_addresses.#: <computed>
key_name: <computed>
network_interface.#: <computed>
network_interface_id: <computed>
placement_group: <computed>
primary_network_interface_id: <computed>
private_dns: <computed>
private_ip: <computed>
public_dns: <computed>
public_ip: <computed>
root_block_device.#: <computed>
security_groups.#: <computed>
source_dest_check: "true"
subnet_id: <computed>
tenancy: <computed>
volume_tags.%: <computed>
vpc_security_group_ids.#: <computed>
Plan: 1 to add, 0 to change, 0 to destroy.
------------------------------------------------------------------------
Note: You didn't specify an "-out" parameter to save this plan, so Terraform
can't guarantee that exactly these actions will be performed if
"terraform apply" is subsequently run.
If we are happy, we can proceed, I am using ‘auto-approve’ flag so command doesn’t stop and asks for annoying confirmation:
➜ terraform_demo terraform apply -auto-approve aws_instance.ubuntu_zesty: Creating... ami: "" => "ami-6b7f610f" associate_public_ip_address: "" => "<computed>" availability_zone: "" => "<computed>" ebs_block_device.#: "" => "<computed>" ephemeral_block_device.#: "" => "<computed>" instance_state: "" => "<computed>" instance_type: "" => "t2.micro" ipv6_address_count: "" => "<computed>" ipv6_addresses.#: "" => "<computed>" key_name: "" => "<computed>" network_interface.#: "" => "<computed>" network_interface_id: "" => "<computed>" placement_group: "" => "<computed>" primary_network_interface_id: "" => "<computed>" private_dns: "" => "<computed>" private_ip: "" => "<computed>" public_dns: "" => "<computed>" public_ip: "" => "<computed>" root_block_device.#: "" => "<computed>" security_groups.#: "" => "<computed>" source_dest_check: "" => "true" subnet_id: "" => "<computed>" tenancy: "" => "<computed>" volume_tags.%: "" => "<computed>" vpc_security_group_ids.#: "" => "<computed>" aws_instance.ubuntu_zesty: Still creating... (10s elapsed) aws_instance.ubuntu_zesty: Still creating... (20s elapsed) aws_instance.ubuntu_zesty: Creation complete after 23s (ID: i-0aa512bde10cb1358) Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
We can check the state from file:
➜ terraform_demo cat terraform.tfstate
{
"version": 3,
"terraform_version": "0.11.1",
"serial": 1,
"lineage": "1fa856fe-712d-4467-85ab-309a05254fdb",
"modules": [
{
"path": [
"root"
],
"outputs": {},
"resources": {
"aws_instance.ubuntu_zesty": {
"type": "aws_instance",
"depends_on": [],
"primary": {
"id": "i-000014b0382871d78",
"attributes": {
"ami": "ami-6b7f610f",
"associate_public_ip_address": "true",
"availability_zone": "eu-west-2a",
"disable_api_termination": "false",
"ebs_block_device.#": "0",
"ebs_optimized": "false",
"ephemeral_block_device.#": "0",
"iam_instance_profile": "",
"id": "i-000014b0382871d78",
"instance_state": "running",
"instance_type": "t2.micro",
"ipv6_addresses.#": "0",
"key_name": "",
"monitoring": "false",
"network_interface.#": "0",
"network_interface_id": "eni-e103a6b4",
"placement_group": "",
"primary_network_interface_id": "eni-e103a6b4",
"private_dns": "ip-172-31-0-208.eu-west-2.compute.internal",
"private_ip": "172.31.0.208",
"public_dns": "ec2-52-56-204-79.eu-west-2.compute.amazonaws.com",
"public_ip": "52.56.204.79",
"root_block_device.#": "1",
"root_block_device.0.delete_on_termination": "true",
"root_block_device.0.iops": "100",
"root_block_device.0.volume_size": "8",
"root_block_device.0.volume_type": "gp2",
"security_groups.#": "1",
"security_groups.3814588639": "default",
"source_dest_check": "true",
"subnet_id": "subnet-42f80a39",
"tags.%": "0",
"tenancy": "default",
"volume_tags.%": "0",
"vpc_security_group_ids.#": "0"
},
"meta": {
"e2bfb730-ecaa-11e6-8f88-34363bc7c4c0": {
"create": 600000000000,
"delete": 600000000000,
"update": 600000000000
},
"schema_version": "1"
},
"tainted": false
},
"deposed": [],
"provider": "provider.aws"
}
},
"depends_on": []
}
]
}
Let’s see how we can destroy it now, you can also view it in EC2 console if you like, before destroying.
Again, I am running it with ‘force’ flag, so it doesn’t ask for confirmation.
I have to admit there is no consistency between apply and destroy, as each uses different argument for same semantic:
➜ terraform_demo terraform destroy -force aws_instance.ubuntu_zesty: Refreshing state... (ID: i-000014b0382871d78) aws_instance.ubuntu_zesty: Destroying... (ID: i-000014b0382871d78) aws_instance.ubuntu_zesty: Still destroying... (ID: i-000014b0382871d78, 10s elapsed) aws_instance.ubuntu_zesty: Still destroying... (ID: i-000014b0382871d78, 20s elapsed) aws_instance.ubuntu_zesty: Still destroying... (ID: i-000014b0382871d78, 30s elapsed) aws_instance.ubuntu_zesty: Still destroying... (ID: i-000014b0382871d78, 40s elapsed) aws_instance.ubuntu_zesty: Still destroying... (ID: i-000014b0382871d78, 50s elapsed) aws_instance.ubuntu_zesty: Destruction complete after 51s Destroy complete! Resources: 1 destroyed. ➜ terraform_demo
Please note that terraform created a backup for the state:
➜ terraform_demo ls instance.tf terraform.tfstate terraform.tfstate.backup ➜ terraform_demo
3. Externalise secrets and other resources with terraform variables.
Let’s look at Terraform Variables now.
Obviously you don’t want to save the AWS secret in the terraform sources. For that reason
you would normally use variables. Let’s split our config into two parts:
➜ terraform_demo tail -n +1 *.tf
==> instance.tf <==
provider "aws" {
access_key = "AKIAIUWY24KNLBE3OQKA"
secret_key = "${var.secret_key}"
region = "eu-west-2"
}
resource "aws_instance" "ubuntu_zesty" {
ami = "ami-6b7f610f"
instance_type = "t2.micro"
}
==> variabels.tf <==
variable "secret_key" {}
➜ terraform_demo
As you see instance.tf refers to secret_key variable and variabels.tf declares it, but as the value is not assigned, terraform will ask it upon running:
terraform_demo terraform plan var.secret_key Enter a value:
Now if you want to automate this and not enter manually, you would need to read the value with some tool, I used vault for instance.
4. Set up Vault to as secret repo
If you don’t want have vault installed and don’t want to install it either (in case you are very lazy guy) you can simply test this with echo:
➜ terraform_demo echo NVMqHQYd3/hw6uXX+EQk57MNfKEP6MImUl0l8Lfd | terraform plan var.secret_key Enter a value: Refreshing Terraform state in-memory prior to plan... The refreshed state will be used to calculate this plan, but will not be persisted to local or remote state storage. .. ....
Here is a quick way to test it with vault. Go to https://www.vaultproject.io/downloads.html and download version for your machine, then unzip it:
➜ apps wget https://releases.hashicorp.com/vault/0.9.0/vault_0.9.0_darwin_amd64.zip\?_ga\=2.252075081.1958392905.1512420796-1969448237.1511121920
--2017-12-05 17:25:10-- https://releases.hashicorp.com/vault/0.9.0/vault_0.9.0_darwin_amd64.zip?_ga=2.252075081.1958392905.1512420796-1969448237.1511121920
Resolving releases.hashicorp.com... 151.101.17.183, 2a04:4e42:4::439
Connecting to releases.hashicorp.com|151.101.17.183|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 17470290 (17M) [application/zip]
Saving to: ‘vault_0.9.0_darwin_amd64.zip?_ga=2.252075081.1958392905.1512420796-1969448237.1511121920’
vault_0.9.0_darwin_amd64.zip?_ga=2.252075081. 100%[=================================================================================================>] 16.66M 351KB/s in 59s
2017-12-05 17:26:09 (289 KB/s) - ‘vault_0.9.0_darwin_amd64.zip?_ga=2.252075081.1958392905.1512420796-1969448237.1511121920’ saved [17470290/17470290]
➜ apps unzip vault_0.9.0_darwin_amd64.zip\?_ga=2.252075081.1958392905.1512420796-1969448237.1511121920
Archive: vault_0.9.0_darwin_amd64.zip?_ga=2.252075081.1958392905.1512420796-1969448237.1511121920
inflating: vault
➜ apps vault -v
Vault v0.9.0 ('bdac1854478538052ba5b7ec9a9ec688d35a3335')
Once you done these steps, add vault installation to your PATH, also add this line to your PATH:
export VAULT_ADDR='http://127.0.0.1:8200'
Otherwise you will get error asby default it tries to connect to https:
➜ ~ vault write secret/aws value=NVMqHQYd3/hw6uXX+EQk57MNfKEP6MImUl0l8Lfd Error writing data to secret/aws: Put https://127.0.0.1:8200/v1/secret/aws: http: server gave HTTP response to HTTPS client
Now start the vault:
vault server -dev
You are now ready to persist new secret and then read it:
➜ ~ vault read secret/aws No value found at secret/aws ➜ ~ vault write secret/aws value=NVMqHQYd3/hw6uXX+EQk57MNfKEP6MImUl0l8Lfd Success! Data written to: secret/aws ➜ ~ vault read secret/aws Key Value --- ----- refresh_interval 768h0m0s value NVMqHQYd3/hw6uXX+EQk57MNfKEP6MImUl0l8Lfd
We can now ready to feed the secret to terraform, note I used ‘field’ argument so there is not need to do any hack with grep/sed,
it alwasy worth reading it’s docs, ‘vault read -help’ in this case:
➜ terraform_demo vault read -field=value secret/aws | terraform plan var.secret_key Enter a value: Refreshing Terraform state in-memory prior to plan... The refreshed state will be used to calculate this plan, but will not be persisted to local or remote state storage. .. .... ......
Please note that terraform plan is not a dry-run sort of a thing, it actually goes to AWS
and makes necessary arrangements to make sure ‘apply’ is going to work, as an example, if I
delete the user from IAM and try again, I will get an error:
➜ terraform_demo echo NVMqHQYd3/hw6uXX+EQk57MNfKEP6MImUl0l8Lfd | terraform plan var.secret_key Enter a value: Refreshing Terraform state in-memory prior to plan... The refreshed state will be used to calculate this plan, but will not be persisted to local or remote state storage. ------------------------------------------------------------------------ Error: Error running plan: 1 error(s) occurred: * provider.aws: InvalidClientTokenId: The security token included in the request is invalid. status code: 403, request id: f67be4a2-d938-11e7-bb3b-6f82b8195005 ➜ terraform_demo
That is it, we now familiar with these two nice devops tools made by hashicorp. Consul is also made
by them by the way, and I have a nice article about how you can use it for service discovery.