Testing Terraform Scripts without spending a fortune

Introduction

How many times did you test your infrastructure by actually setting it up entirely? How much did it cost? In this post I showcase the localstack framework, aiming to be a mock of AWS. Yes you heard it well, the idea is to test your AWS services against the framework, rather than spending money to actually set it up and test your application on it, with the many moving parts involved and possibly breaking on the way.

Localstack

Localstack is a mock service for AWS, that means, you can test your AWS-based services without actually needing to ramp up anything. Specifically, this is the list of AWS services available in the free Localstack version:

• ACM
• API Gateway
• CloudFormation
• CloudWatch
• CloudWatch Logs
• DynamoDB
• DynamoDB Streams
• EC2
• Elasticsearch Service
• EventBridge (CloudWatch Events)
• Firehose
• IAM
• Kinesis
• KMS
• Lambda
• Redshift
• Route53
• S3
• SecretsManager
• SES
• SNS
• SQS
• SSM
• StepFunctions
• STS

Localstack can be installed as pip package or directly ran as docker container. The project also contains a docker-compose.yml file.

All services are exposed at the so-called edge port, on 4566. To test localstack you can setup the AWS client, directly downloading it from here, specifically:

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

we can now create an aws profile with dummy credentials, since they are not validated anyways by localstack:

aws configure --profile localstack

We can now test the creation of a S3 bucket:

aws s3 --profile localstack --endpoint-url http://localhost:4566 mb s3://mybucket

The bucket is empty.

aws s3 --profile localstack --endpoint-url http://localhost:4566 ls s3://mybucket

Let's create an empty file and upload it to the bucket: 

touch test_file.txt

aws s3 --profile localstack --endpoint-url http://localhost:4566 cp test_file.txt s3://mybucket

upload: ./test_file.txt to s3://mybucket/test_file.txt

Cool, now we got something: 

aws s3 --profile localstack --endpoint-url http://localhost:4566 ls s3://mybucket
2020-11-03 19:49:32          0 test_file.txt

Terraform

Terraform has become a de-facto standard Infastructure-as-a-Code (IaC) tool for building and managing infrastructure in an unambiguous and repeatable way.

Getting Started with Terraform

For a tutorial with Terraform, please have a look here.

Terraform consists of:

• Resources to be managed; orthogonally to those, meta-arguments can be defined, such as depends-on, count to create multiple instances of the same resource type, lifecycle to define Terraform-related behavior such as upon update or deletion;
• Modules - grouping a set of resources into a reusable named component that can be be published and maintaned as a whole; this enables code reuse and a more maintainable architecture; a natural design pattern is to separate code in a repository for modules and another one for live infrastructure;
• Providers - managers of specific resource types; providers are indexed on the Terraform Registry and can come from either Hashicorp, verified organizations or community members; No longer maintained ones are listed as "Archived". For instance, the AWS Provider is maintained directly by Hashicorp. The documentation is available here and the Github repo here.
• Input Variables - used to abstract and parametrize providers;
• Outputs - specifying values to export from a module; Terraform prints those specified output values to stdout when applying the configuration; You can alternatively explicitly query those values using the terraform output command, which is optionally provided the output name (e.g. terraform output region) to act as a resource query;
• Data Sources - definying a reference to information defined outside of Terraform;

As a declarative language, Terraform has no control flow constructs such as for-loop, although it provides with a basic if-else conditional construct, such as to define multiple variants of the modeled infrastructure, by deploying either these or those resources based on data or variable values.

These are the required steps to run terraform applications:

1. Sign up for an AWS account, create a new non-root user and assign some policies
2. Create a ~/.aws/credentials file with a new profile for the account created at 1 or export AWS_SECRET_ACCESS_KEY and AWS_ACCESS_KEY_ID
3. Install Terraform using a package manager or by downloading the binary from here or here
4. terraform init to initialize the Terraform project
5. terraform plan to see changes to the infrastructure with respect to the applied tf file
6. terraform apply to apply the changes to the infrastructure (or terraform apply -auto-approve to skip confirmation)
7. Once done terraform destroy to terminate all resources managed by the current configuration;

In this github repo, we provide a few tutorials with Terraform. Especially, in the last one we create and deploy a lambda function on Localstack.

Enjoy!