What Is AWS CloudFormation?
If you need to deploy more than just a handful of services or applications on AWS, you will quickly discover that it is tedious and time-consuming to set each one up manually.
Not to mention, having to configure each AWS resource by hand also leaves you at a much higher risk of making errors or introducing inconsistencies.
Enter AWS CloudFormation.
CloudFormation is an infrastructure automation platform for AWS that deploys AWS resources in a repeatable, testable and auditable manner.
This article offers an overview of AWS CloudFormation, including how it works, its benefits, and how to create and deploy CloudFormation templates using the console, CloudFormation Designer, and the AWS command line.
We also introduce the concept of CloudFormation Hooks and share details on our own CloudFormation Hook which you can use to embed IAM best practices within your organization.
What Is AWS CloudFormation?
AWS CloudFormation is an AWS service that uses template files to automate the setup of AWS resources.
It can also be described as infrastructure automation or Infrastructure-as-Code (IaC) tool and a cloud automation solution because it can automate the setup and deployment of various Infrastructure-as-a-Service (IaaS) offerings on the AWS CloudFormation supports virtually every service that runs in AWS. (A full list of supported services is available here.)
You can use CloudFormation to automate the configuration of workloads that run on the most popular AWS services, like the EC2 compute service, the S3 storage service, and the IAM service for configuring access control.
You can also apply CloudFormation templates to AWS services that cater to niche use cases, like Ground Station, the AWS satellite management solution.
In general, if a service runs on AWS, it is a safe bet that you can use CloudFormation to automate its configuration and deployment.
It is worth noting that CloudFormation is not the only way to configure and deploy services on AWS. You can handle these processes manually using the AWS command-line interface, API, or Web console. Manual provisioning is the approach that teams typically take when they are just getting started with AWS and learning how to deploy services. However, as they scale their environments up in size, many teams quickly realize that they need a solution like CloudFormation to make the deployment process faster and more consistent.
Benefits of CloudFormation
CloudFormation and other AWS-compatible IaC tools offer a range of benefits that make cloud service deployment and management faster and more efficient.
Deployment speed
When you create CloudFormation templates to manage how AWS resources are configured and deployed, you can deploy multiple instances of the same resources almost instantaneously using just one template. This approach leads to much faster deployment than you could achieve if you had to manually set up each deployment by running commands on the CLI or pressing buttons in the AWS console.
The caveat, of course, is that you have to spend time setting up your CloudFormation templates. However, if you will be repeating the same type of deployment several times, it will be much faster overall to create a CloudFormation template that you can reuse for each deployment than to configure each one manually.
Scaling up
Even if you do not initially expect to deploy multiple instances of the same AWS resources, CloudFormation templates are useful because they ensure that you can scale your environment up quickly when the time comes. By keeping CloudFormation templates on hand, you will know that you can add more virtual machine instances or storage space, for example, at a moment's notice if your applications experience increased traffic and you need to scale your environment up.
Alternatively, when demand decreases and you want to scale down to save money, you can take some of your deployments offline while still retaining the ability to redeploy them quickly using CloudFormation when demand increases.
Service integration
A single CloudFormation template can manage the deployment of individual services or resources and multiple resources. This management ability means you can use CloudFormation to integrate different AWS cloud services. For example, you could write a template that sets up an EC2 virtual machine within an AWS Virtual Private Cloud (VPC) or deploys an S3 storage bucket and configures access control for it using the IAM service.
Managing multiple services through a single template makes it easy to integrate AWS services as you build out a complete cloud environment.
Consistency
When you use CloudFormation templates to define and deploy AWS resources, you can apply precisely the same configuration repeatedly. In this way, CloudFormation ensures that your applications and services will be consistent and identical, no matter how many instances you create.
The alternative approach, which is to set up each resource by hand, introduces the risk that the engineer who performs the work might apply different settings to different instances, resulting in inconsistency. In turn, your environment would be more challenging to manage because some resources would look different than others, even if they perform the same primary job. You might have different types of EC2 instances hosting replicas of the same application, for instance, or different IAM access controls for the same service. This inconsistency would make it challenging to manage resources uniformly.
Security
Along similar lines, although CloudFormation is not a security tool per se, it can improve the overall security of your AWS environment by reducing the risk of oversights or human errors that could turn into breaches. As long as you design your CloudFormation templates to be secure, you do not need to worry that an engineer who deploys resources will forget to turn on important access control, for example, or leave data exposed to unrestricted, public access.
Easy updates
In addition to deploying new resources, you can apply changes to existing resources with CloudFormation templates. This ability simplifies the process of, for example, adding more storage to a fleet of ec2 instances or changing access control rules.
Auditing and change management
When you use CloudFormation to manage your infrastructure, you can track changes based on which templates you have applied and how they change over time. Change tracking in CloudFormation means that you will be able to determine how your AWS services and resources have changed over time without looking through logs to reconstruct the timeline of updates.
Alternatives to CloudFormation
CloudFormation is Amazon's own IaC solution, but it is not the only cloud automation or IaC tool that you can use in the AWS cloud to achieve the benefits described above. Your team can also take advantage of third-party IaC platforms compatible with AWS, such as HashiCorp Terraform or Ansible.
The main advantage of choosing an alternative to CloudFormation is that most third-party IaC tools can configure resources running on various public clouds, not just AWS. If you use multiple clouds at once -- in other words, if you run some workloads on AWS and others on Microsoft Azure -- a third-party IaC tool platform is convenient because it allows you to use the same configuration management tool for all of your workloads.
On the other hand, as AWS's native IaC platform, CloudFormation offers the deepest level of integration with the AWS cloud, including features like Designer, which lets you create and modify CloudFormation templates directly on the AWS website.
CloudFormation also provides a high level of assurance that your templates will always remain compatible with AWS services, even if Amazon makes changes to its services. If Amazon were to update one of its services, there is a possibility that third-party IaC tools would stop working with it, at least until the developers of the tools have time to catch up with the changes. This is not a frequent issue because Amazon does not often make significant changes to its cloud services. New Services may take time to be available in CloudFormation, but third-party updates will take longer still. Nevertheless, it is a factor to weigh if you consider using CloudFormation or alternative infrastructure automation and cloud automation solutions.
The Definitive Guide to Cloud Migration in the Enterprise: Here Be Dragons!
The road to cloud migration is full of DRAGONS...
We explore the four dragons that are most likely to burn your transformation to the ground and the tools to slay them!
CloudFormation Template Terms and Concepts
It is helpful to understand the core concepts around which CloudFormation templates structure resources, variables, and functions.
Template
A CloudFormation template is simply a text file, formatted in a specific way (see below for details on formatting), that defines how AWS services or resources should be configured and deployed.
Stacks
A stack is a term AWS uses to refer to a collection of multiple AWS resources -- such as EC2 virtual machines, S3 storage, and IAM access controls -- that you can manage together using a single template.
Formatting
CloudFormation supports templates that are formatted using either JSON or YAML. These are widely used file formats for structuring text files. Most other IaC tools use the same formatting languages, as do platforms like Kubernetes.
Parameters
If you need to apply unique settings for each deployment, you can do so using parameters. Parameters let you define custom values for each deployment that CloudFormation will apply at runtime.
Conditions
You can also fine-tune deployments by setting conditions, which let you define conditional rules to govern precisely how each deployment proceeds.
Change sets
If you want to update a deployment using CloudFormation, you can update the template you used to create the deployment. You can then create a change set, which summarizes the changes that the updated template will apply before making the change.
Functions
There are several ways to get data into a CloudFormation template, with parameters being the primary. But those parameters may not be known at deployment time. CloudFormation Functions allow CloudFormation Designers to retrieve data from resources deployed in the current CloudFormation or from external sources in the AWS account. Ref is used extensively to reference other resources inside the template like the example below. It creates an EIP for the instance created earlier in the template.
"MyEIP" : {
"Type" : "AWS::EC2::EIP",
"Properties" : {
"InstanceId" : { "Ref" : "MyEC2Instance" }
}
}How to Create a CloudFormation Template
There are two ways to create a CloudFormation template:
- Using a pre-existing template as the foundation or
- Writing an entirely new template from scratch
In most cases, the former approach will be faster and more comfortable, especially if you are new to CloudFormation and have a simple configuration to deploy. It usually only makes sense to create new templates from scratch if you have an incredibly complex deployment, or you need to work with niche AWS resources that are not well represented among AWS's existing template library.
Pre-existing templates
To use a pre-existing template as the basis for your deployment, browse the AWS template collection to find a template that aligns with your needs. For example, if you want to deploy an EC2 instance and configure storage for it, the template called "Amazon EC2 instance with an ephemeral drive" would be an excellent place to start.
Once you have chosen a template, you can either download it to your computer, edit it in a local text editor, or open it in AWS CloudFormation Designer, an online tool that AWS offers to create and modify CloudFormation templates. Designer provides visualizations and drag-and-drop functionality that makes it easy to build out deployments without a great deal of coding (although you should expect to write some code if you need to implement more than just basic functionality).
Here is the view from the CloudFormation Template Designer
New templates
If you choose to create a new template, you can do so using any text editor. You can quickly build out new deployment rules using CloudFormation Designer's visualization features because it supports new templates as well as previously created ones.
How to Deploy a CloudFormation Template
Once your template is ready, you can begin the deployment process to create the resources you have defined in the template in your actual AWS environment.
There are multiple ways to deploy a CloudFormation template. The approach you take will depend in part on how you created your template and in part on which AWS interface you prefer:
- The AWS console: If your template is a text file stored on your local computer, the easiest way to deploy it is to log into the AWS CloudFormation console https://console.aws.amazon.com/cloudformation and click Create New Stack. The console will walk you through the steps for naming your template and uploading your computer's template file. Once you have completed the steps and reviewed your configuration, click the Create button to deploy your template.
- From CloudFormation Designer: If you create your template in CloudFormation Designer, you can deploy it directly from there by clicking the Create Stack button, following the directions on the screen, and pressing Create when you are ready to apply the template.
- The AWS CLI: Using the AWS CLI tool, you can use the aws cloudformation deploy command to deploy a template. Use command-line arguments to define where your template is stored (typically, you would upload it to S3 first and point the CLI to that file) and other options you may want to configure. The AWS documentation provides complete details on CLI deployments.
Updating a CloudFormation Stack
There are two basic ways to make changes to a stack that you have deployed using a CloudFormation template.
One is to update the corresponding template, then immediately deploy the updated template using one of the deployment methods described above. This method is what AWS calls a direct update. It is the fastest but also the bluntest way to update your deployment. It does not provide a chance to view the impact of your intended changes before applying them.
The other approach, which offers more control and the ability to preview changes before they take effect, creates a change set and uses it to make the updates. You can create a changeset in the CloudFormation console by selecting Stacks and then Stack actions. You can then select the template you want to update, and the console will then walk through the steps of changing your template, reviewing the changes, and applying them. You can also create a change set on the AWS CLI using the create-change-set command.
Advanced CloudFormation
There are several advanced functionalities that can be utilized in CloudFormation. These functionalities are helper scripts in CloudFormation. They extend CloudFormation to allow designers to execute scripts outside of CloudFormation
- CloudFormation Modules: Modules allow you to encapsulate a reusable CloudFormation template into a single resource definition to make it easier to modularize and reuse your infrastructure definitions, capturing best practices for security, resiliency, scalability, operational excellence, and cost optimization.
- CloudFormation Hooks: Hooks allow you to embed custom resource validation rules into the CloudFormation engine. This gives you full control to validate the configuration and make AWS API calls before the resource configuration continues. Hooks can be triggered off resource creation, updates and deletion, giving you the freedom to augment the entire lifecycle of any resource (more on this below).
- cfn-init: The cfn-init helper script reads template metadata from the AWS::CloudFormation::Init key and acts accordingly to Fetch and parse metadata from CloudFormation. It can also install packages, write files to disk and enable or disable services on an EC2 instance.
- cfn-signal: Designers use cfn-signal as a signal with a CreationPolicy or WaitCondition, to synchronize resources in the stack when the prerequisites are ready. This allows for resources to update or execute work and then allow CloudFormation to continue working on creating resources in the stack.
- cfn-get-metadata: Used to retrieve metadata from the CloudFormation stack.
- cfn-hup: Used to check for updates to metadata and execute custom hooks when changes are detected such as updates to scripts and files.
Services like AWS SAM and eksctl use CloudFormation on the back end to ease the use of Serverless and Kubernetes cluster deployments.
What Are AWS CloudFormation Hooks?
AWS has just launched a new addition to the CloudFormation service called Hooks, enabling you to embed custom resource validation rules into the CloudFormation engine.
This prevents users across your entire AWS estate from deploying resources outside of your guardrails and standards. Hooks can be triggered off resource creation, updates and deletion, giving you the freedom to augment the entire lifecycle of any resource.
What Are the Benefits of Using CloudFormation Hooks?
At the simplest level, a hook is a Lambda function automatically invoked in the background of a CloudFormation stack execution.
This gives you full control to validate the configuration and make AWS API calls before the resource configuration continues and opens up a wide range of use cases where CloudFormation Hooks can elevate your cloud governance and company policy adoption.
Plus, it provides a much lower TCO approach that is embedded directly into CloudFormation, making it impossible to circumvent this when using the service, enforcing security integrity across the cloud estate.
Then Vs Now: How Is This Better Than Before?
Previously, teams have had to embed and manage tooling in their pipelines which can be circumvented, hand-crafted CloudFormation modules and service catalogue products, or write custom resources—All options that fail to provide the same guarantees and require significantly more time investment.
Example Use Case for CloudFormation Hooks
Imagine the scenario where your company policy requires S3 buckets be secured with KMS keys for Server Side Encryption.
CloudFormation Hooks can validate the AWS::S3::Bucket resource definition and block the creation of an unencrypted S3 Bucket or manipulation of the Server Side Encryption configuration during an update execution.
This allows you to be confident that the policy is being enforced, actively preventing misconfigured S3 buckets from ever existing.
Use Contino’s CloudFormation Hook to Embed IAM best Practices
Here at Contino, we work with some of the biggest enterprises in the world on implementing a least-privilege identity perimeter that accelerates cloud driven innovation instead of stifling it.
Given the inherent complexity of IAM on AWS— and with IAM policy writing being a decentralized activity— we need to be able to validate policies at scale for common antipatterns. To support this endeavour, we have published a CloudFormation Hook that validates your IAM policies against AWS Access Analyzer and stops the CloudFormation execution if AWS recommended standards are not followed.
To find out how our CloudFormation hook can be used to…. check out our post on Contino Engineering.
Conclusion
Manually setting up and deploying AWS resources is an unproductive use of your team's time. It also increases the risk of configuration oversights and inconsistencies that can lead to management problems and security risks. Furthermore, it makes it harder to update or scale resources quickly.
By leveraging an IaC tool such as CloudFormation, your team can streamline the AWS deployment process. You can define your resource configurations once, then deploy them as many times as you need. You can also quickly and predictably update resources via change sets, and you can use your CloudFormation templates to keep track of how you configure your resources and how they have changed over time.
If you're interested in learning more about CloudFormation's capabilities, check out our post on Using CloudFormation Modules to improve IAM Least Privilege over on our Contino Engineering Blog.
Note: This blog was originally published in February 2021 by Cloud Native Director James Strong and has been updated to include new information on CloudFormation Hooks.
