A Model for Scaling Terraform Workflows in a Large, Complex Organization
It's easy to run HashiCorp Terraform on your laptop and go wild managing your infrastructure.
But what do you do when you start working on more complex arrangements in a large, complex organization? At that point, the problem is not scaling infrastructure, but scaling your working practices to support a large landscape of teams and applications as well as scaling your workflows to enhance collaboration within your team and between teams.
In this blog, we will look at how to achieve this with a model we used at a recent engagement for a large enterprise organization to manage around 200 developers across multiple teams.
Terraservices
A Terraservice is an opinionated Terraform module that provides an architectural building block. A Terraservice encapsulates how your organization wants to build a particular component.
Nicki Watt coined the term “Terraservices” at HashiDays London 2017. This view of Terraform modules is key to scaling.
For example, this is a Terraservice to build a Web Application Service on Azure.
module "frontend" { source = "git::github.com/contino/module-webapp?ref=master" product = "${var.product}-frontend" location = "${var.location}" env = "${var.env}" app_settings = { REDIS_HOST = "${module.redis-cache.host_name}" RECIPE_BACKEND_URL = "${module.recipe-backend.url}" WEBSITE_NODE_DEFAULT_VERSION = "8.8.0" } }Copy
This Terraservice encapsulates how the Web Application should be configured. Scaling parameters, logging, alerting and sane defaults are set inside the Terraservice. The Terraservice exposes an interface that doesn’t require the consumer to know too much about the details of the underlying infrastructure and is opinionated towards productivity.
In this instance, the Terraservice is consumed by an application development team, to build out their product stack.
Terraservices have their own pipelines
A Terraservice has it’s own CI/CD pipeline. The pipeline runs unit, integration and compliance tests. It can even autogenerate documentation.
The integration and compliance tests run in a sandbox environment, and spin up their own test harnesses to exercise changes the Terraservice.
At the end of the pipeline, when tests are green, the Terraservice is versioned and tagged.
This ensures that a consumer picks a specific published version or stays on master to be on the bleeding edge. Keeping master stable and use Trunk Based Development practices.
Terraservices and Terraform Projects
Terraservices are used to compose infrastructure in Terraform projects. The Terraform projects each have their own pipelines. Terraform projects use remote state to share outputs, which are consumed by other Terraform projects. This is how it all flows together.
Click here for the original diagram.
The Terraform project pipelines build the actual infrastructure. They are decoupled from the application pipelines intentionally. The pipelines create empty environments that code can be deployed to. The master branch of a Terraform project reflects the current state of infrastructure. Each environment is built the same way. Environment-specific variables can be tuned to change the behaviour, but once a change is committed to master, the pipelines apply the change to all environments as long as each stage in the pipeline stays green.
See a larger, higher-quality version of the image here.
Pick the Right Architecture
There is a delicate balance to achieve and there is no right answer. Pay attention to the automation pain points. Some infrastructure components can take a long time to provision. Split these out into their own Terraform projects so that they can be pre-provisioned, to be consumed by other downstream Terraform projects.
Keep core network and components that deal with core security away from Terraform projects that deal with application stacks. The application infrastructure should have dependencies on core components and use what has already been created but not be able to change it.
The infrastructure should be architected in a way that an application's infrastructure can be destroyed without affecting other applications or core components.
Pick the Right Team Structure
A crucial part of this is how the teams are organized and how they use Terraservices. A platform engineering team is needed to shepherd Terraservices and continuously drive the right ways of working. This team has to look across the organizational landscape and define how the Terraservices are built, ensure they help teams build their applications quickly and provide enough so that teams don’t have the need to build their own infrastructure differently.
The Terraform projects that drive the application stacks are under the control of the application teams. This means that they can iterate rapidly, and use components from an approved library of Terraservices to compose their infrastructure.
The Terraservices themselves should be maintained by a platform engineering team, but can accept pull requests from anyone in the organization, using an internal OSS model. The role of this platform engineering team is to provide enough tooling for teams to self-serve.
Will It Work for Us?
This model provides a way for large organizations to allow individual product teams to build their own infrastructure, whilst providing control over how they are built.
What this doesn’t cover is a scenario in which a team decides to use Terraform primitives instead of the available Terraservices. This can be solved via peer reviews and the use of a Janitor monkey, which ensures that only components built the right way are allowed to exist.
In addition, Terraservices provide a happy path for teams to get up and running quickly, and are less likely to build their own.
Note that it is the Terraform Enterprise Module Registry that gives organizations a workflow to build, publish, and share their customized, validated, and versioned modules through a private service catalog. You can get more info about it here.
Further reading
- Evolving Your Infrastructure with Terraform
- How to use Terraform as a team
- Terraform Recommended Practices
