Cloud Automation: What It Is, Use Cases and Benefits
The companies most committed to digital innovation—Amazon, Netflix, Facebook—all use the public cloud.
And many enterprises are following in their footsteps, with 77% reportedly using the public cloud in some capacity according to our recent research in The State of Public Cloud in the Enterprise.
Why?
Because the public cloud effectively allows you to bypass years of legacy technology, process, thinking and culture. And this freedom from legacy is fertile ground for genuine digital innovation.
One of the cornerstones of this fertility is cloud automation. Less manual work, means more time spent executing great ideas!
Yet, the building of sophisticated cloud automation and the overall management of these systems across separate teams and functions is something we often find lacking, even in sophisticated enterprise environments.
The State of the Public Cloud in the Enterprise: Contino Research Report 2020
Why is the public cloud the greatest enabler in a generation?
We asked 250 IT decision-makers at enterprise companies about the state of the public cloud in their organisation.
So what is cloud automation and how can you utilise it to take full advantage of the cloud’s potential?
In this article, we walk through everything you need to know about cloud automation including cloud automation use cases, the differences between cloud automation and cloud orchestration and why it’s critical for DevOps.
What Is Cloud Automation?
Cloud automation is the use of automated tools and processes to execute workflows in a cloud environment that would otherwise have to be performed manually by your engineers, like configuring servers or setting up a network.
It enables you to take advantage of cloud resources efficiently and to avoid the security pitfalls that arise in contexts where teams rely too heavily on manual, error-prone workflows.
Cloud automation should therefore be a central component of your overall cloud strategy. Understanding what you can automate in the cloud, and which cloud automation tools can help you achieve the level of automation you need, is essential for leveraging the cloud effectively and at scale. Equally important are practices and tools associated with cloud orchestration, a domain that is related to but distinct from cloud automation (more on that later).
But first, what makes cloud automation different from regular automation in an on-premise IT environment?
Cloud Automation Vs On-Prem Automation
Cloud automation is not fundamentally different from automation in other types of contexts, such as on-premises.
Indeed, in some cases you can use the same automation tools in both the cloud and on-premises (although other automation tools work only with the cloud). If you’ve ever used disk imaging software to configure on-premises PCs automatically, for instance, or used monitoring tools on your local network to perform automatic restarts of servers when they crash, you’re already familiar with the principles behind cloud automation.
However, cloud automation is distinguishable by the following:
1. Cloud automation focuses on automating services and virtual infrastructure
The main difference between cloud automation and other types of automation lies in the types of services to which cloud automation applies.
Because cloud-based environments give users different levels of access to resources than they would have on-premises (for example, cloud environments don’t typically provide end-users with control over physical servers), cloud automation focuses more on automating services and virtual infrastructure than it does on physical devices.
2. Cloud automation is key to handling the scalability and complexity of cloud environments
You may be able to manage a half-dozen local servers by hand easily enough. But in the cloud, where there are dozens of different types of VM instances to choose from, and where it’s important to avoid running servers when they’re not necessary in order to avoid wasted costs, automation plays an especially crucial role.
To put this another way, the cloud is a prime candidate for automation, even more so than other types of IT environments.
Because cloud environments consist of an array of different types of services that can be scaled up and down constantly, automating the management of cloud resources is critical for getting the most value out of the cloud. If you attempt to manage your cloud by hand, you simply won't be able to take greatest advantage of the opportunities for scalability and agility that the cloud offers to your broader IT strategy.
Use Cases for Cloud Automation
Cloud automation can apply to a wide variety of workflows and tasks. We’ve outlined six key use cases below.
Cloud automation can apply to a wide variety of workflows and tasks. We’ve outlined six key use cases below.
Infrastructure provisioning
Probably the most obvious example of cloud automation is the use of cloud automation tools for infrastructure provisioning.
When you need to set up a collection of virtual servers, for example, it would take a long time to configure each one individually. Cloud automation tools like HashiCorp Terraform or AWS CloudFormation allow you to perform this task automatically by creating templates that define how each virtual server should be configured. The tools then apply the configurations for you.
You can take a similar approach to configuring other types of cloud resources, such as network setups and storage buckets or volumes. In general, any type of cloud automation tool that supports this type of infrastructure provisioning is known as an Infrastructure-as-Code, or IaC, tool.
Note, however, that IaC tools are not strictly limited to use in the cloud. Many are platform-agnostic and work with on-premises environments, too. On the other hand, certain IaC tools, especially those made available by cloud vendors themselves, typically work only with a specific public cloud.
By automating cloud infrastructure provisioning, organisations can scale their cloud infrastructure more quickly. In turn, they gain agility and an enhanced ability to innovate.
Identity provisioning and management
In large-scale cloud environments, a single company may have hundreds of different users, each requiring a different level of access to the various resources in the cloud. Setting up all of these access policies by hand would be a monumental task. Updating them as business needs change and users come and go from the organisation would be harder still.
Using cloud automation, identity management becomes much more efficient. You can use predefined Identity and Access Management (IAM) templates to set up user roles within your cloud environment.
You can also integrate your cloud IAM framework with a centralized enterprise directory service, like Microsoft Active Directory, to centralize identity management across your entire IT infrastructure, including on-premises resources as well as the cloud.
The automation of identity management within the cloud adds organizational agility by making it easier and more efficient to onboard new team members, modify the roles of existing ones and revoke access for employees who leave the company.
Application deployment
Application deployment, which refers to the process of moving a new application release or version from the environment where it was built and tested into the one where it will run in production, can be a time-consuming task if performed by hand.
It’s especially inefficient if you embrace the principles of DevOps and continuous delivery, which may entail pushing out a dozen or more new releases each week.
Cloud automation can help by automatically handling the application deployment process for you. Most modern CI/CD platforms, such as Jenkins, can automatically deploy applications into any major public cloud.
Public cloud vendors themselves also offer automated application deployment solutions, like Azure App Service and AWS CodeDeploy.
By automating application deployment in the cloud, development and IT teams achieve faster release cycles. By extension, they can push out new application features and fix bugs more quickly.
Monitoring and remediation
Once you have provisioned your cloud infrastructure, configured user credentials and deployed workloads, you need to monitor them and respond to incidents that may impact application performance. This is another juncture at which automation is very valuable.
Most public clouds offer built-in monitoring solutions, such as AWS CloudWatch, that automatically collect metrics from your cloud environment. They allow you to configure alerts that will be triggered when certain predefined thresholds are met, such as a cloud server running out of memory or a cloud database that has become unresponsive.
A variety of third-party vendors offer cloud monitoring solutions that allow you to do the same thing. Some also extend their functionality into the realm of automated remediation, which makes it possible to write predefined workflows that the tools automatically execute in response to certain conditions.
For instance, you could configure a workflow so that in the event that a virtual machine fails, another one will be automatically created based on an IaC template that you created ahead of time.
For companies with large-scale cloud environments, the ability to automate monitoring and take automatic steps to fix problems detected by monitoring systems leads to more stable and higher-performing clouds.
Multi-cloud management
Cloud automation is also becoming increasingly crucial in the context of multi-cloud architectures, in which companies use multiple public or private clouds at once.
Cloud automation tools play an important role in this type of environment by allowing teams to deploy workloads to multiple clouds at once and manage them from a central interface, rather than having to juggle disparate tools for each of the clouds they use.
For example, OneOps, a cloud management platform originally developed by Walmart that is now open source, can automate the deployment of applications to multiple public clouds. Monitoring and performance-optimization tools that work with multiple clouds also enable a type of multi-cloud automation.
For organizations with a multi-cloud strategy, being able to manage all of their clouds with a centralized, automated toolset adds crucial efficiency to their cloud strategy.
Data discovery and classification
Another use case for cloud automation -- one that is currently relatively rare, but likely to become increasingly important as more and more organizations face stricter compliance requirements from regulations like GDPR -- is the automated discovery and classification of data in the cloud.
Tools like AWS Macie can automatically scan cloud environments for data that may be sensitive in nature. They may also be able to identify situations where data is improperly secured; for instance, they could alert admins to an AWS S3 bucket that contains private address data and can be accessed by anyone on the Internet. Third-party data discovery and classification tools for the cloud are available as well, such as Open Raven.
Because discovering and classifying sensitive data by hand would require enormous time and effort, the automation of these processes enables much faster and more efficient protection of sensitive information. In turn, it helps organisations meet compliance goals.
Cloud Automation Benefits
Cloud automation offers a range of benefits:
- Time savings: By automating time-consuming tasks like infrastructure provisioning, cloud automation tools allow human engineers to focus on other activities that require higher levels of expertise and cannot be easily automated.
- Faster completion: Cloud automation enables tasks to be completed faster. An IaC tool can set up a hundred servers in minutes using predefined templates, for instance, whereas a human engineer might take several days to complete the same work.
- Lower risk of errors: When tasks are automated, the risk of human error or oversight virtually disappears. As long as you properly configure the rules and templates that drive your automation, you will end up with clean environments.
- Higher security: By a similar token, cloud automation reduces the risk that a mistake made by an engineer -- such as exposing to the public Internet an internal application that is intended only for internal use -- could lead to security vulnerabilities.
- Scalability: Cloud automation is essential for any team that works at scale. It may be possible to manage a small cloud environment -- one that consists of a few virtual machines and storage buckets, for example -- using manual workflows. But if you want to scale up to hundreds of server instances, terabytes of data and thousands of users, cloud automation becomes a must.
Put together, all of these benefits put businesses in a stronger position to build value. Instead of wasting time and resources managing cloud environments by hand, organisations that leverage cloud automation are able to focus their resources on activities that deliver direct business benefits, like developing new services and keeping customers pleased. They can also quickly deploy or modify their IT assets whenever necessary in order to support a new business initiative.
Cloud Automation and Mature DevOps
Cloud automation and DevOps are distinct concepts. Technically speaking, it’s possible to do one without the other.
In practice, however, cloud automation and DevOps typically go hand-in-hand. And if you want to reach DevOps maturity, cloud automation is an absolutely essential step.
Before we get into why that is, let’s remind ourselves of the critical role automation plays in DevOps.
DevOps Automation
DevOps places enormous emphasis on automation.
DevOps relies on practices including automated infrastructure-as-code, continuous delivery and tight feedback loops – all of which are dependent on automation.
Automation is critical not only for reducing the complexity and variability of your tech stack and infrastructure, but then subsequently scaling these across the business in a sustainable, repeatable fashion.
From a DevOps perspective then, automation (in general) focuses primarily on application development and delivery, which is distinct in most respects from cloud management.
So what about cloud automation?
DevOps and Cloud Automation
Because many application delivery pipelines feed into cloud-based production environments, being able to automate cloud management is crucial for building the type of reliable and efficient application delivery pipeline that DevOps prioritizes.
How so?
Cloud automation enables the following:
Continuous Improvement
Cloud automation can help provide the consistent feedback that is essential for achieving the continuous improvement goals associated with DevOps.
By automatically collecting and sharing data about your cloud environment, your team is in a better position to identify and act on opportunities to improve.
Self-Service Visibility
The templates associated with cloud automation tools provide a level of consistent visibility that benefits all members of the DevOps team.
For example, if a developer wants to know how a production environment is configured, a quick look at the IaC templates that govern that environment will yield the answer.
Because DevOps places a priority on communication and transparency across technical teams, this type of self-service visibility is highly valuable.
In Summary
Although cloud automation and DevOps automation each focus on different types of processes and resources, they reinforce each other in ways that make them inseparable.
That’s especially true for any team that wishes to put DevOps principles into practice at scale in a large, fast-moving cloud environment. Although, again, it may technically be possible to build a well-automated CI/CD pipeline without also using cloud automation tools, or to automate your cloud without also having an automated CI/CD process in place, doing so in practice is almost unimaginable.
So if you’re looking to build a more efficient DevOps/CloudOps pipeline, cloud automation is a great place to start.
Automations can be put in place to respond to identified needs or opportunities for optimization at some point (or points) in the pipeline. However, these automations may be disparate themselves or not coordinated from an overall perspective.
That’s where cloud orchestration comes in.
What Is Cloud Orchestration?
Cloud orchestration is automation for all your disparate automations, across separate services and separate clouds. It paints the picture of cloud automation services overall.
Yes, all your automations can be coordinated (and automated) from a higher level!
Cloud orchestration allows you to create an automation environment across the enterprise that coordinates more teams, functions, cloud services, security and compliance activities, for repeatable end-to-end automated processes–sky-rocketing productivity and throughput, and eliminating costly mistakes. It commonly outlines particular workflows by the series of steps involved, timelines if necessary, and tasks such as manual sign-offs if required.
Let’s take a look at some examples.
Example 1: Repeatable cloud test infrastructure
An example of cloud orchestration may be spinning up a fully functional test environment, running all software tests, then reporting and shutting down infrastructure upon completion.
When used for similar projects, this kind of cloud orchestration template can be efficient, repeatable, and save cloud resources, as it terminates all processes itself.
Example 2: Location-dependent security policies
Your business opens a new branch in a new country, with similar teams in place. You want to spin up a similar cloud environment to what your team uses at your main site, but with location-specific security policies.
You utilise the infrastructure-as-code architecture you used for your main site, but layer location-dependant security over the top, which can then be tweaked if you open a branch in another country.
Example 3: Triple redundancy systems
You have mission critical systems in the cloud. While you’ve built in redundancy into your AWS implementation, you’re still concerned about relying on one vendor’s platform completely.
You build a Microsoft Azure implementation that can be switched on automatically in the case of systems going down on AWS.
Why Cloud Orchestration?
Put simply, cloud orchestration brings together a series of lower-level automations, again through infrastructure as code, and for the enterprise environment it’s a must moving forward.
There are simply too many cloud automations to manage on a case-by-case or team-by-team basis. You need to get the bigger picture, and more importantly, you need to be able to effectively manage the bigger picture.
As with cloud automation, there are cloud orchestration tools to help you perform this complex task. Industry leader Terraform (which we’ve talked about previously) and AWS CloudFormation offer complete orchestration, with built-in support for common cloud automation services, and they also utilise the infrastructure-as-code paradigm.
As we’ve seen, cloud automation and cloud orchestration reinforce each other and often feature within the same conversations. So what’s the difference really?
Cloud Automation Vs. Cloud Orchestration
For most teams today, it makes sense to take advantage of cloud automation and cloud orchestration at the same time. However, these are distinct concepts that are driven primarily by different tools.
The key difference between cloud automation and cloud orchestration is that cloud automation focuses on automating individual types of processes. In contrast, cloud orchestration automates entire workflows, which are themselves composed of various individual processes.
Cloud Automation Use Case:
- A cloud automation process might allow you to install an operating system on a server.
- Another cloud automation process could configure the network for that server.
- A third could set up IAM policies that define who can log into the server.
Cloud Orchestration Use Case:
A cloud orchestration solution would combine these three distinct tasks into a single workflow that automates all aspects of the server’s setup.
In essence, then, you could think of cloud automation as a sub-category of cloud orchestration, or as a building block for it. You can do cloud automation without cloud orchestration, but you can’t have cloud orchestration without cloud automation.
Cloud Automation Tools
The market surrounding cloud automation tools is vast. At a high level, however, it can be broken into two categories:
1. Tools and Services Built Into Public Cloud Platforms
- Examples: AWS CloudFormation, Azure Resource Manager
- Advantage: These tools offer the highest level of integration with their respective platforms.
- Disadvantage: Their main drawback is that, in general, they support only the clouds of which they are a part. You can’t take an AWS CloudFormation template and directly apply it to an Azure environment, for example.
2. Tools From Independent Vendors
- Examples: HashiCorp Terraform, Puppet, Ansible, Chef and Salt. Most of these solutions are open source in their core form, although many of them serve as the basis for commercial editions, too.
- Advantage: In general, all of these solutions will work with any type of public, private or hybrid cloud platform.
- Disadvantage: These tools inherently have a lag in implementing functionality when a cloud provider introduces a new feature or product.
Conclusion
In summary, there are several key takeaways to bear in mind about cloud automation:
- It’s a must-have for any large-scale cloud environment.
- DevOps and cloud automation go hand-in-hand, and it’s very difficult to do one without the other at scale.
- Cloud orchestration relies on, but is different from, cloud automation. Cloud orchestration also remains a hazier concept.
- The main differences between cloud automation tools include whether they support only one or multiple clouds, and whether they are available in free and open source form or only as paid products.
Finally and most significantly, cloud automation is the only way to leverage the most value out of cloud environments.
By automating management tasks that would otherwise consume tremendous time and resources, cloud automation empowers organisations to update their cloud environments more quickly in response to business challenges.
In turn, it breeds a greater ability to react to changing business conditions (like the need for more or fewer virtual machine instances, or the addition of new users to a cloud application) by modifying IT configurations accordingly.
