Marcus Maxwell

1 December 2020

20 DevSecOps Best Practices Across People, Process and Technology

How can secure software be developed at speed and scale!?

This is the ultimate IT paradox for modern global enterprise organisations: go faster and innovate. But always stay secure.

DevSecOps is the answer to integrating these seemingly contradictory enterprise challenges into a coherent and effective approach to software delivery.

In this article, we’ll cover:

• What is DevSecOps?
• What are the benefits of DevSecOps?
• DevSecOps best practices across:
  • People
  • Process
  • Technology.

What Is DevSecOps?

DevSecOps is the integration of security into DevOps practices. Taking a DevSecOps approach, security issues can be identified early in the development process rather than after a product is released.

How does it work?

Testing, monitoring and reporting are codified and embedded in the continuous delivery pipeline and fast feedback loops are then generated regarding the state of your infrastructure security, across your system.

Essentially, all the governance standards of your organisation can be ‘hardened’ into your infrastructure via code before you ever d

eploy applications onto it.

What Are the Benefits of DevSecOps?

There are many benefits of including security at every stage of the software delivery lifecycle. We’ve listed the key ones out below:

• Cost reduction is achieved by detecting and fixing security issues during the development phases.
• Speed of delivery is increased as security bottlenecks are minimised or eliminated.
• Speed of recovery is enhanced in the case of a security incident by utilising templates and pet/cattle methodology.
• Enhanced monitoring and auditing leads to improved threat hunting, which reduces the likelihood of a breach, avoiding bad publicity and reputational damage (to say nothing of regulator fines).
• Immutable infrastructure allows companies to tear down infrastructure while managing an attack vector identified by scanning. If a node is compromised, it won’t remain compromised for long, as it will be torn down and rebuilt with new credentials. Zero defects in the code is the ideal to aim for, although zero variations are the minimum requirement.
• Immutable infrastructure improves overall security by reducing vulnerabilities, and increasing code coverage and automation. It also encourages companies to move to the cloud instead of using depreciating and increasingly vulnerable hardware.
• Security auditing, monitoring, and notification systems are managed and deployed so that they can be continuously enhanced, to keep in step with the frantic innovation intrinsic to cybercrime.
• Ensures the ‘secure by design’ principle by using automated security review of code, automated application security testing, educating, and empowering developers to use secure design patterns.
• Creates targeted customer value through secure iterative innovation at speed and scale.
• Security is federated and becomes the responsibility of everyone, not just a specialised team, or even individual.
• DevSecOps fosters a culture of openness and transparency from the earliest stages of development.
• Increased sales as it is much easier to sell a demonstrably secure product.

In summary:

By taking a DevSecOps approach, the cost of complying with regulation and governance standards is reduced overall and the speed of software delivery is increased. Simultaneously, greater transparency enables superior threat hunting across the board and much more flexible reaction and recovery times. Fundamentally though, DevSecOps helps enterprises to innovate securely at speed and scale.

So what are the ingredients of a successful DevSecOps transformation? It’s time to take a closer look at the DevSecOps best practices across three key pillars: people, process and technology.

DevSecOps Best Practices

DevSecOps Best Practices: People

No matter how many technologies you decide to implement, the weakest link of that chain will always be the human factor, and this must be the starting point for any DevSecOps implementation.

One of the most important aspects of DevSecOps is challenging the way traditional security teams integrate with the wider business. Changing habits and raising awareness across all levels of a company are not easy tasks and require a top-down approach if attitudes are to change.

Let’s dive into some specific practices you can use when designing the people component of your transformation.

1. Breaking Down Barriers and Silos with Security Champions

For security to be effective, we need to include security concerns - and the security ‘mindset’ - as early as possible in the software delivery pipeline.

One way of doing is this is with security champions.

Security champions are members of a team that help to make decisions about when and how to address security concerns. Security champions act as the ‘voice’ of security for a given product or team, and they assist in the triage of security bugs for their team or area. They are evangelists for the security mindset, obsessively expounding on the importance of security across all areas!

Some of the most important duties of the security champion include the following:

• Emphasize security concerns across all teams - not just the ‘Security Team’
• Evangelize the ‘security mindset’
• Ensure that security is not a blocker on active development or reviews
• Empowered to make decisions
• Work with AppSec team on mitigations strategies
• Help with QA and Testing
• Write Tests (from Unit Tests to Integration tests)
• Help with development of CI (Continuous Integration) environments.

2. Training and Upskilling Your Staff

Any successful DevSecOps program will invest in good training and professional development for its staff.

Training must be rooted in company goals, policies, and standards for software security, and learning media must be flexible and tailored. To foster and develop good security staff, organizations must provide new hires with the appropriate training and tools they need to do their jobs well, and to contribute to the successful release of secure software.

Engaging specialist security and DevOps training organization(s) to raise staff skills and awareness are essential for maintaining consumer trust. Good training ensures that standards are implemented correctly.

3. Culture is Everything

Simply having the proper DevSecOps processes and technologies will not be enough to achieve anything if the company culture – embedded in people across all areas of the business – does not enable those processes and technologies to be properly utilized.

The security team has traditionally been a drag on release performance. They become the ‘Department of “No”’ and, as a result, are marginalized over time, creating a self-reinforcing downward spiral of division between teams. DevSecOps aims to break down these barriers and stop security from being its own echo chamber without taking into consideration the wider business when implementing policies or tooling.

When DevSecOps is fully embraced there is no longer a single ‘Security Team’ but a constantly improving security mindset across the business.

DevSecOps Best Practices: Process

The word is sometimes spoken in hushed tones so as not to upset any engineers or developers: process.

But process doesn’t need to be a four letter word. It is a crucial enabler of the most challenging aspect of any DevSecOps transformation: your people.

As W. Edwards Deming (considered the grandfather of quality) once said: “A bad process will beat a good person every time.”

We strive to make sure that doesn’t happen. DevSecOps aims to align and implement common enterprise processes to facilitate cooperation and achieve more secure development processes.

Prior to implementing these processes, organizations would often respond too late and much too slowly to security issues. But when you step back, you realize that’s actually not surprising because, typically, processes are siloed within separate IT teams, which can lead to miscommunication, bottlenecks and, ultimately, delays. These bottlenecks and delays then increase your risk and will inevitably manifest as a lower bottom line.

DevSecOps, in contrast, makes it possible to create short, feedback-driven security loops that can quickly identify problems and react swiftly to them.

How does that work?

Let’s get back to our list of DevSecOps best practices–this time, it's all about process.

4. Integration of Processes

Integrating information security into agile development enables organizations to have a fully secure workstream through every single stage of the project development cycle.

In the agile world, the integration of security must start at the earliest possible stage, which in most cases is the ‘requirement definition’ stage. This methodology has been called ‘shifting security left’ and it strives to reduce the cost of implementing security.

5. Compliance

Implementing compliance doesn’t have to be a paper-based exercise. You can create metadata representing the compliance requirement and integrating it into your assets.

This can also be used by security policy automation by tagging assets that can implement the desired security architecture, for example, zoning. Imagine the ability to respond to a breach under the new GDPR rules in under 72 hours.

6. Version Control, Metadata, and Orchestration

Within an automated world, the only constant is change, and change needs to be both consistent and traceable. To track all changes, you must ensure that adequate and immutable versioning is in place.

To allow for quick recovery, every action needs a version, so that it can be managed in the same way that code is. Once turned into metadata, operations teams can efficiently track a change and measure it.

Orchestration software doesn’t only provide a repeatable way to deploy infrastructure, it also provides a huge amount of metadata regarding any task. This metadata can not only be used by the orchestration software itself, but as an authoritative source for integrated tooling. Once coupled with versioning, orchestration software becomes a powerful source of information for all operational teams.

7. Security Tooling in CI/CD

Security has fought against shadow IT for a while, although it created its own shadow IT by having separate tooling for security.

Wouldn’t it make more sense to let the operations teams run the security tooling as part of their pipeline?

If you take Vulnerability Management and hook it to your pipeline via APIs, you can then let the orchestration call them for every build. Security sets the requirements, then DevOps teams manage the frequency of scan occurrences according to the development practices.

8. Incident Management

Responding to security incidents should not be an improvised or non-scripted activity. Workflows and action plans should be created in advance to ensure the response to an incident is consistent, repeatable, and measurable.

In a DevSecOps world, proactive and preemptive threat hunting, as well as continuous detection and response to threats and vulnerabilities, means that there are fewer major incidents and more mitigations.

9. Red Teams, Blue Teams and Bug Bounties

The use of red teams, blue teams and bug bounties also mitigate against breaches. The purpose of red teams is to test the effectiveness of security programs. Blue teams defend against the red team’s attacks.

All companies should deploy a red team to hunt for threats as part of the DevSecOps methodology. Red teams are built from security team personnel and are usually virtual to facilitate their ad hoc nature. Instead of discussing what is wrong with an application, the red team demonstrates what is wrong and provides the solution.

All companies should have a clear process for security researchers to disclose vulnerabilities. Otherwise, many do not get reported for fear of legal repercussions. It is also important that this be a secure method of communication as some countries have laws that would still put the individual who disclosed the information at risk if the vulnerability is disclosed in a way that could be intercepted (e.g. by email). Publishing a PGP (Pretty Good Privacy) key along with the method of communication gives you the best hope of being informed of current vulnerabilities -- hopefully before they are exploited.

All companies should also, occasionally, implement bug bounty programs -- rewards given for finding and reporting a bug in a software product.

DevSecOps Best Practices: Technology

Technology and tools tend to make people giddy. It’s the shiny new toy that everyone wants to get their hands on.

And because of that, people may sometimes be a little too eager to use the latest and greatest thing they heard about at the last conference, even if it’s not in the company’s best interest.

That’s why figuring out the why behind the project is important. Picking the right tools should be a function of the process and not dictate the process.

Once you are in position to pick the right tools for the job, you’re one step further along the DevSecOps journey, because those tools will enable your people to execute a proper DevSecOps process.

We’ve listed out some of the required technologies to implement a successful DevSecOps methodology within your enterprise below.

10. Automation and Configuration Management

Leveraging automation and using orchestration is key to DevSecOps success. Orchestration and automation make auditing easier through the use of metadata, which makes decisions easier to achieve as they are based on data points and repeatable processes.

11. Secure Coding Practices/Security as Code

All coding standards must be constantly checked against new security recommendations. Setting this up to be event-driven is an excellent way of finding vulnerabilities as quickly as possible (there is a big difference between finding an issue on day one as compared to day zero!).

All changes to the code need to be verified and tested against these recommendations: no change is too small during this process. This is not a trivial exercise, and the benefits associated with such practices should not be underestimated, they are not limited to the amount of changes occurring in the development lifecycle.

12. Host Hardening

The practice of host hardening is not new, but if it were used more often, fewer services and applications would be unnecessarily exposed to the internet. Countless examples of security incidents can be directly related to leaving a generic attack surface that allows automated attack tooling to succeed in even the most basic attacks.

Minimizing the attack surface by not installing or running anything that is not required for the core application and utilizing security features native to your OS (e.g. kernel security modules in Linux) make this task easier.

The Center of Internet Security has developed a set of industry-standard benchmarks for infrastructure hardening.

13. CI/CD for Patching

Once your metadata has been associated with each asset, we can use this data to implement patching at the CI/CD level. Feeds from threat intelligence and vulnerability management are compared to the deployed software stack to identify matches in the templates that are queued for deployment. Patching live systems becomes a thing of the past, thus limiting the impact of downtime. This will also provide the ability to determine risk exposure in near real time. 14.

14. Application-level Auditing and Scanning

Auditing and scanning are a crucial aspect of DevSecOps that allows business to fully understand their risk posture. Each of the following solutions represent a higher degree of security assurance of the code, as reflected in the organization’s risk appetite.

15. Source Code Scanning

Source code scanning should be covered by implementing Static Application Security Testing (SAST). SAST is used for scanning the source code repository, usually the master branch, identifying vulnerabilities and performing software composition analysis. It can be integrated into existing CI/CD processes.

16. Dynamic Application Scanning Tool (DAST)

Dynamic Application Scanning Tools are designed to scan staging and production websites in running state, analyze input fields, forms, and numerous aspects of the web application against vulnerabilities. It’s important to recognize that any time you allow users to provide you with data (form fields, query strings, HTTP headers, etc.), you are allowing them to provide data that your web server or application code will have to deal with.

17. Pre-Deployment Auditing

Pre-deployment auditing uses a pre-defined template for building assets to ensure the desired internally certified security level. The check is event-driven: when target code is changed, a check is triggered. Validations should be blocked and required to be integrated into a CD pipeline at this stage, since this is the last opportunity before the exit.

This principle can be applied to infrastructure-as-code to enhance compliance by ensuring that not only your software, but the infrastructure you deploy it on is automatically compliant. Tools like terraform-compliance or HashiCorp Sentinel are effective here.

This kind of auditing also has the advantage of engaging security teams early in the software development pipeline, rather than announcing their requirements at the end.

18. Post-Deployment Auditing

Post-deployment auditing, compared to pre-deployment, is also event-driven, but the events that trigger checks include changes to policy, as well as code. So when either the infrastructure, or the standards (policies) that that infrastructure must meet, change, a check is triggered.

The idea behind Post-Deployment Auditing is to ensure that the certified security level which you achieved with Pre-Deployment Auditing is still applicable and valid. That’s why the number of Post-Deployment tests usually exceeds Pre-Deployment tests.

19. Automated Host/Container/External Vulnerability Scanning

Compliance can be achieved using automated security configuration assessments to reduce risks and maintain continuous compliance. This helps to cut compliance costs by reducing the effort and time required to assess the systems.

Hosts: run scheduled checks to determined if the host configuration is invalid or outdated. Some tools give you a ‘security score’. You can easily see how your security improves over time if it is integrated with monitoring systems.

Containers: monitor every call/command that your container makes. Anything suspicious or non-compliant is blocked.

External Vulnerabilities: a check that is run periodically against your infrastructure (updated continuously with the latest threats) to identify vulnerabilities.

20. Secrets Management

‘Secrets’ in an information security environment include all the private information a team should know, for example, database credentials, or a third-party API. Secrets should be accessed or generated temporarily with specific authentication mechanisms that are different for each environment such that no one – not even the developers – can reverse the logic or exploit a backdoor around secrets by just having access to source code.

The main purpose of managing these secrets is to eliminate (or at least minimize) the potential for human error in the handling of such private information, e.g. losing it, accidentally posting it in a public forum such as GitHub etc. The ideal technique is a synchronized, encrypted, auto-generated secrets store in which entities are temporary, with as short a time-to-live (TTL) as possible.

Want to dive deeper into DevSecOps?

Check out our white paper for a complete guide to DevSecOps: https://www.contino.io/resources/devsecops-best-practice-guide

And if you'd like to see what a hands-on DevSecOps approach looks like in practice, take a look at this video which showcases the work Contino delivered at Allianz, the world's largest insurance company:​ https://www.youtube.com/watch?v=CS_ztJOoeHc&t=69s ​