Using DevOps to Deliver Your Electronic Trading Platform More Efficiently
I co-founded Contino after working as a developer on electronic trading systems for investment banks, and observing how hard it was to iterate on these systems quickly and safely.
The development teams that I worked on would develop features relatively quickly, but the cycles of testing and deployments were long and arduous. In the worst cases, code would take up to 12 weeks to be deployed after it left my fingers as the developer, with multiple iterations between developers and testers, and much pain as the code moved from development into production deployment.
Getting code out of the door felt like a marathon, and full of risk. The deployment scripts were almost as complex as the application, putting different components and configuration files onto different servers. We would carry out multiple rehearsals of the deployments, and find lots of issues on each cycle. The deployments were carried out by big teams out of hours, and required lots of people all over the globe to be on standby to carry out their pieces or be available for escalation. It all felt as though we were walking a tight-rope.
Like many industries in 2017, investment banking is faced with the challenge of needing to innovate in a more challenging and cost sensitive environment.
The systems themselves were big and complex, with connections to exchanges around the globe, trading thousands of instruments of various kinds through manual trading, FIX, or algorithms. There would also be integrations with upstream and downstream systems which combine and to carry out the trade lifecycle. 24x6 uptimes with high requirements for stability and low latency were also standard.
The need for quality and safety was also very high. Working in the exchange traded derivative and high frequency trading markets, the potential for huge cash losses was very real if a system fell over or a subtle bug crept in which impacted trading.
DevOps Techniques and Tools Have Huge Potential To Improve This!
Over the years working with our clients at Contino, I have seen DevOps emerge and mature, bringing with it all of the techniques required to really improve the delivery of the kind of electronic trading systems that I used to work on.
Some of the specific techniques that spring to mind include the following:
Configuration Management
In trading systems, configuration is critically important. All of the user details and connections details for various middleware and systems are stored in configuration files, which need to end up on the right boxes, pointing at the right endpoints. As we move from dev to test to production, we need to switching in different configuration values, which becomes a key source of risk on the path to production.
DevOps Solution: Configuration management tools such as Puppet, CHEF, or Ansible and the more modern container space are ideal for automating this processes and adding repeatability, safety and controls to the configuration promotion lifecycle. This is much, much safer than achieving the same with custom shell scripts.
Data Agility
Likewise, trading applications are very data centric. There will be lots of data about instruments, trades, accounts and mappings stored in databases which impact the trade lifecycle. These data changes need to be propagated from development to test to production, and tested as carefully as any code or configuration change.
DevOps Solution: DevOps-related tooling such as Delphix or database lifecycle management will help us to bring more agility into the data space, snapshotting clones, roll forward, roll back, or moving DDL/DML changes through a defined and rigorous software development lifecycle.
Latency Reduction
Systems in the trading space are always latency sensitive. Whether you are at the micro second or second level, you want to execute faster as this is a key reason why clients choose to trade with you.
DevOps Solution: DevOps techniques will help us to deliver systems which scale better. If we have a modern infrastructure platform, we should be able to instantly stand up new nodes which scale in and out elastically, dynamically discovering each other in response to changing load conditions.
Security
Obviously trading systems need to demonstrate high degrees of security from both external hackers and an internal controls point of view. The cost, repetitional, and regulatory risks of security incidents in the trading world are incredibly high.
DevOps Solution: The sub field of DevSecOps brings in lots of thinking such as code scanning, controlled deployment pipelines, and audit and controls in the systems and middleware that you work with. Docker containers are also very handy tool in this arsenal, where we can isolate processes and sandbox what they can achieve.
Controls and Compliance
Investment banks under stacks of regulatory constraints to show that systems are being modified in an audited and controlled way. They need to show that every change is access controlled and audited and tight permissioning and approvals are in place.
DevOps Solution: DevOps techniques add in these controls, checks and balances, and actually help you to lock down your access whilst still enabling your teams to go faster. A DevOps mature team for instance, should never even need to log into development, instead defining infrastructure as code using virtual machines on their desktop.
Artefact Management
Most of the teams that I worked on had very fragile artefact management. We would use Nexus or Artifactory, but the versioning would be glued together with scripts. We would have little confidence that development, test and production environments would be the same, running exactly the same versions and binaries.
DevOps Solution: A tight CI/CD pipeline integrated with automation tools and containers has the potential to manage all of this. We should be able to confirm that al environments are equivalent down to the binary level.
Monitoring and Metrics
The complexity and resiliency requirements of trading applications are very high. Teams need to be informed very quickly of issues with the application or infrastructure, and potentially need quite sophisticated analytics to identify when things are going wrong whilst also minimising mean time to recovery.
DevOps Solution: From day one, DevOps has recognised the role of monitoring in supporting safer iteration of systems. It brings in lots of techniques for improving monitoring and metrics capture. DevOps also has a lot to say about cycle time metrics and metrics such as mean time to recovery, which will support continuous improvement of the team.
Why Do This?
Like many industries in 2017, investment banking is faced with the challenge of needing to innovate in a more challenging and cost sensitive environment. We believe that by implementing the specific changes and techniques above, your teams will be able to deliver faster and work more efficiently, whilst also increasing the levels of controls and governance in the environments. This combination of speed + control is a win/win, which all people responsible for legacy or greenfield trading systems need to look at very seriously.
