High-Performance Computing for Financial Services
During the last week of July, I attended High-Performance Compute (HPC) Training for Financial Services in AWS. During this week-long training, AWS professionals in Financial Services shared their outlook on the needs that large financial institutions face from the industry, regulators, and markets.
The first thing that caught my attention was this question:
“If you could add 100,000 cores to your HPC grid, return them once you're done and pay only for what you use, just like many of your peers do, how would that improve your business?”
In a few words, HPC allows organizations to process data and perform complex calculations at high speeds. The cloud allows you to have access to almost infinite processing power.
That is a powerful question. With powerful implications. With that ability, your IT department could speed up Quantitative Analysts, Quants, ability to make informed decisions. With that capability, you could also increase the performance of your grid compute while lowering costs and maintaining compliance in order to compete effectively. What CIO or CEO doesn't want that? Here are some examples of doing just that, albeit with 1,000,000 cores.
- Western Digital, 1m vCPU cluster
- AWS and Tibco, 1.3m vCPU on Spot
- Get into the Top 500 for just $5k
“Regardless of whether 2022 is a deadline for reporting only or full capital implementation, banks must have FRTB infrastructure in place and models approved by supervisors by then.” - Bloomberg Professional Services
With the Fundamental Review of the Trading Book (FRTB) being pushed to 2022, this gives banks more time to understand the impact on their risk models and the shift to expected shortfall calculations. More and more pressure is coming from regulators for reform and reporting, FRTB will not be the end. This also adds additional pressure from IT to save more while also increasing capacity.
Running your HPC grid on AWS can be the answer to both regulator and IT pressures. However, with that comes with its own risk, that you can read about here, here, and here. It is understandable that financial institutions are hesitant to move to the cloud. But the need is greater than ever.
With a partner like Contino and the AWS Well Architected framework, that risk is reduced. Contino believes in forming a partnership with your cloud team, through workshops, upskilling, and Cloud Education Days. Our team of highly skilled engineers will work with yours to get them up to speed on cloud migrations and the AWS Well-Architected Framework.
The AWS Well-Architected Framework can help you realize the following:
- Operational Excellence - Scalable resources for repositories, build pipelines as code, and metrics for all levels of operations.
- Security - Managed and secure access to data sources via AWS IAM, encryption at rest and In transit.
- Reliability - Leveraging capabilities like autoscaling and services like S3, you be sure that your solutions are always available when needed.
- Performance - There are many options to run your workload and with AWS you can experiment to find the right one for the performance your workload needs.
- Cost Optimization - Spot instances for computing can reduce costs up to 90%, You can track and analyze spend to ensure services are scaled and being utilized properly.
Also included in the Well Architected Framework is AWS HPC Lens, it covers common HPC scenarios and identifies key elements to ensure your workloads are architected according to AWS’ best practices.
AWS has lots of example architectures in their Architecture Center, below is an example of how to run an HPC Grid on AWS securely.
https://media.amazonwebservices.com/architecturecenter/AWS_ac_ra_financialgrid_12.pdf
With AWS, you can allocate compute capacity on-demand without upfront planning of data center, network, and server infrastructure. This is a huge advantage when compared to on-prem grid setups. Below are just some of the options that can be used for an AWS powered HPC:
Parallel Cluster - With a couple lines of YAML you can have an HPC grid up and running in minutes. Parallel Cluster supports these schedulers; AWS Batch, SGE, Torque, and Slurm, to customize the needs of your firm’s algorithms. Submit a job and let Autoscaling take care of the number of hosts needed to run it, you only pay for what is needed and save massively on the cost with spot instances.
AWS Batch - AWS Batch enables developers and Quants to easily and efficiently execute hundreds of thousands of batch jobs on AWS. AWS Batch dynamically provisions the optimal quantity and type of compute resources based on the volume and specific resource requirements of the batch jobs submitted.
Lambda - No servers, no maintenance needed, just code. Lambda will execute what you need and when you need it. Couple this with streaming data from Kinesis and you can analyze data in near real-time.
EC2 Spot - With spot instances you could save up to 90% of your compute cost. Spot instances allow you to use AWS spare capacity; in return for that you pay less than the on-demand price. Launch a cluster with EC2 spot instances and watch the savings roll in.
These are just a few of the AWS technologies that can be used, for more information on how to get started with a Proof of Concept here is our white paper on Transforming Grid Compute in Investment Banking: High Performance Grid Computing on AWS.
In 2017 Contino was awarded the AWS Financial Services Competency for our work with Allianz. With that momentum we continue to help other large financial institutions with their move to the cloud. Watch this recording to see how we moved a trading app and all dev and test data from on-premises into AWS in under ten minutes.
A Contino and AWS partnership can help to make sure you have the right HPC strategy for your business. Having a misalignment or lack of direction can cost millions to your organization. Specifically, Contino can help with the following HPC solutions:
- Financial Services - Real track of stocks and risk management
- Insurance Companies - Risk management and insurance calculations
- Media - Render special effects and media streaming
- Healthcare Companies - Develop new drugs and cure diseases
- Banking Industry - Detect credit card fraud
More Resources
- https://www.contino.io/
- https://aws.amazon.com/financial-services/grid-computing/
- https://d1.awsstatic.com/Industries/Financial%20Services/Grid%20Computing/aws-financial-services-grid-computing.pdf
- https://github.com/aws/aws-parallelcluster
- https://docs.aws.amazon.com/parallelcluster/latest/ug/what-is-aws-parallelcluster.html
- https://www.bloomberg.com/professional/blog/2022-market-risk-odyssey/
