AWS Lambda and Elastic File Storage: Welcome to the Serverless Promised Land
At the dawn of cloud computing, we were given a visionary glimpse of a world where technology is on-demand, pay-per-use and ubiquitous.
The vision was gradually refined with the evolution of serverless computing. Now with AWS Lambda’s announcement of Elastic File Storage (EFS) support, we have finally set foot into the Serverless Promised Land, where the perfect convergence between compute and storage enables unprecedented flexibility in digital business solutions.
The Business Value of Serverless
A key value proposition of the cloud is the freedom to focus on solving business problems instead of infrastructure ones. Just as you would use electricity without having to build your own power plant.
In 2014, AWS launched Lambda, pioneering the concept of serverless computing and offering a glimpse of what the ubiquitous cloud could look like. It provided the ability to access granular compute that can be scaled up and down at millisecond intervals to address business needs on demand.
Over the years AWS has continuously improved Lambda (75 releases since 2014 to be exact) based on customer feedback: adding service integrations, more RAM, wider programming language support, faster startup within private networks, SLA of 99.95, PCI and SOC compliance, increased execution duration, reserved concurrencies etc.
However one request remained until now: attachment of file based storage.
Separation of Compute and Storage for Traditional Scalability
The challenge with attaching storage to Lambda is the fact that scalability fundamentally requires a separation of computing from storage (also known as stateless computing) where all states in the form of persistence, configurations and session management are either pushed up to the edge such as user browsers or mobile devices, or pushed down to a backend database.
So instead of attaching storage, Lambda functions take an event-driven approach.
It takes in an event input such as API gateway request or new file in S3, applies scaled compute, then outputs the results to a DynamoDB or back to S3. This one-way function (input → output) approach maintains the separation of compute from storage, and works well for most use cases.
Conventional stateless functional compute relies on separation of storage
Convergence of Compute and Storage for New Scalability Paradigms
However, not all workloads can be treated in an event-driven or functional way. Not all files are suited for storage in S3 or DynamoDB. Oftentimes you just need to attach some good old fashioned files storage and Elastic File Storage (EFS) as the AWS managed Network File System (NFS) offering, is the perfect candidate.
Lambda combined with EFS ushers in a new era where compute and storage can converge whilst remaining independently scalable side by side.
This opens up a whole new paradigm of serverless solutions that were previously impossible.
Instead of responding to event-based triggers, Lambda can now work on the file system shared with other compute services such Elastic Compute Cloud, or container-based services such as ECS and Fargate.
New convergent paradigm using different compute architectures sharing the same storage
Business Use Cases of Lambda and EFS
I sat down with Paul Stafford, AWS Senior GTM Specialist - Serverless, to go through some of the key business use cases of Lambda with EFS that we see in the market.
1. Serverless CMS for more Scalable Digital Experience
Customer journeys typically start and end at the digital experience layer, often supplemented by a Content Management System (CMS). For greater agility in responding to customer needs, there is a growing trend of digital enterprises evolving away from slow monolithic websites tightly coupled to legacy CMS, towards rapidly deployed micro-frontends or Single Page Apps (SPAs). These are deployed at edge via global Content Delivery Networks (CDNs) and only call back to a “headless CMS” via API for content enrichment, personalisation and A/B testing.
However, most headless CMS are still monoliths and suffer the same symptoms as legacy CMS around scalability, maintenance, upgrades and licensing etc. Which is why a new approach known as “serverless CMS” emerged. However the serverless CMS revolution was hamstrung by the lack of file storage right from the start, since S3 and databases are nowhere near the file I/O speeds that a serverless CMS needs.
My prediction is with EFS support, the serverless CMS ecosystem will begin to thrive and over the next five years go toe-to-toe with traditional CMS vendors.
Want to start innovating like a startup?
The serverless model is a game-changer.
In our introduction to serverless we take you through the advantages of AWS Lambda, how it integrates with other AWS services, key use cases and give you a tutorial for getting started.
2. Serverless Flat Files Transactions for Core Banking and Payments
Flat files still make the world go around, whether we like it or not. Despite the real-time nature of digital transactions, (S)FTP and batch jobs are abundant amongst financial institutions and even challenger/neo-banks. Lift under the hood of most core banking, payments and mainframe systems, and you’ll find a lot of flat file processing in all sorts of non-human readable formats designed for a bygone era such as SWIFT, FIX or CopyBooks.
But what if we replace the job of flat file processing, which is essentially transactional functions, with serverless compute? Interestingly enough, Capital One embarked on a Serverless Transaction Journey to Migrate Mainframe workloads back in 2017 with a very well thought out architecture.
Now, with the addition of EFS support, such architecture can be further simplified to remove the need for complex state machines coordinating S3 downloads or DynamoDB writes. EFS can be mounted in a Virtual Private Network (VPC) so it removes the compliance concerns that would apply to moving certain files to S3. By allowing existing jobs to continue to work with familiar file systems, it is a lot more feasible to migrate existing jobs into serverless functions without major re-architecting.
This makes the modern serverless flat file transactions an even more attainable reality!
3. Serverless Data Processing and Machine Learning
Every enterprise holds a wealth of data that needs to be processed in real-time to unlock valuable business insights that can be used to make business decisions and predictions.
EFS support opens up many opportunities for data processing by allowing Lambda to work with files and directories directly without having to go through the overhead of coordinating S3 access or using Step Lambdas, which is more tailored for workflow orchestration rather than data processing.
Once the data has been cleaned, processed and transformed, it is important to train Machine Learning (ML) models to infer and mine useful insights and predictions. Hosting and serving ML Models for inference can also become quite expensive with dedicated machines. It was possible to serve models in Lambda but involved either downloading from S3 or baking the model into Lambda layers.
Now with EFS, a trained model can simply be outputted to a EFS directory from SageMaker, and all Lambdas with the EFS mount will have immediate access to an updated model to serve it.
Summary
The above is by no means an exhaustive list and I didn’t go into some of the more infrastructure related use cases such as file backups. but hopefully will convince you of the game changing nature of this recent announcement. If you wish to find out how you can use Lambda with EFS to accelerate your digital experience, transaction processing or real time data, please reach out to me directly on mrserverless@contino.io.
Also do check out some of our relevant case studies below:
Transport for New South Wales: Enabling Serverless Real-time Visualisation and Forecasting
