Building a Stock Forecast Application in 2 Weeks Using Serverless AI and Responsive Design
“Prediction is very difficult, especially if it’s about the future.”
On 7th of November, Team Quantino (mashup between Quantitative and Contino) participated in the Melbourne Data Showdown 2018 event, hosted NAB at their fantastic venue over in The Village Docklands. Team Quantino demonstrated a stock forecasting application for predicting the stock price movements of all four major Australian banks over a period of two weeks, which we built in two weeks.
It was a lot of fun and we were quite surprised at how easy it was to create a responsive AI application in such a short period using AWS Serverless and SaaS APIs such as Anodot. We managed to achieve predictions up until November 21st with < 1% Symmetric Mean Absolute Percent Error (SMAPE). Our app is hosted at quantino.ai, slides below:
Hopefully, this article will also inspire you and your organisations to look into the many practical use cases for AI that are already emerging among leading enterprises. More importantly, it might demonstrate the possibility of using powerful high level AI services to solve business problems without necessarily having teams of data scientists on hand.
I would like to give a massive shout out to Quantino team members: Lucas Rafagnin, Ira Cohen (co-founder Anodot), Sami Raines and Raymond Au, without whom none of this would've been possible.
Part 1: Forecasting and AI
A brief history of forecasting techniques
The desire to foresee future events before they unfold has gripped our ancestors since the dawn of time. Ancient civilisations resorted to various forms of divination and primitive record keeping such as Oracle Bones and Quipu knots to cater for forecasting needs ranging from weather and farming to commerce and military conquest.
Fast forward a few millennia and data collection and record keeping has increased in sophistication from bones, bamboo, stones, paper to digital storage and display. In the 70s Box and Jenkins created the ARIMA mainframe algorithm while in the 80s the PC revolution saw Excel spreadsheets becoming popular among traders and quantitative funds. Successful investment algorithms, however, were still quite complex and very much a 'dark art'.
In more recent times, with the major breakthrough in AI in the form of Deep Learning (machine learning inspired by the structure of the human brain), open source frameworks such as Tensorflow, and open communities such as Kaggle, it has become possible for anyone to easily learn, build and share powerful predictive models.
Combining these breakthroughs with cloud computing, which allows us to train our predictive models with unprecedented amounts of data, and the predictive power at our fingertips is greater than ever.
But is it really possible for AI to predict the Stock Market and get rich?
The answer is yes.
Whilst AI is not a crystal ball (yet), it is very capable of making forecasts and decisions using powerful quantitative algorithms given the right data inputs. The process is actually quite similar to that of a human analyst or investor.
Ray Dalio, founder of the world's largest hedge fund Bridgewater Associates, is known to come up with portfolio ideas based on old newspaper articles. He sees the economy is like a machine and uses information (input X), applies his intelligence and experience, to build up repeatable investment templates (output Y). The formula looks something like:
Y = Brain(X) + Deep Experience
If we were to swap in the AI equivalent, the formula would become:
Y = Neural Network(X) + Deep Learning
Now considering both the exponential processing power of the machine, the amount of multi-faceted data it can consume and ability to learn, the question is no longer if AI can replace humans in financial forecasting, but when they will do so.
For some, the when is now. AI Pioneer Kai-Fu Lee indicated he already gained 8x return using quantitative systems compared to private bankers. He also predicts that research analysts will be completely replaced by AI in 15 years. Goldman Sachs has famously replaced 600 of its traders with 200 computer engineers, Uber applies AI-based financial forecasting, the RoboAdvisor 2.0 movement is gaining momentum...the list goes on.
Part 2 The Quantino.ai application
The goal was to build an application that would accurately predict the stock price movements of the four major Australian banks over a period of two weeks.
Responsive User Experience
In building our application, Responsive Design was our number one priority.
We want our users to visualise data and insights on their mobile devices, not having to load up traditional BI.
By using React.js and SPIN template Lucas was able to rapidly build out my mock design above into the quantino.ai app below.
Deep Learning Algorithm
The stock market ultimately consists of multiple dynamic time series. Based on our research, the forecasting task is quite suitable to be tackled with Recurrent Neural Network (RNN) alongside either Long Short Term Memory (LSTM) or Gated Recurrent Units (GRU). Our team tried out a few platforms and implementations:
- AWS SageMaker with DeepAR (a high level algorithm by AWS researchers that abstracts away LSTM and GRU)
- Google ML Engine with Tensorflow, RNN with LSTM.
Both are excellent fully managed ML platforms and it was a great learning experience trying out both DeepAR and standard LSTM. Unfortunately, in the end we simply didn't have the time to adequately fine tune our models/hyperparameters to a desirable state.
Thus, we had to seek out an alternative solution.
Anodot Beta Forecasting API
About a week prior to the event, I worked with Anodot's co-founder Ira on his latest beta forecasting API. Anodot is a general purpose AI company specialising in Anomaly Detection for any time series data. See below where the shaded areas are "normal" ranges constructed using 1-day-ahead predictions for Open, High and Closing prices:
With the new forecasting API, the predictions are extended further into the future (in our case 14 days in advance). Like the general purpose Anomaly Detection, the Forecast API will take any combination of related time series and automatically work out the best algorithms to use behind the scenes (in our case it ended up being a hybrid between RNN and Holt-Winter algorithm for seasonal trends).
Real Time Feeds
Twitter sentiment was an area of great interest to us considering a single tweet can either wipe out a company, have significant correlation to a company, or have significant correlation to an entire index (Dow Jones Industrial Average).
We built the necessary Lambda functions to extract tweets but again ran out of time to properly apply comprehension and scoring. However we did manage to set up daily stock feeds thanks to Alpha Vantage.
Serverless Architecture
To glue all of the above together, our standard Serverless Architecture consisted of PCI compliant AWS Lambda for events processing, S3 for file storage and static hosting, and API Gateway to expose functionalities to the outside world.
The Result!
We stopped invoking the Anodot beta forecasting API after the 7th November event. Our 14 day day forecast ended on the 21st of November with the following results:
- ANZ: Actual $25.42, Predicted $25.56
- CBA: Actual $70.06, Predicted $68.22
- NAB: Actual $24.01, Predicted $25.24
- WBC: Actual $25.59, Predicted $26.54
The SMAPE as of 7th of November was less than 1%. Pretty good results for two weeks of work! Based on possibly more than two decades of research from some really smart people.
Part 3 How does all this effect you?
If you take away one thing from this article, it's this:
AI is about human beings and the ability to solve real world problems
I strongly believe that, in time, AI will reduce repetitive and mundane work, allowing us to focus on more meaningful pursuits in life. I would equally encourage you not to believe incorrect depictions of robotic overlords or mystic arts practised by academic data scientists. If we can build the above in two weeks, imagine what you can do with a dedicated team and resources!
Some real-life AI examples outside of the financial industry our team have already witnessed/assisted with:
- Energy companies predicting gas well failure and maintenance
- Transport and logistics management, self-driving cars
- Government agencies using facial recognition for licensing and customs processing (this already happens in many countries with E-Passports)
- Businesses dealing with real time anomalies around fraudulent activity, incorrect pricing and sudden spikes in demand.
Please reach out if you can have any questions or queries. We are living in exciting times indeed and I am more than happy to assist in the interesting journeys ahead.
