6 Reasons Why Your Enterprise Needs a Data Platform
As an enterprise data advisor, I know that building a data platform is a key objective for many of my clients. But there’s no escaping the fact this is time consuming, so how do you know this is the right thing to do?
In this post I’ll explain what a data platform is and what problems it solves, and where it can accelerate your data exploitation.
What Is a Data Platform?
I recently told someone that I’m working on building a data platform for one of my clients. The response I got surprised me: “So what is that, anyway? A ‘data platform’ sounds a bit jargony—a bit hard to visualise”. Two things about this shook me. One, I hate buzzwords and jargon and I consciously try hard to explain what I mean in real words, so I was surprised to be caught out. And two, this comment was from someone I thought was an insider, a data guy, so I had felt safe with the odd insider term.
But this got me thinking: what do I mean when I say data platform? Let’s break this term down first.
What Do We Mean By Platform?
A platform is a set of capabilities which help build or run applications. Think the Apple App Store or Android App Store. They are platforms which enable people to build and deploy apps.
A platform handles the non-differentiated heavy lifting (I know this phrase is used by AWS, but I love it and it applies here too). This means no team that is building an app or exploring their data should have to worry about the tools, security, deployment pipeline, infrastructure. An engineering or analytics team can come and focus on building their business logic and adding value.
Typical elements of a platform:
- Developer environment
- Security built in - firewall, RBAC, federated IDP
- Development pipeline
- Source control
- API publishing
- Development templates, CLIs
Data Platform Capabilities
A data platform builds data capabilities on top of the standard platform. Typically, this covers connectors to different types of data source to help ingest data, some transformation capability, and somewhere to query the data and then visualise it. It could be single or multi-tenanted, meaning it serves one or multiple end user groups. Multi-tenanting tends to mean people can’t see each other’s data, but they can access the same tools and infrastructure, so benefiting from economies of scale. A public cloud provider such as Azure is mostly a multi-tenanted offering, for example.
Typical elements of a data platform:
- Data connectors
- Pipeline to transform and clean data
- Storage in a choice of databases or object storage
- Analytics tools such as Jupyter notebooks
- SQL access to data, whatever kind of data store it is in
- Data catalogue to enable data discovery
- Visualisation tools
- Data publishing
Enterprise Data Pain Points
Enterprises I work with are making the shift from data as a byproduct of running applications and recording transactions, towards data being the focus for analytics and future decision making. This involves handling A LOT more data, and in different ways from the traditional approaches.
Many seem to have some version of the same pain points when it comes to managing and leveraging their data:
Solving these challenges is important to make the most of the data and to stay competitive.
How Can an Enterprise Data Platform Help Me?
1. Reduce friction in accessing data
The greatest barrier to getting value from your data is getting hold of it in the first place. A data platform reduces the number of systems the end user needs to engage with. It simplifies the process of getting the data into the users’ hands, but it also enables new questions to be asked by joining data sets and enriching the original data.
A data platform is not a point solution like a data warehouse or data lake claims to be. This is much the same as the fact that moving your application stack to the cloud will not solve your application stack problems. However, running on the cloud will enable you to address those issues faster. Likewise, a data platform is the enabler here: it will allow you to spend more time on the value you can get from the data and less time on the infrastructure.
2. Start solving data problems, not integration problems
Example Use Case
A company has data distributed around a number of systems. Some systems feed out to a data warehouse where a BI tool, like SAS, is used to produce reports, some feed into an accounting system which runs its own set of reports, and others still are pretty much left to themselves. A second business unit was acquired a few years ago and has its own set up, similar but different enough to be mostly incompatible.
A data platform helps to join up those silos of data by enabling people and systems to use common approaches to identify, ingest, analyse, and visualise the data. Data doesn’t have to all be in one place of course (more on data mesh architectures here), but the tools to work with it are part of the data platform. The idea is to reduce the friction experienced by people who need to work with their company’s data.
Your platform is a place that comes with approvals and sign offs from the various interested parties (security, compliance etc.). Plus it offers common tooling that is easy to access and to use—an SQL query capability, a data science notebook, a visualisation tool such as PowerBI or Superset. You now have an environment offering a lower barrier to entry for analytics workloads than seen traditionally.
3. Trust in your governance process
With a secure data platform, concerns around regulated data can be mitigated. There’s no need to run scared of PII data for example, as long as the necessary checks are in place to make sure consent has been given and data is used for that specific purpose. This is a world away from where your organisation might be right now—downloading data to uncontrolled spreadsheets perhaps. The data platform is not a governance hurdle to overcome; it’s an enabler to allow your analysts to work with governed data in comfort.
4. Industrialise your Excel and ad hoc analysis
Relying on spreadsheets for anything more than ad hoc calculations and visualisations gets into scaling difficulties very quickly in the enterprise landscape: you will struggle to run large data transformations, share data, and make good use of reuse and automation.
Your data platform will provide tools to collaborate and publish analytics, and most importantly (for me anyway, as an engineer at heart) a way to check in and commit analytics work, recording it in the way software is recorded, and therefore meaning it is reusable and can be more easily automated and scaled.
5. Speed up time to value
Things that slow down the production of value from your data range from tooling, to silos, to running business logic in spreadsheets. Outdated data tooling and siloed data means the significant gains of modern data exploitation methods are not available to the business. Expensive licences and difficulty hiring expertise in a declining market for those legacy tools compound this issue.
A good data platform will allow you to use modern technologies and modern skills, making it attractive to existing employees and giving a wider pool of external talent to recruit too. With governance, access, exploitation tools, and publishing your analytics all taken care of by the data platform, all of this shrinks the time to value for a data project.
6. Enable future data exploitation
This is the secret sixth reason you need a data platform. Often talked about as a main driver for adopting data platforms and new technologies is the ability to lift your analytics beyond reporting what happened in the last month or quarter and towards predictive and prescriptive analytics.
In reality this is much more likely to be a longer term benefit: the data platform enables this through tooling and availability of data, but the real challenges are in culture, skill sets, and understanding what kinds of questions your business wants to answer. The data platform gives you this opportunity though and enables your future data exploitation.
In Summary
We’ve seen that getting value from enterprise data can come with a number of challenges. You will notice by reading carefully that a data platform on its own will solve precisely none of these challenges. But it is a powerful enabler you can’t afford to be without. It will help your organisation unify its approach to exploiting its data.
In one place you will be lifting the hard work off your employees’ shoulders: getting access to data, offering reusable components to transform and analyse data, and bringing prebuilt analytics and publishing tooling to a wide user base.
The data platform will offer order to the world of spreadsheet analytics, and set your company up for success in future analytics endeavours.
