Poor data quality is the leading cause of digital transformation failure
Written by Farah Kim, Product Marketing Manager, Data Ladder
Companies need to prioritize data transformation over digital transformation if they are looking to be data-driven.
In a digitally empowered age, businesses across the globe have grand ambitions of leveraging the power of AI, big data, and machine learning. Innovation is happening at a rapid pace. Companies are investing millions of dollars in building data lakes, moving to the cloud, hiring data scientists and chief data officers to run their digital transformation plans.
Yet, they fail. Spectacularly. Plenty of reports and surveys show that over 85 per cent of big data projects are failing with varying causes.
Enough has been written lately about how business cultures and unchecked ambitions lead to big data project failures. This piece will focus on how poor data quality is often overlooked and makes for one of the leading cause of digital transformation failure.
Clearing the confusion between digital and data transformation
Data transformation, the process of transforming raw data into a usable format is often, incorrectly, juxtaposed with digital transformation. Companies assume that because they are implementing data lakes, data centres or new ERPs, (which are all part of digital transformation), they are transforming their data.
This is a dangerous assumption. It takes the focus away from the real problem and gives companies a false sense of security. New systems are expected to resolve problems and help achieve transformational objectives but fails to do so.
The new ERP your company implemented six months ago, does not boost operational processes because data issues in the legacy system were not addressed. The new CRM your marketing team invested in to get in-depth customer insight doesn’t return the expected ROI because the team does not have data governance or data quality framework in place.
Understanding the difference between digital and data transformation can save a company from making costly mistakes. If organisations want to be data-driven, they have to start by understanding their data, fixing inconsistencies & transforming their data. Digital transformation is the end of the process — data transformation is the start!
Common data quality issues that business leaders overlook
We’ve worked with Fortune 500 clients who prioritised digital transformation only to find out their data was not ready for it. Some of the most common issues usually are:
- Data siloed away in disparate sources. The larger the business, the higher the chances of data being stored in multiple databases, giving the organisation a disproportionate and inaccurate understanding of their data.
- A human-dependent data collection process. With people manually entering data, there is always a high chance of poor data. A human-dependent data collection process will always be the leading cause of data quality issues. A typing mistake, a contextual understanding of a name or location, a missed number etc are all minor instances that over time degrades the quality of data.
- Duplicated data that has been lying in the dumps for eons: A company could be collecting the same consumer data for multiple purposes. Year after year, the same consumer data is recorded in a hundred different ways stored in multiple data sources. An insurance company had a hard time with yearly reporting because of duplicated data that would be collected over the months. A retailer had to delay its business expansion plans by six months because their data didn’t give the right picture.
- Data that does not give a unified source of truth: A bank had a hard time creating personalised experiences for their customers because each of their service (loan, mortgage, small business loans, insurance etc) had its own data sources. Customer information would be replicated over and over again as they used various services of the bank. Without a consolidated view of their customers, the bank was unable to understand the customer’s journey & failed to deliver personalised experiences.
- Data that is not prepared for business intelligence: Data preparation or wrangling is a technical ETL (Extract, Transform, Load) process but with real-world impact. Data that is not prepared; meaning not cleansed or optimised cannot be used for business intelligence. If a business is hoping to derive competitive opportunities or key audience insights, they cannot do so with incomplete, inaccurate, obsolete, duplicated data.
Poor data, bad data or dirty data is data that is the result of all these causes. Over time, poor data becomes an emergency, a security breach, a disaster that can break your business
How does bad data happen and why do business leaders overlook it?
An old but still relevant Gartner report revealed that at any given time, poor data quality is a primary reason for 40 per cent of all business initiatives failing to achieve their targeted benefits. Although a decade old report, this research still stands its time. Today, as organisations use on average more than 400 applications at any given time, data is constantly streaming in, most of which is raw, dirty and unusable.
Managing this data and ensuring that your organisation has data it can trust is a regular job, one that everyone in the organisation should perform.
And herein lies the answer to the second part of this question.
Business leaders overlook data quality problems because for eons now, data is considered an IT responsibility.
- Wrong data collected by sales, marketing, or customer support reps? IT is responsible to clean it up.
- C-level executives need data for analysis? The IT team needs to clean up that data and deliver reports.
- Costly return mails because of flawed contact addresses? IT needs to ensure address validation.
Almost every data issue is itself siloed away in the IT department. Business leaders, including C-level personnel, are either unaware or uninterested in resolving data quality issues. Compared to other grand transformation plans, data quality issues are seemingly so inconsequential that it’s completely overlooked by decision-makers. Of course, until the transformation plan is stalled or fails and again….the IT department will take the blame.
This disconnect between IT and C-level suites is the fundamental reason why experts cite, ‘company culture’ as one of the leading causes of transformation failures. For an organisation to be data-driven, the responsibility has to be shared by business teams.
Why prioritise data transformation over digital transformation?
Because new technologies depend on accurate data. Whether you’re creating the next generation robots or whether you’re banking on big data to understand your audience — you need data you can trust.
Data transformation, therefore, becomes the means of achieving the digital transformation goal. As in our experience, companies that actively transformed their data and implemented data governance in place were able to increase their ROI, optimise their operational processes, make valuable use of their workforce & eventually were able to transition into a digitally empowered enterprise.
The conclusion is simple — to be data-driven, you need data you can trust. To get data you can trust, you need to implement a data quality framework.
Originally published here.
Originally published at https://digileaders.com on April 28, 2020.
