Data quality is not the most glamorous area of technology. If technology were a dance then data quality would sit on the sidelines getting little attention, looking on while mobile app development, big data and even dull old security got the admiring glances.
Let’s face it, no ambitious young executive dreams of going home after work and boasting “Hey, today I got promoted to data quality manager”.
Yet data quality is causing all kinds of problems for corporations.
A few of these make headlines, such as when in 1999 the $125m Mars Orbiter spacecraft crashed due to a mix-up between imperial and metric units.
Others seem comical. One major telephone company, in a scene reminiscent of the movie Spinal Tap, ordered boxes for its new range of phones with dimensions in centimetres rather than millimetres, the problem only coming to light when lorries started rolling into warehouses with boxes 1000 times too large in volume (this was a €25m mistake).
Most data quality issues are more mundane: old addresses for customers, incorrect product codes and wrong inventory details.
Yet all these seemingly small issues add up to a very large data quality problem.
I conducted a survey through my firm The Information Difference in 2010 in which participants were asked what proportion of their master data management budgets were allocated to data quality.
The average was 10 per cent of the budget, the problem being that the same companies admitted that the average spent on dealing with data quality was not 10 or even 20 per cent but a huge 30 per cent of their original project budget.
In another 2010 survey of 192 companies, just 12 per cent of respondents rated their data quality good or better.
Dr Thomas Redman’s 2001 book Data Quality, The Field Guide, estimated that data quality costs added up to the equivalent of around 10 per cent of the sales of a typical company.
Why does a problem of this scale remain?
In that same 2010 survey, a third of respondents admitted that they had no data quality programme of any kind.
Yet data quality software solutions abound, with dozens of products on the market, plus an army of consultants who are more than willing to advise companies on how to improve data quality.