CIOs have been fighting to keep the deluge of data at bay for years. But could they finally be losing the battle?
Analysts are predicting the worst. IDC says that there is already more data in the world than there is space to store it and that demand for storage capacity will, over the next three years, increase at a compound annual rate of nearly 50 per cent.
The figures are mind-boggling: according to a 2010 The Economist report, the US retail firm Wal-Mart “handles more than one million customer transactions every hour, feeding databases estimated at more than 2.5 petabytes — the equivalent of 167 times the number of books in America’s Library of Congress”.
CIOs themselves are concerned: three years ago Rick Chapman, CIO of Kindred Healthcare told CIO magazine that the firm’s data volumes were increasing by 40 per cent a year, and that it was already dealing with 400 terabytes of data.
Only a few years ago these quantities would have been unimaginable. So what’s driving the change?
Much of it is down to the growth in enterprise data, including the customer transactions of the type stored by Wal-Mart. Such data provides organisations with rich information about customer behaviour.
Unstructured content is another culprit: emails, PDF files, Word documents, PowerPoint slides and voicemail messages have been around for a while, but now firms also have to deal with social media output, such as blogs, tweets and instant messages.
Aman Munglani, principal research analyst at Gartner, predicts that unstructured data will make up 80 per cent of the total available capacity by 2014.