iStock Big Data

In the US, department store Target claims to have guessed which women shoppers were pregnant based on the rate at which they bought certain vitamins. In France, local governments are now able to recognise graffiti patterns and identify vandals based on a comparison of style with other known samples. In the UK, supermarket chain Tesco has decided to reduce heating and lighting costs by installing sensors throughout the stores and analysing enormous amounts of information to make decisions real time.

All of these cases are examples of what we call Big Data. Big Data is so big that Google CEO Eric Schmidt estimates that every two days we create as much information as we did from the time the first hunter painted an animal in a cave up until the year 2003. Perhaps Schmidt’s observation can be used as a measuring stick in the same way we’ve used the following observation from Gordon Moore in the 1960s, which has morphed into what we now know as 'Moore’s Law'. Moore’s exact words were: "The number of transistors incorporated in a chip will approximately double every 24 months."

But just how big is it going to get?

The UK government thinks Big Data is going to be humongous. Earlier this year, David Willetts MP, Minister of the UK Department for Business Innovation & Skills, gave Big Data the number one spot on the list of eight areas where the government is investing in innovation. The list consists of initiatives where the UK already excels in research, and where it could also excel in commercialisation.

The European Union thinks Big Data is going to be a hairy monster. Just this year, legal frameworks have been proposed by the EU to ensure, among other things, consumer rights to digital oblivion, which is the right to be forgotten, and data portability, which is the right to access data on yourself in a structured format, and to modify it or transfer it to another automated processing system.

Many companies think Big Data is going to be monumental. New job functions have been created. Technology specialists are being hired to manage tools and administer platforms. Data scientists are being recruited to analyse, combine, and monitor information. A new breed of marketing manager is sought to act on new opportunities created by the piercing insights made possible by Big Data.

Industry analysts think Big Data is going to be whopping one day, but that it will take time for companies to really understand what to do with Big Data sets. Gartner says of the US market, which is the most developed Big Data market, "through 2015, 85% of Fortune 500 organisations will be unable to exploit Big Data for competitive advantage".

Vendors think Big Data is going to be vast. Platform providers are developing tools that can be used across industries in a variety of applications. A new approach to storing and searching information, called NoSQL (which, depending on who you talk to, means either "no SQL at all", or "not only SQL"), has emerged to allow searches on unstructured data. A new set of vendors offer NoSQL platforms. Rather than replace traditional RDBMS systems, these new technologies supplement platforms provided by vendors such as Oracle and SAP, who have already enhanced their in-memory database offerings in response to the challenges of sifting through enormous amounts of data in near real time.

Business and technology schools think Big Data is going to be huge. A number of schools in the UK are offering degrees in Big Data - and now, even the French are starting to follow suit. Just say the words 'Big Data' with a French accent, and preceded by the article le, and you can inquire about the appropriate programme opening up at your favourite French school.

[Part 2 - Ethical questions around Big Data]

[Part 3 - What UK organisations need to do now with Big Data]