Big data.

We’re all used to data – the sample sized stuff you get from media agencies, consumer polls, sales data and market research panels – but it’s the “big data” that we’ll soon have at our fingertips that is going to revolutionise the marketing industry.

This “big data” is different – not just because it comes in vast quantities but also because it arrives constantly, in real time and in a variety of formats: text, audio, video, click streams, log files and more.

But to describe it as “big” is a bit like saying that the surface of the sun is “hot” or that the earth is “heavy”.

To give it come perspective 90% of the data in the world today has been created in the last two years alone and this data comes from everywhere: from sensors used to gather climate information, posts to social media sites, digital pictures and videos posted online, transaction records of online purchases, and from cell phone GPS signals to name a few.

When the Sloan Digital Sky Survey started work in 2000, its telescope in New Mexico collected more data in its first few weeks than had been amassed in the entire history of astronomy. Now, a decade later, its archive contains a whopping 140 terabytes of information. When its successor, the Large Synoptic Survey Telescope, comes on stream in 2016, it will acquire that quantity of data every five days.

As powerful computers become ever more adept at spotting patterns in this data pile the potential for it to be used to sell us more stuff is huge.

Big data now means big knowledge and companies with big knowledge can earn big money. In 2010 Apple poured $1B into a big data ‘cloud’ storage center in Maiden, North Carolina and in last five years alone IBM has invested more than $14B in firms that offer big data analytics tools.

Big Knowledge.

Armed with this “big knowledge” companies like IBM will be able to make decisions with a whole new level of accuracy and specificity. They will no longer be based on best guesses about the customer but rather by real behaviours of the customer.

The days of marketing via web, mobile and social by “gut feel” is over. It’s not a matter of what you think your customer might like, it has to be what your customer is telling you they want, individually.

And clever quantitative analysis is being applied to many aspects of life, not just missile trajectories or financial hedging strategies, as in the past. From Amazon’s “you might also want …” recommendations that are based on information available about your buying patterns and the buying patterns of those purchasing the same item. To Bing advising us whether to buy an airline ticket now or wait for the price to come down by examining 225 billion flight and price records.

It’s hard to think about these things in the abstract, so I thought a couple of examples might be helpful.

First example: EcoFactor, based in California, is using big data to help tens of thousands of US homeowners to reduce their energy bills and improve their energy efficiency. EcoFactor collects thousands of data points — from weather to regional building codes to home value — that give clues about how an individual home might use energy more efficiently and respond in a way that promotes energy saving. After EcoFactor is installed, all homeowners have to do is adjust their thermostats as usual for several days. The software remembers what they like, in relation to seasons, weather conditions and size of the property, so that they never have to worry about it again. By making over 1,000 micro adjustments per month to the thermostat – bumping it up and down ever so slightly, they are able to optimize energy usage, without the user even noticing the temperature change – and shave energy demand to reduce its customers’ monthly energy bills. Whilst it’s still early days for the service, by calculating each home’s individual “dynamic signature,” EcoFactor learns how much energy is required to heat and cool the home to reduce average energy bills by 17%.

Second example: MIT-spinoff Bluefin uses publicly available social media commentary from Twitter, Facebook and blogs to measure viewer engagement with television shows and ads. Each month, the company collects more than 3 billion posts on Twitter and Facebook that it maps to a growing bank of over 200k TV shows and commercials. By looking at the context of words expressed by individuals, clever algorithms “ground” the meaning of these comments in the larger content and then connect these back to the events, people, products, brands, and viewing contexts. All this means marketers and brands can understand where and when a brand’s TV ad creative triggers high social media commentary; agencies can optimise social media conversations to guide the planning and buying process to targeting networks or shows with high response levels; and TV Networks can see in real time how audience react to shows and then how they trend over time. You can imagine the stuff advertising-data scientists are working on: connecting insights about types of ads that will be successful with certain programs, advising brands on how to tailor their message to their audience or forecasting with near 100% accuracy whether something is going to be successful or not. Powerful stuff.

Big responsibility.

There can be a lot of personal information in the big data pile and there are big questions being asked about who actually owns this information, most of which is collected, without your knowledge.

A recent Technology Review piece introduced readers to “I Can Stalk You”, a website set up by two researchers to warn consumers that they are unwittingly providing too much information about themselves. At a recent conference the founders showed the audience how your cell phone can be used to disclose too much personal information. They started with cell phone pictures posted on an anonymous Twitter account. Since each snapshot was encoded with location metadata, they were able to use a variety of sources to find the person’s home address, name, place of work, wife’s name, and information about his kids.

The big ask. 

Some of the ways we measure things right now are going to be overtaken very quickly and to stay competitive, Marketers need to begin to adapt their marketing systems and their departments to include marketing analytics across every aspect. Hal Varian, Google’s chief economist, predicts that the job of statistician will become the “sexiest” around. Whilst I very much doubt that data can ever be sexy I do believe that the companies that can extract wisdom from these data piles will succeed and those that don’t will fail.

Is that scary or exciting?

Funny thing you notice about having children is how simple life is. I was at the park with son the other day and started playing with some kids he didn’t know. I watched him stroll up to the group of kids, size them up then sit down and begin playing. When we left he proudly told me he’d been playing with his friends. I bet if Ben had a Facebook page he’d have no problem making friends if the performance in the park was anything to go by.

So how come marketers find it so hard to keep things simple?

I’m convinced most of the time it’s because they don’t understand social media – after all the majority of marketeers in senior positions haven’t grown up with social media – and I include myself in that. The rest of the time it’s because they accept bad advice – when in fact the best thing they could be do is rolling up their sleeves and getting stuck in, rather than trying to learn social media from a text book.

Bring on the world awash with Whopper Sacrifice’s and Skittles update-the-rainbows because we don’t want to see geniuses like Hugh Laurie deflowered in advertising monologues that don’t seek to engage the consumer.

So when ENO posed the question: ever thought how odd your online life is? It connected. I watched their charming video and I reflected on how social media has made the world a more complicated place and wished I could have gone and watched their opera.

Hope you enjoy it as much as I did 🙂