The Unbearable Bigness Of Data

(And What We Should Be Doing About It)


Welcome to the Data Deluge.  

By now you’ve probably gotten sick of hearing about big data, little data, fat data, thin data and all manner of data. You’ve gotten your head around Terabytes, Exabytes and Zetabytes. You’ve noted that the price of data has crashed by 90% over the past few years on a per unit basis. Your CIO has mastered Hadoop and MongoDB and you understand the benefits of data lakes over, say, data puddles. The scary part of all of this is that we’re still in the early days of the data deluge. We are hurtling into a quantified universe fed by smart cities, homes and cars; platform driven models and clickstream driven relationships. In fact, I was having coffee this morning with the well travelled, well informed, and always insightful John McCarthy from Forrester, and we were positing that in a few years from now, data will take over from ‘Digital’ as the centrepiece of the organisational transformation and focus across the world.

Right now, though, we’re caught in a deluge with no real clarity about how we’re going to actually use all the data that’s floating around. And here are three key challenges we’re going to have to deal with:

What, not Why – A New Mindset

A question I often ask my colleagues who are experts in data sciences is as follows: let’s suppose that when it rains, people drink more cappuccinos. Now, if Starbucks knew this, it could advertise or promote cappuccinos every time it rained. It could even launch branded umbrellas. But how would it discover this? Historically, the story would be one of a smart store manager who one day realises that rainy days increases his cappuccino sales, and having defined the premise, starts to collect the data to validate his hypothesis. Or even more traditionally, Costa Coffee runs focused groups, and the link between weather and coffee preferences is established. Critically, a qualitative hypothesis would be at the front of the process and data collection would follow. Because, how else would we know if it’s the rainfall or the pollen count or indeed, the volume of traffic on the roads that we should be correlating coffee sales with?

In the new world of data, or ‘big data’, this works the other way around. A brand like Caffe Nero could take all their sales data across the world, and run hundreds or thousands of analyses, searching for correlation, with any number of external and easily accessible data sources. This includes the obvious ones such as weather, or transport, but also for example days of week or month, time of day, and train and bus schedules, sales in other retail stores, etc. This list is only limited by your creativity and the data availability.

But most fundamentally, this is a shift from why, to what. As well highlighted by Cukier and Schonberger in their book on Big Data, in this new world, we find the correlation first and then the hypothesis. And we actually don’t care why. Let’s suppose we discovered that the coffee consumption actually varied with the tides. We would need to verify whether this was simply a spurious correlation, but from there on, we could go straight to predictability and dispense with the causality, or the ‘why’ question. This is a mind shift for those of us who are used to a ‘scientific’ mentality which requires us to establish causality in order for any approach to rise beyond heuristics into a scaled and logical argument.

The Crown Jewels?

If you haven’t read Adrian Slywotzky’s great book on Value Migration, this would be a great time to start. The book talks through how value migrates from older to newer business models or from a segment to another, or even one firm to another.

We are going to see significant value moving to those companies in each industry that get the value of the data. Be it healthcare, or education, or automobiles, or even heavy industry. Either an incumbent, such as GE, with it’s smart engines and its Predix platform, or challengers such as Amazon, in retail, or upstarts such as 23andme.

The question you want to be asking yourself is, in your industry and in your firm, what are some of the areas of opportunity where you can create new platforms to data-enable processes, or value to customers. How can you converge the primary and ancillary meaning in your data onto areas of your competitive strategy? And also, you may want to perform an audit of what data you might be giving away, perhaps because you feel that it’s not core to your business or you have a player in the industry who has historically be collecting this data. For example, Experian and credit scores. Ask yourself are you merely giving away data that you don’t use, or are you handing over the source of competitive differentiation in your industry? Remember the story about IBM, Microsoft and Intel? I argued this point in my post about Uber and taxi companies, too.

To underscore the earlier point, I believe that value will increasingly migrate, in each industry, to those who best manage, and build strategic & competitive alignment with their data strategies and/ or new offerings based on the data and its meaning.

Adding Love To Data

A couple of years ago, at the annual FT Innovate conference, a lively round table discussion followed after a well known retail CEO had made a presentation about data and analysis. The presentation covered examples of analysing customers to great and occasionally worrying insight, within the industry. From knowing if a woman is pregnant even before she knows it herself, to people having affairs, or stacking beer and nappies together, in front of the stores, all of this can today be deduced from data itself. The debates afterward spilled over onto lunch led to the insight that while there’s been a lot of talk about analysing customers, it misses the point of empathy.

Let’s remind ourselves though – the customer does not want to be analysed. As with any relationship, he or she wants to be loved, cherished, understood and served better.At the end of the day, for most businesses, this translates to a mind-shift again, of adding a layer of human understanding to data, to creatively and emotionally assess the customers’ needs and to allow the analytics to feed off the empathy and emotional connect, rather than be driven purely by the algorithm.

In Sum:

You will hear a whole lot more about data in the coming weeks and months. However, for starters, you could keep these 3 guidelines in mind:

  • Look for correlations, not causality. You want to throw tons of data together and find patterns that aren’t born in some logical causal hypothesis but is simply an observed correlation done at the data level.
  • Be aware that the future of your industry, just like any industry, will involve the value of data. So try to identify and own areas of data which help you drive competitive advantage and/or new products and services, and start building proofs of concept.
  • Add love to data. Don’t just analyse your customers. Bring observation and empathy to the table as well, and marry the analytics with the empathy for best results.
What are your lessons from working with big, small and tiny data so far?

Welcome to the 1980s

Antique Telephone

Data Antiquity Award

A fortnight ago, I lost my debit card. I say lost, but my 4-year-old daughter discovered it under the car seat the next day. Of course, by then I had cancelled the card and my bank assured me that a new one was on it’s way. We could do that on the website – it was easier than calling on the phone and listening to ‘music’ for hours. As it turns out this was within a day of my wife’s card expiring, so she was also talking with the same bank for the same purpose – a new debit card.

Cards ordered, we could relax, and get on with our lives. Although we had to rely on using our credit cards at ATMs to withdraw cash. But a week passed and no cards showed up. So we got onto the phone and spoke with the advisor at the call centre. Imagine our surprise, when we were told that the card had been dispatched, but to our previous address – which we had left exactly 13 months ago. The bank didn’t know that we had moved. How was this possible? Even worse, they had my old mobile phone number – which I have not used in 6 months.

We moved house at the end of May 2014. Having done this a few times, we have a fairly comprehensive checklist for all the various updates. From utility providers, to post office, to banks to employers, it’s all there and we’re pretty sure we did it all. In fact, our credit cards, with the same bank have all the right information. We get the statements, and our online purchases go through with the new address confirmation. Absurdly, this information has not filtered through to the savings account side of my bank.

Let’s assume for a moment that we may have made a ‘mistake’ in informing the credit card issuer, and not the retail bank. Is this really a mistake though? As consumers, do we need to inform each part of the bank individually? How bizarre that in the 13 intervening months, the bank has not picked up the fact that our address for the credit card issued by them is different from the address for the debit cards issued by them. This, by the way, is a major high street bank in the UK. I’m not naming them because that’s not the point of this story.

And consider this: quite apart from the inconvenience and the confusion, the bank has effectively posted my card AND my pin to the wrong person. It’ll get sent by registered post – but as we know, anybody can really sign for it. The gentleman who now lives in our house is a very nice man, who hails from China and works in the City, in London. But what if he was a villainous man, easily tempted into transgression? How ironic is it that after all the effort of sending the card and pin in separate packs and taking all the precautions of masking the pin, it gets sent to the wrong person! A reminder that you’re only as secure as your weakest link!

And there were so many opportunities to get it right! Even a simple pop up while ordering the new card, to say thanks, we will be sending your card to this address, and showing the last 3 digits of the post code could be an easy way to trap this error. To be fair, Samir, the guy at the other end of the phone at the call centre, did what he could to rectify the errors and the bank has offered us a £60 payment as an apology. Assuming that the new cards get here by Monday, I’m inclined to get over this and move on. Of course, if the cards don’t arrive as expected, and we have go on holiday on Wednesday without them, I will have to tell them what to do with the £60, and it will not be polite.

Fresh from this brush with data antiquity, we ran into another one, this time a Harley Street clinic, who called my wife for an appointment. “Can you come tomorrow?” they said. We made a herculean effort to get there, the next day, beating tube strikes and insane traffic. The appointment was actually on the following day, and the doctor was unavailable. I asked them, why didn’t you send through a confirmation email which would have sorted out the error or misunderstanding? “Mumble mumble” was the only answer I got.

It seems to me that despite all the hype about connected worlds, smart products and big, gargantuan data, we’re still at the starting block in so many ways. I’ve been in London for 12 years now. I’ve never tweeted a single word about Bollywood, yet Twitter regularly asks me if I want to follow the latest Bollywood stars, or Indian TV personalities. And I’m sure many of you, like me, have been unwilling recipients of re-targeting ads – being told about great new folding cycles a month after you searched for and actually bought it. In all these ways, we’re still in the trough of digital disillusionment, to borrow a phrase from Gartner.

I guess the question left in my mind is, how many businesses, big and small are discussing big data and digital transformation projects before getting the basics right? How many are trying to leap into the 21st century, with one foot still stuck in the 1980s?  A good digital strategy should ensure an appropriate choice architecture which allows you to focus on getting the basics right while simultaneously creating a roadmap towards a bigger vision.

So the next time you encounter data-antiquity as a customer, or in your business, remember that in the digital era, there is no excuse for getting the basics right and that a good data foundation is at the heart of any digital transformation roadmap.