The Unbearable Bigness Of Data

(And What We Should Be Doing About It)


Welcome to the Data Deluge.  

By now you’ve probably gotten sick of hearing about big data, little data, fat data, thin data and all manner of data. You’ve gotten your head around Terabytes, Exabytes and Zetabytes. You’ve noted that the price of data has crashed by 90% over the past few years on a per unit basis. Your CIO has mastered Hadoop and MongoDB and you understand the benefits of data lakes over, say, data puddles. The scary part of all of this is that we’re still in the early days of the data deluge. We are hurtling into a quantified universe fed by smart cities, homes and cars; platform driven models and clickstream driven relationships. In fact, I was having coffee this morning with the well travelled, well informed, and always insightful John McCarthy from Forrester, and we were positing that in a few years from now, data will take over from ‘Digital’ as the centrepiece of the organisational transformation and focus across the world.

Right now, though, we’re caught in a deluge with no real clarity about how we’re going to actually use all the data that’s floating around. And here are three key challenges we’re going to have to deal with:

What, not Why – A New Mindset

A question I often ask my colleagues who are experts in data sciences is as follows: let’s suppose that when it rains, people drink more cappuccinos. Now, if Starbucks knew this, it could advertise or promote cappuccinos every time it rained. It could even launch branded umbrellas. But how would it discover this? Historically, the story would be one of a smart store manager who one day realises that rainy days increases his cappuccino sales, and having defined the premise, starts to collect the data to validate his hypothesis. Or even more traditionally, Costa Coffee runs focused groups, and the link between weather and coffee preferences is established. Critically, a qualitative hypothesis would be at the front of the process and data collection would follow. Because, how else would we know if it’s the rainfall or the pollen count or indeed, the volume of traffic on the roads that we should be correlating coffee sales with?

In the new world of data, or ‘big data’, this works the other way around. A brand like Caffe Nero could take all their sales data across the world, and run hundreds or thousands of analyses, searching for correlation, with any number of external and easily accessible data sources. This includes the obvious ones such as weather, or transport, but also for example days of week or month, time of day, and train and bus schedules, sales in other retail stores, etc. This list is only limited by your creativity and the data availability.

But most fundamentally, this is a shift from why, to what. As well highlighted by Cukier and Schonberger in their book on Big Data, in this new world, we find the correlation first and then the hypothesis. And we actually don’t care why. Let’s suppose we discovered that the coffee consumption actually varied with the tides. We would need to verify whether this was simply a spurious correlation, but from there on, we could go straight to predictability and dispense with the causality, or the ‘why’ question. This is a mind shift for those of us who are used to a ‘scientific’ mentality which requires us to establish causality in order for any approach to rise beyond heuristics into a scaled and logical argument.

The Crown Jewels?

If you haven’t read Adrian Slywotzky’s great book on Value Migration, this would be a great time to start. The book talks through how value migrates from older to newer business models or from a segment to another, or even one firm to another.

We are going to see significant value moving to those companies in each industry that get the value of the data. Be it healthcare, or education, or automobiles, or even heavy industry. Either an incumbent, such as GE, with it’s smart engines and its Predix platform, or challengers such as Amazon, in retail, or upstarts such as 23andme.

The question you want to be asking yourself is, in your industry and in your firm, what are some of the areas of opportunity where you can create new platforms to data-enable processes, or value to customers. How can you converge the primary and ancillary meaning in your data onto areas of your competitive strategy? And also, you may want to perform an audit of what data you might be giving away, perhaps because you feel that it’s not core to your business or you have a player in the industry who has historically be collecting this data. For example, Experian and credit scores. Ask yourself are you merely giving away data that you don’t use, or are you handing over the source of competitive differentiation in your industry? Remember the story about IBM, Microsoft and Intel? I argued this point in my post about Uber and taxi companies, too.

To underscore the earlier point, I believe that value will increasingly migrate, in each industry, to those who best manage, and build strategic & competitive alignment with their data strategies and/ or new offerings based on the data and its meaning.

Adding Love To Data

A couple of years ago, at the annual FT Innovate conference, a lively round table discussion followed after a well known retail CEO had made a presentation about data and analysis. The presentation covered examples of analysing customers to great and occasionally worrying insight, within the industry. From knowing if a woman is pregnant even before she knows it herself, to people having affairs, or stacking beer and nappies together, in front of the stores, all of this can today be deduced from data itself. The debates afterward spilled over onto lunch led to the insight that while there’s been a lot of talk about analysing customers, it misses the point of empathy.

Let’s remind ourselves though – the customer does not want to be analysed. As with any relationship, he or she wants to be loved, cherished, understood and served better.At the end of the day, for most businesses, this translates to a mind-shift again, of adding a layer of human understanding to data, to creatively and emotionally assess the customers’ needs and to allow the analytics to feed off the empathy and emotional connect, rather than be driven purely by the algorithm.

In Sum:

You will hear a whole lot more about data in the coming weeks and months. However, for starters, you could keep these 3 guidelines in mind:

  • Look for correlations, not causality. You want to throw tons of data together and find patterns that aren’t born in some logical causal hypothesis but is simply an observed correlation done at the data level.
  • Be aware that the future of your industry, just like any industry, will involve the value of data. So try to identify and own areas of data which help you drive competitive advantage and/or new products and services, and start building proofs of concept.
  • Add love to data. Don’t just analyse your customers. Bring observation and empathy to the table as well, and marry the analytics with the empathy for best results.
What are your lessons from working with big, small and tiny data so far?

Why Is Good Design So Difficult?

Golf has been described as a game where you try to put a very small ball into a even smaller and remotely distant hole, with instruments singularly ill adapted for the purpose. 
While a degree of difficulty is desirable in sport, it’s safe to say, we don’t want to gamify everything we do to by adding levels of challenges. On the contrary, we spend much of our lives individually and collectively working towards the opposite objective. How to make it easier. Easier to shop, to commute, to acquire and consume. Easier to access, easier to use. Making things easier has always been the holy grail of progress, the end game of science and technology and of design.

So why is good design so infernally difficult? Why do we feel at so many points in our lives, and during our everyday routines, that a product or a service has been really badly designed? Just this morning we discovered that the water filter in our fridge needed replacing. This can only be done by moving the cupboard sized fridge away from the wall, and unscrewing a plate behind, and then following nearly non-existent instructions in the manual, which are definitely not for the model we own. We decided to call in the professionals. Samsung recommends that the filter be changed every 6 months. Would you pay £80 every 6 months to a plumber to keep your fridge running. No, me neither. Which is why we’re going to video record the plumber and ensure we can do it ourselves next time round. Computer hardware manufacturers solved a similar design problem a long time ago, by putting the USB ports and network points in front of the device, rather than behind, and colour coding the ports.

But surely you must see instances in your daily lives that make you want to pull your hair out, simply because of the thoughtlessness of a service or a product? Some are unintended such as the walkie talkie building and the hitler tea pot. But others are just examples of callousness in design, while still others are simply relics from an older world that need a makeover.

In fact this lethargy is probably at the heart of any number of products and services which were fine a few years ago, but you simply would not design them the same way today. Consider home security. Some of the leading brands offer you the choice of “key-holder” response and “police response” at different price points. But why not a combination? Why can’t I opt for police response when I’m on holiday? Why does the alarm signal go to the provider first and then as a phone call to me? Why can’t it trigger a message to an app on my phone, with an option to turn on one or more video streams from inside the house? If you had to design a home security system from scratch today how would you design it? Chances are you’ve walked down a road where an alarm has gone off, in a home or a shop, and you’ve just walked past it, because everybody around is just ignoring the noise and getting on with their lives.

A friend of mine had an instance of an alarm going off while he was on holiday and as I was the nominated person the alarm company called me and I went round to his house at 6 AM on Sunday morning. As it turned out, there wasn’t much going on but a friendly neighbour popped by and  we looked around the house together. The neighbour found a way to get into the house because he discovered that a back window was open upon entering the house the alarm went off again. At which point the alarm company called on the land-line inside the house. The neighbour picked up the phone and he told the alarm company that he was the neighbour and he was with the key-,holder and that everything was alright and incredibly, the alarm company said in that case it was fine and hung up. Needless to say my friend changed the alarm company as soon as he got back.

Today you would probably design something like this, which allows you to see what’s going on in your house. DIY web cams aren’t the answer though, as they’re susceptible to be hacked. The right combination of institutional and personal information sharing is key.  You might also enable some community and collaborative features which allows the local community to get involved – after all they’re the ones putting up with the alarm shrieking, so they have a vested interest. Not to mention that they run a risk of a similar burglary.

Tony Fadell, the designer behind Nest and now at Google, calls this the challenge of habituation, the way in which we get used to how things are. (Like ignoring the neighbours house alarm). This is a brain feature which allows us to make somethings second nature so we can take our environment for granted without having to process everything we see all the time. The inertia, or lethargy for businesses to change a product or service stems from our habituation process. But also baked into this are network effects, accepted semiotics and social norms. The QWERTY keyboard could arguably be improved on, but the network cost of billions of people relearning how to type is mind-boggling. But the Android app Swype is a great example of a marginal design improvement. You do not want to try and improve universal symbols such as the red-amber-green of traffic lights or the signs for male and female toilets. Yet, we’ve all seen the attempts at creative improvements on toilet signs and I’ve certainly scratched my jaw on occasion trying to figure out the right sign when it’s been too abstract, or when it’s at the bottom of a flight of stairs in a pub where I’m already not at my thinking best.

Having said that, there are hundreds of products and services which could and should be better designed. Increasingly because of the way technology evolution is creating a change in our behaviour. A simple example is business cards, for those of you who still use them. Previously you would try to perhaps make your business card memorable, or be creative about how it conveys your key messages, especially if you weren’t working for a well known and recognizable brand. Increasingly though, you may want to ensure your business cards scan well, mobile apps for scanning business cards becomes a mainstream behaviour. This means avoiding vertical cards, and preserving clarity of the data in an OCR situation. The way Apple Pay or Uber scan your credit/debit cards into your mobile phone apps, is leaps ahead of earlier apps where you had to manually enter the information.

The design challenge of today, is that as products get smarter, and software gets baked into every product, how can we ensure that the experience is a good one. This isn’t simply about traditional interface design, because many products won’t have a traditional ‘user interface’ i.e. a screen of some description. In fact the user interface is now a combination of interaction, transaction, experience and environment design. You have to be aware of how the user is engaging with the product through it’s lifecycle – from unwrapping and setting up the product through to it’s ultimate replacement or disposal – and also how it engages with the environment. Parking sensors in a car is an excellent example of how the product engages with the environment and provides the feedback and engagement with the user so you know how close you might be to the object behind your car. But my earlier example of the fridge filter is a clear example of not considering the environments. I would think most people keep their fridges with their backs to the wall and probably wedged in between other appliances. As such accessing the back of the fridge is difficult and potentially dangerous if you’re moving a large fridge yourself.

One of the best principles for redesigning objects is to re-imagine their evolving purpose. Here are two examples. The first is the ambulance. Traditionally thought of as a vehicle to transport people to hospitals in the shortest time, and designed accordingly. Based on some research, it became apparent to the redesign team that a better way to consider the ambulance as a mobile unit for delivering energy care and medical services. I.e. the treatment begins in the ambulance itself. This made the principles of the redesign clearer and in turn led to a series of changes which dramatically changed the concept of the ambulance design. The second example is the wheelchair. Again, traditionally seen as a way for people with walking disability to gain the freedom of movement,  wheelchairs have always been visually  overpowering. Now, viewed as a way of giving the same people a normal social life, the redesign principle is that when you see somebody in a wheelchair, you should notice the person first and not the wheelchair. In fact even the standard wheelchair symbol is getting a redesign, in New York.

Both of these examples highlight the need for empathy, for research and for re-imagination. Sometimes the hardest of the three is the empathy and this is what makes great design so difficult. Even though plenty of design principles are now chronicled, perhaps none as well as those of Dieter Rams.

dieter rams design principles

In these times of digital transformation, the big opportunity for most organisations is to use software and smart technologies to redesign their products and services in a way that dramatically improves customer engagement, solves the problem or creates delight. The opportunities for rethinking and redesigning are immense, and if you don’t do it, the chances are that there’s a start up somewhere reinventing the design of your products.

Innovation in Pictures

These 2 cartoons wonderfully illustrate the nature of transformation – both the problem of structuring innovation and the opportunities to see things from a different perspective.

Think outside the box

The problem of structuing innovation….

Coffee new business model

The ability to map your business model to true customer needs….

3 Things Most Organisations Miss About Innovation

innovation-illustration live outside the box
There are probably more definitions of innovation than there are books about management, but I have my own favourite definition. It’s simple, and it says “Doing more with less”. To put it more technically – it’s shifting the input/ output curve. Imagine that it takes £100 to build each floor of a house, so a 10-storeyed house would require and investment of £1000. Let’s also suppose that it takes £50 per month to maintain the house. A business with a budget of, say 2000 could expect to build a 10 story house and maintain it for 20 months. Or it could build an 8 story house and maintain it for 24 months. This is the kind of resource allocation choice which most people make routinely in the course of their daily work. Usually the choices are more complex, but they follow the same principles. Innovation is the process where you can bring the cost of building per floor down to say, £80, so now your choice curve has shifted. You can build a 12 story house, and maintain it for 24 months.

So far so good, and so obvious, you might say. But now consider the following questions:

What if your closest competitors are already able to build houses at £80 per floor. Are you still being innovative? What about if companies elsewhere in the world are doing it, but not your competitors?

1: Benchmarking & Exploiting Safe Innovation.

This is the first key thing that people often miss. All innovation needs to work off a baseline. Before we try to do more with less, we need to understand what the benchmark is, at present. And that benchmark needs to consider your company, your competitors, your industry and even at a global level, depending on the area of innovation. In a remote village where people have never heard of pulleys, and people manually pull water out of wells in buckets, the arrival of a pulley may represent a local innovation. But of course, to the outside world, it’s not an ‘innovation’ per se. This is both good news and bad. The vast majority of companies are doing ‘catch up’ innovation of this kind. After all, once something has been done once, it really isn’t innovative any more. It’s just imitation. It’s not innovative to go direct to consumer once Nespresso has shown us the model. It’s not innovative to build an App Store – now that we can just copy Apple. And it’s not innovative to build a ‘wiki’ for your next project. Perhaps this is a harsh view and may be your organisation poses some specific challenges. Actually, on the positive side, this means that there’s enormous innovation opportunity in this catch up space. I call this ‘safe’ innovation, because it’s relatively low risk. The key risk is execution related. Somebody else has proven the model.

So when you next think about innovation, step 1 should be to conduct a benchmark, and identify the safe innovation opportunities based on your competitors, industry, region or global examples. Only once you’ve achieved parity with the benchmark do you really need to step into unchartered territory. And although I don’t have data on this, I’m willing to bet that safe innovation offers much higher risk-adjusted ROI than venturing into new areas.

2: Find A Problem First

Necessity, we know, is the mother of invention. It turns out therefore that Innovation and Invention are actually close siblings. The current thinking on innovation is that you shouldn’t have an innovation department or team. But equally you should first identify the problem you want to solve, in order for innovation to be effective. This is why you get the most innovation in resource starved communities. This is why the Indian ‘jugaad’ style of innovation is getting airtime now. In New Delhi, taxi drivers sometimes use a fake seat belt strap which can be slung over the shoulder at short notice to suggest to any law enforcement official aka policeman, that the driver is wearing a seatbelt. Less ethically cloudy, is the example of the fake steering lock, which is a cheap but realistic plastic decoy which looks just like a metal steering lock, the kind often used in India by owners when the leave a car parked. The plastic decoy acts as a deterrent, and costs a fifth of the actual lock. Or indeed stories about creating post natal incubators out of auto parts so they can be easily maintained.

The resource may not just be money – it may be time. Thus is is that the pit-stops for formula 1 cars have evolved dramatically from 67 seconds to 3 seconds over the past 65 years, and is now proving the inspiration for surgeons in operating theatres.

Resource constraint is often a key problem but identifying a specific business problem or opportunity also helps. The role of serendipity should not be understated – considering the history of 3M post its and many useful tools are the result of a ‘solution in search of a problem’ – notably the flash drive. But almost always, the original effort was aimed at solving a problem – even if it ended up solving a different problem.

Invention and innovation are also connected by concentric ripples – usually a new invention sets off a ripple of innovation as people start to use the invention in newer and unplanned ways. Some of these lead to further inventions but others live happily ever after as great innovations. Almost everything we see on the web today represents innovation around the core invention of the world wide web and a few other key inventions. You can expect to see great applications of graphene over the next few months and years, and everybody knows the story of text messaging, invented for one purpose, but made famous for a completely different reason.
What is common though, is the application of innovation thinking to a specific problem or opportunity. Not just throwing people and resources into a blue sky mode of ‘let’s think of some cool stuff’. That’s usually the least effective way of getting returns on innovation investment.

So the next time you think about innovation, spend the time upfront to ask the difficult questions like ‘what’ and ‘why’ and ‘which’ until you land on a specific problem that needs to be solved. Then unleash your innovation.

3: Where Is The Creativity?

90% of the literature and discussions around innovation focus on how to manage the process. Including methodologies, how to allocate funds, how to build teams, how to nurture ideas, how to measure them, and so on. Yet at the very heart of innovation lies creativity, and there’s not enough discussion about this.

Creativity is often a personality trait. Some people are creative – usually by nature, and often by virtue of the breadth and variety of their work and life experiences. Others are not. Do you employ creative people? Do you employ a variety of people? Do you encourage the variety of experiences? Every time a bank hires an employee on the basis of his or her banking experience alone, it loses a bit of creativity. Obviously, the majority of people working in any industry should be experts in that industry, but creativity is often about introducing an element of managed entropy by adding people who bring a different set of experiences, mindset and thinking to the organisation. When I was first offered a job in a technology company, I said that I knew little about technology. I remember clearly the reply: “I have hundreds of people who know about technology, don’t worry about that” (thanks Pradeep!)

Beyond the individual, the culture and environment play a big role. The physical manifestations of this are easy to achieve – most companies have labs or spaces intended to foster creativity. The harder part is to build an organisational culture that fosters creativity. This means an ability to encourage risk taking, thinking differently, the ability to evaluate a problem through multiple lens, the willingness to continuously look outside of the business for ideas and perspectives and interestingly, a trade off with productivity at some levels. Call it the breathing space for creativity. Steven Johnson calls this the primordial soup, as exemplified by the natural environment where new species flourish. Remember, you can’t process your way to innovation, but you can significantly improve the probability by fostering the right environment.

The third requirement for creativity is the quick feedback system that recognises and rewards early and good outcomes, instead of killing them because they don’t fit the current model. You have to be prepared to be wrong, when you innovate but you also have to rely on being recognised when you’re right. And often, right and wrong can be quite confusing if measured against the yardstick of the past or even the present. The only measure should be – does it solve the problem, and does it do it in a more resource efficient way? No better example here than Kodak, which had created digital cameras long before they were popular, but chose to ignore them.

Bottom Line

So the next time you sit down to think of innovation, in your business or industry, or even in a small group, ask yourself these 3 questions up front.

1. What’s the problem we want to solve

2. What is the currently accepted and best way of doing this, and are we doing that already?
3. Are we bringing creativity to the mix in order to get different outcomes?

Around this you can wrap all the well known lean and fast-fail processes and build structures and systems to scale your innovations. But missing these three things can lead to the creation of a shell of processes without the spark of innovation to actually deliver results.