When Technology Talks

Conversational Systems aka chatbots are starting to become mainstream – here’s why you should stay ahead of the game:


The shape-shifting of the yin-yang between humans and technology is one of the hallmarks of digital technologies, but it is perhaps most pronounced and exploit in the area of Conversational Systems. But to truly appreciate conversational systems, we need to go back a few steps.

For the longest part of the evolution of information technology, the technology has been the unwieldy and intransigent partner requiring humans to contort in order to fit. Mainframe and ERP system were largely built to defend the single version of truth and cared little for the experience. Cue hours of training, anti-intuitive interfaces, clunky experiences, and flows designed by analysts, not designers. Most of us have lived through many ages of this type of IT will have experienced this first hand. If these systems were buildings they would be warehouses and fortresses, not homes or palaces. Too bad if you didn’t like it. What’s ‘like’ got to do with it! (As Tina Turner might have sung!)

Digital technology started to change this model. Because of its roots in consumer technology rather than enterprise, design and adoption were very much the problem of the providers. This story weaves it’s way through the emergence of web, social media and culminates with the launch of the iPhone. There is no doubt – the iPhone made technology sexy. To extend the oft-quoted NASA analogy, it was the rocket in your pocket! With the emergence of the app environment and broadband internet, which was key to Web 2.0, it suddenly introduced a whole new ingredient into the technology cookbook – emotion! Steve Jobs didn’t just want technology to be likable, he wanted it to be lickable.

The balance between humans and technology has since been redressed significantly – apps and websites focus on intuitiveness, and molding the process around the user. It means that to deal with a bank, you don’t have to follow the banks’ convenience, for time and place, and follow their processes of filling a lifetime’s worth of forms. Instead, banks work hard to make it work for you. And you want it 24/7, on the train, at bus stops, in the elevator and before you get out from under your blanket in the morning. And the banks have to make that happen. The mouse has given way to the finger. Humans and technology are ever closer. This was almost a meeting of equals.

But now the pendulum is swinging the other way. Technology wants to make it even easier for humans. Why should you learn to use an iPhone or figure out how to install and manage an app? You should just ask for it the way you would, in any other situation, and technology should do your bidding. Instead of downloading, installing and launching an app, you should simply ask the question in plain English (or a language of your choice) and the bank should respond. Welcome to the world of Conversational Systems. Ask Siri, ask Alexa, or Cortana, or Google or Bixby. But wait, we’ve gotten ahead of ourselves again.

The starting point for conversational systems is a chatbot. And a chatbot is an intelligent tool. Yes, we’re talking about AI and machine learning. Conversational systems are one of the early and universal applications of artificial intelligence. But it’s not so simple as just calling it AI. There are actually multiple points of intelligence in a conversational system. How does a chatbot work? Well for a user, you just type as though you were chatting with a human and you get human-like responses back in spoken language. Your experience is no different from talking on WhatsApp or Facebook Messenger for example, with another person. The point here is that you are able to ‘speak’ in a way that you are used to and the technology bend itself around you – your words, expressions, context, dialect, questions and even your mistakes.

Let’s look at that in a little more detail. This picture from Gartner does an excellent job of describing what goes into a chatbot:

The user interface is supported by a language processing and response generation engine. This means that the system needs to understand the users’ language. And it needs to generate responses that linguistically match the language of the user, and often the be cognizant of the mood. There are language engines like Microsoft’s LUIS, or Google’s language processing tool.

Behind this, the system needs to understand the user’s intent. Is this person trying to pay a bill? Change a password? Make a complaint? Ask a question? And to be able to qualify the question or issue, understand the urgency, etc. The third key area of intelligence is the contextual awareness. A customer talking to an insurance company in a flood-hit area has a fundamentally different context from a new prospect, though they may be asking the same question ‘does this policy cover xxx’. And of course, the context needs to be maintained through the conversation. An area which Amazon Alexa is just about fixing now. So when you say ‘Alexa who was the last president of the US’ and Alexa says ‘Barack Obama’ and you say ‘how tall is he?’ – Alexa doesn’t understand who ‘he’ is, because it hasn’t retained the context of the conversation.

And finally, the system needs to connect to a load of other systems to extract or enter data. And needless to say, when something goes wrong, it needs to ‘fail gracefully’: such as “Hmm… I don’t seem to know the answer to that. Let me check…” rather than “incorrect command” or “error, file not found”. These components are the building blocks of any conversational system. Just as with any AI application, we also need the data to train the chatbot, or allow it to learn ‘on the job’. One of the challenges in the latter approach is that the chatbot is prone to the biases of the data and real-time data may well have biases, as Microsoft discovered, with a Twitter-based chatbot.

We believe that chatbots are individually modular and very narrow in scope. You need to think of a network of chatbots, each doing a very small and focused task. One chatbot may just focus on verifying the customer’s information and authenticating her. Another may just do password changes. Although as far as the user is concerned, they may not know they’re communicating with many bots. The network of bots, therefore, acts as a single entity. We can even have humans and bots working in the same network with customers moving seamlessly between bots and human interactions depending on the state of the conversation. In fact, triaging the initial conversation and deciding whether a human or a bot needs to address the issue is also something a bot can be trained to do. My colleagues have built demos for bots which can walk a utility customer through a meter reading submission, for example, and also generate a bill for the customer.

Bots are by themselves individual micro-apps which are trained to perform certain tasks. You can have a meeting room bot which just helps you find and book the best available meeting room for your next meeting. Or a personal assistant bot that just manages your calendar, such as x.ai. We are building a number of these for our clients. Bots are excellent at handling multi-modal complexity – for example when the source of complexity is that there are many sources of information. The most classic case is 5 people trying to figure out the best time to meet, based on their calendars. As you well know, this is a repetitive, cyclical, time-consuming and often frustrating exercise, with dozens of emails and messages being exchanged. This is the kind of thing a bot can do very well, i.e. identify (say) the 3 best slots that fit everybody’s criteria on their calendars, keeping in mind travel and distances. Chatbots are just a special kind of bot that can also accept commands, and generate responses in natural language. Another kind of bot is a mailbot which can read an inbound email, contextualise it, and generate a response while capturing the relevant information in a data store. In our labs we have examples of mailbots which can respond to customers looking to change their address, for example.

Coming back to chatbots, if you also add a voice i.e. a speech to text engine to the interface, you get an Alexa or Siri kind of experience. Note that now we’re adding yet more intelligence that needs to recognise spoken words, often against background noises, and with a range of accents (yes, including Scottish ones). Of course, when it’s on the phone, there are many additional cues to the context of the user. The golden mean is in the space between recognising context and making appropriate suggestions, without making the user feel that their privacy is being compromised. Quite apart from the intelligence, one of the real benefits for users is often the design of the guided interface that allows a user to be walked step by step through what might be a daunting set of instructions or forms or a complex transaction – such as an insurance claim or a mortgage quote.

Gartner suggest that organisations will spend more on conversational systems in the next 3 years than they do on mobile applications. This would suggest a shift to a ‘conversation first’ interface model. There are already some excellent examples of early movers here. Babylon offers a conversational interface for providing initial medical inputs and is approved by the NHS. Quartz delivers news using a conversational model. You can also build conversational applications on Facebook to connect with customers and users. Chatbots are also being used to target online human trafficking. Needless to say, all those clunky corporate systems could well do with more conversational interfaces. Imagine just typing in “TravelBot – I need a ticket to Glasgow on Friday the 9th of February. Get me the first flight out from Heathrow and the last flight back to either Heathrow or Gatwick. The project code is 100153.” And sit back while the bot pulls up options for you, and also asks you whether you need to book conveyance.

Conversational systems will certainly make technology friendlier. It will humanise them in ways we have never experienced before. I often find myself saying please and thank you to Alexa and we will increasingly anthropomorphise technology via the nicknames we give these assistants. You may already have seen the movie “Her”. We should expect that this will bring many new great ideas, brilliant solutions and equally pose new social and psychological questions. Consider for example the chatbot that is desi§gned just for conversation – somebody to talk to when we need it. We often talk about how AI may take over the world and destroy us. But what if AI just wants to be our best friend?

My thanks to my colleagues and all the discussions which have sharpened my thinking about this – especially Anantha Sekar – who is my go-to person for all things Chatbots.

My book: Doing Digital – Connect, Quantify, Optimise – is available here, for the price of a coffee!

As with all my posts, all opinions here are my own – and not reflective of or necessarily shared by my employers.


CityMapper Does Connect, Quantify, Optimise

I know hundreds of people who know and love the Citymapper app, but they did something recently which really impressed me. As you know the app uses a number of public data streams to help you navigate your city – London being a good example. So you just have to say ‘get me to work’ or ‘get me home’ or any other destination and it tells you the best ways across buses, trains, walking, cycling, or driving. It also helpfully offers an Uber connection and for good measure includes a futuristic option such as ‘catapult’ or ‘teleportation’ to appeal to your quirky side.

They work across about 40 cities across 4 continents currently and base future cities expansion on a popular vote. Needless to say, they collect a ton of data about where people are travelling to and from. But the really interesting thing is what they do with all the data they collect.

In my recently published book Doing Digital, I proposed the model of Connect/ Quantify/ Optimise for digital. The model suggests these 3 stages for digital. Designing something that is easy and frictionless to use, allows you to get to Connect. Having thousands, or even millions of people use your app gives you the data which allows you to Quantify – for example, Citymapper can see where it’s most commonly visited areas are, where they have or lack coverage and market their app accordingly. They can build revenue models with Uber which allows commercialise the traffic they send to Uber. But the last step is where the magic often is – this is where you start to see new value and tweak your business or commercial model based on the opportunity that the Connect & Quantify stages throws up.

In the case of Citymapper, this is a bus service. According to CityMapper they can see based on their patterns, which areas and routes are underserved by public transport. Using this knowledge, they have launched a bus service in coordination with Transport for London to launch a small, green bus which runs on a fixed route. It’s called CMX1 and it’s a ‘pop-up route’ which presumably means that they will validate the route based on the data it generates over a trial period. What is even more fascinating is to consider some of the underlying assumptions that the CityMapper model is challenging. One of them being that bus routes are cast in stone and have to be long term commitments. But what if these routes could be intelligently introduced in response to shorter term needs and changes? The team are even trying to improve the experience of the bus journey by redesigning the bus outside and inside.

Photo 22-05-2017, 12 17 35 (1)
I love that their blog exhorts customers to ‘come and watch an app company fumble around with learning how to run a service with real vehicles and drivers’. This ticks the box of building learning organisations in the classic Eric Ries model. It is also an excellent example of ‘Connect / Quantify/ Optimise’. And I fully expect CityMapper to be thinking about Autonomous Vehicles in their R&D room – as they will probably be in a position to unleash an autonomous fleet in a few years based on their accumulated lessons from this exercise. Yet another Connect-Quantify-Optimise cycle at work.

Service Design Drives ‘Affordable Luxury’ Business Models

handcrafted shoes
One of the manifestations of digital business models built around good service design is the burgeoning of affordable luxury, which carves an entirely new aspirational category of of the sizeable middle class market.
But to illustrate, let me tell you a story, based on my experience of last week. I always have to buy trousers and get them altered because I don’t fit the shape that they come in, off the shelf. Or, as Garfield the cartoon cat once said, “I’m not overweight, I’m undertall”. So I was pleasantly surprised to find a high street retailer who offer an alteration service for their chinos (this is not common in London, btw). I bought a pair and took it to the counter to ask if they would measure and alter it for me. They said I would need to measure it myself and fold it to the point where I wanted the length reduced. Now, I don’t know if you’ve ever tried measuring your own trouser length. It’s about as easy as painting a smiley in the middle of your own back. So I said I’d take it home, measure it and bring it back. Next morning I was back with the trousers duly folded. Stood in the check out line and 10 mins later, I was told I needed to put a pin in to keep the fold. I asked for a pin, but of course, they didn’t have one. It took me another 10 mins of queuing at their alternation desk on another floor, and then a final wait in the original queue.  If you’re like me, at these moments you feel the life force seeping out of you.
For people who are time poor, which is most of us in most cities across the world, this ability to value the customers time is such a critical aspect of any service, that I’m always amazed when people don’t get it. In this case my joy at the finding the alteration service has definitely been tempered by the half an hour of my time I lost in the process. And based on my simple and one-off experience, you can immediately see how service design could be used to improve this dramatically – i.e. if somebody thought through the experience end to end, for the customer. Upfront information about the service terms is a simple idea. Just below the in-store poster announcing the service should be a simple list of what the shopper needs to do to use this wonderful service. Expectation setting often makes all the difference. Having a tape measure with a small weight that can be used like a plumb line in front of the mirror, to get an accurate length is another simple idea. These should be in the dressing room. On the basis that I might want to come back for more, why not let me store my measurements in the store app (they don’t). In my perfect world, I could sit at my desk at work at the end of day and order another couple of pairs, based on the new colours available, and they would have trousers ready in the store across from my work at a time that they could commit. The world is full of people like me who will repeat buy clothes from brands they trust and have had a good experience with. This is fundamentally the difference between a more traditional view of the business and an outside in view – driven by service design which puts the consumer in the centre and tries to remove all the friction in the entire buying cycle.
There are parts of the world, such as in most parts of India where this is an easy and people driven process. You buy a pair of trousers, and an in-house tailor takes your measurement and give you a time for delivery. This kind of people driven process, and infect the very idea of customisation is a luxury in the western world – especially when it comes to high street apparel brands. People are expensive. Factory made clothes cost less than half of tailor made ones. Thanks to improved stock management and product design, you can now get more options within the clothes you wear – a longer sleeve, a different collar, a slimmer cut, etc. But economics demands that any customisation at the point of delivery remains outside the purview of most products. Yet, digital models can significantly lower the bar for the accessibility of a luxury service. In my example of alteration – you can see how the app enablement and ordering based on my specific measurements could even be done in a centralised way and delivered to a store. I can live with a lead time of a week – as long as it’s a reliable one. At the core of this is the ability to take the customisation information off the universe of consumers and deliver the customisation at a much lower cost, at higher scale.
When you walk into your regular coffee shop, you don’t have to tell them each time that you want 2 shots of coffee, half a cup of foamed milk, with semi-skimmed extra hot milk (or as Niles Crane would say, “Double Cappuccino – half-caf, non-fat milk, with just enough foam to be aesthetically pleasing but not so much that it leaves a moustache”.) Instead, you can just say ‘the usual’. Starbucks can also increasingly do that via the app – because no matter which Starbucks you go to, if you order through the app, you can just do it with one click. And Starbucks can even analyse your choices, behaviours, and make suggestions for you.  Industrialisation in all its forms has historically created scale but lost customisation. Digitisation is allowing us to layer the customisation back over the industrial scale. This is why it’s so critical for consumer facing business to embrace this combination of service design and digital customisation.
Starbucks coffee options
We subscribe at home to a brand called Hello Fresh – they are one amongst a few who deliver ready to cook dinners. Each dinner is a dish that you’ve chosen from a menu via the site. It comes with a recipe and all ingredients pre measured and packed individually. If your tiger prawn recipe requires echalion shallot or samphire – don’t worry if you don’t have them in your fridge (let alone if like me you have to google them to learn what they are), they come in the box, in the right amounts. This too is like a luxury service but thanks to the underlying business model and the digital enablement of the ordering, menu and selection process, it can be delivered to a larger non-luxury audience.
If you look around there are dozens of places where this kind of customisation, once outside the purview of industrial models, is now back in vogue thanks to digital tools. Personal financial advisors, customised movie recommendations, configurable holidays, customised trainers – and many more. Remember though, this is not an efficiency play. It’s not enough to build a generic digital front end that will drive this mass customisation. It needs a commitment to service design to see the whole experience through the eyes of the consumer and to understand where her challenges, points of confusion, discomfort or dissatisfaction are and build the flexible digital model to address these.

So You Think The Brain is Better Than The Computer?


Every discussion on the power of computers is bracketed by the comparison to the human brain and the dwarfing of any known computer by the fantastical power of the human brain. Estimates by Ray Kurzweil suggested a calculations per second (cps) capability of 10 16 or 10 quadrillion cps. And it runs on 20 watts of ‘power’. By comparison (according to this excellent article that everybody should read) the worlds best computer today can do 34 quadrillion cps but it occupies 720 sq meters of space, costs $390m to build and requires 24 megawatts of power.

Besides, that’s just the ‘hardware’ so to say. The brain’s sophistication is far, far ahead of the computers, considering all the miraculous things it can do. We know now that the biggest evolution of the human brain was the growth of the prefrontal cortex, which required a rethink of the interior design of the skull. Also, a key facet of the brain is that it is a neural network – capable of massively parallel processing – simultaneously collecting and processing huge amounts of disparate data. I’m tapping away on a laptop savouring the smell and taste of coffee while listening to music on a cold cloudy day in a warm cafe surrounded by art. The brain is simultaneously assimilating the olfactory, visual, aural, haptic and environmental signals, without missing a beat.

It would appear therefore that we are decades away from computers which can replace brain functions and therefore, jobs. Let’s look at this a little more closely though.

The same article by Tim Urban shows in great detail how the exponential trajectory of computers and software will probably lead to affordable computers with the capacity of a human brain arriving by 2025, and more scarily, achieving the computing capacity of all humans put together by 2040. This is made possible by any number of individual developments and the collective effort of the computer science and software industry. Kevin Kelly points to 3 key accelerators, apart from the well known Moore’s law. The evolution of graphics chips which are capable of parallel processing – leading to the low cost creation of neural networks; the growth of big data, which allows these ever more capable computers to be trained; and the development of deep learning – the layered and algorithmically driven learning process which brings much efficiency to how machines learn.

So the hubris around the human brain may actually survive another decade at best and thereafter the question might not be whether computers can be as good as humans but how much better than the human brain could the computer be. But that has been well argued and no doubt will be so again, including the moral, ethical and societal challenges it will bring.

I actually want to look at the present and sound a note of warning to all those still in the camp of ‘human brain hubris’. Let me start with another compliment to the brain. Consider this discussion between two friends meeting after ages.

A: how have you been? What are you doing nowadays?

B: I’m great, I’ve been playing chess with myself for ages now.

A: Oh? How’s that? Sounds a bit boring.

B: Oh no, it’s great fun, I cheat all the time.

A: But don’t you catch yourself?

B: Nah, I’m too clever.

One of the most amazing thing about the brain is how it’s wired to constructively fool us all the time. We only ‘think’ we’re seeing the things we are. In effect, the brain is continuously short circuiting our complex processing and presenting simple answers. This is brilliantly covered by Kahneman, and many others. Because, if we had to process every single bit of information we encounter, we would never get through the day. The brain allows us to focus by filtering out complexity through a series of tricks. Peripheral vision, selective memory, and many other sophisticated tricks are at play every minute to allow to function normally. If you think about it, this is probably the brains’ greatest trick – in building and maintaining this elaborate hoax that keeps up the fine balance between normalcy and what we would call insanity. Thereby allowing us to focus sharply on specific information that needs a much higher level of active processing.

And yet, put millions of all of these wonderful brains together, and you get Donald Trump as president. You get Brexit, wars, environmental catastrophy, stupidity at an industrial scale, and a human history so chockfull of bad decisions that you wonder how we ever got to here. (And if you’re pro Trump then consider that even more people with the same incredible brain voted for Clinton). You only have to speak with half a dozen employees of large companies to collect a legion of stories about mismanagement and how the intelligence of organisations is often considerably less than the sum of the parts. I think it would be fair to say that we haven’t yet mastered the ability to put our brains together in any kind of reliably repeatable and synergistic way. Very much in trial and error mode here.

This is one of the killer reasons why computers are soon going to better than humans. In recent years, computers have been designed to network, to share, pool and exchange brain power. We moved from the original mainframe (one giant brain), to PCs (many small brains), to a truly cloud based and networked era (many, connected brains working collectively, much, much bigger than any one brain). One of the most obvious examples is blockchain. Another is in the example of the driverless car. Now, most of you might agree that as of today you would rather trust a human – (perhaps yourself) rather than a computer at the wheel of your car. And you may be right to do so. But here are two things to ponder. Your children will have to learn to drive all over again, from scratch. You might be able to give them some guidance, but realistically may be 1% of your accumulated expertise behind the wheel will transfer, from your thousands of driving hours. Also, let’s assume you hit an oil slick on the road and almost skid out of control. You may, from this experience, learn to recognise oil slicks, deal with them better, perhaps learn to avoid them or slow down. Unfortunately, only one brain will benefit from this – yours. Every single person must learn this by experience. When a driverless car has a crash today because it mistakes a sky blue truck for the sky, it also learns to make that distinction (or is made to). But importantly, this ‘upgrade’ goes to every single car using the same system or brain. So you are now the beneficiary of the accumulated learning of every car on the road, that shares this common brain.

Kevin Kelly talks about a number of different kinds of minds / brains that might ensue in the future, that are different from our own. But you can see a very visual example of this in the movie – Live Die Repeat – where the protagonists must take on an alien that lives through it’s superbrain – which is all seeing. It gets better. If, like the airline industry, automotive companies agree to share this information – following every accident or near-miss, then you start to get the benefit of every car on the road, irrespective. Can you imagine how quickly your driverless car would start to learn? Nothing we currently know or can relate to prepares us for this exponential model of learning and improvement.

It’s not just the collective, though. The super-computer that is the brain, fails us in a number of ways. Remember that the wondrous brain is fantastic as the basic hardware and wiring, and possibly, if you will allow me to extend the analogy, the operating system. Thereafter, it is the quality of your learning, upkeep and performance management that takes over, and this where we as humans start to stumble. Here are half a dozen ways in which we already lag behind computers:

Computation: This is the first and the most obvious. Our computational abilities are already infinitesimally small compared to the average computer. This should require no great elaboration. But when you apply it to say, calculating the speed of braking to ensure you stop before you hit the car that’s just popped out in front, but not so fast that you risk being hit by the car behind you, you’re already no match for the computer. Jobs that computers have taken over on the basis of computation include programmatic advertisement buying, and algorithmic trading. Another type of computation involves pattern recognition – for example checking scans for known problems, as doctors do.

Observation: Would you know if the grip on your tyres has dropped by 10%? 5%? What if your engine is performing sub optimally, or if your brakes are 3% more loose than normal? Have you ever missed a speed limit sign as you come off a freeway or motorway? Have you ever realised with a fright that there was something in your blind spot? This is a particularly obvious observation as well. A computer, armed with sensors all around the car is much less likely to miss an environmental or vehicular data point than you are. With smarter environments, you may not need speed limit signs for automated cars. All this is before we factor in distractions, or less than perfect eyesight and hearing, and just unobservant driving. Other observation based professions include security and flight navigation, where computers are already at work.

Reaction time: any driving instructor will tell you that the average reaction time is a tenth of a second for humans. In other words, at 40 mph, you will have covered 17 meters before your brain and body starts to react. By the time you’ve actually slammed the brakes or managed to swerved the car – you may well be 20-25 meters down. By contrast there is already evidence of autonomous vehicles being able to pre-empt a hazard and slow down. Even more so if the crash involves another car using the same shared ‘brain’. There is a lot of thought being given currently to the reaction time of a human take over if the autonomous system fails. This is of course a transient phase, until the reliability of the autonomous system reaches a point where this will only be a theoretical discussion.

Judgement: the problem with our brilliant brains is that we rarely allow them to work to their potential. In the US, in 2015, 35,000 people were killed in traffic accidents. Almost 3500 crashes were caused by distracted driving. Or where the driver is cognitively disengaged. There are an endless number of reasons for why we’re not paying attention when we’re driving. Tiredness, stress, anger, conversing with somebody, or worse, alcohol or being distracted by our phones. There have been studies that show that judges decisions tend to be more harsh as judges get hungry. Great though our brains are, they are also very delicate – and easily influenced. Our emotional state dramatically impacts our judgement. And yet, we often use judgement as a way of bypassing complex data processing. Invaluable where the data doesn’t exist. But with the increasing quantification of the world, we may need less judgement and simply more processing. Such as ‘Hawk Eye’ in tennis and ‘DRS’ in cricket.

Training: how long did it take you to learn to drive? A week? A month? Three? How long did it take you to be a good driver? Six months? Going back to my earlier comments – this needs to be repeated each time for each person. So the collective cost is huge. Computers can be trained much faster and do not need the experiential component one computer at a time. So in any job where you have to replace people, a computer will cut out your training time. This can include front desk operations, call centres, retail assistants, and many more. The time to train an engine such as IBM Watson has already gone from years to weeks.

So while we should agree that the human brain is marvellous for all it can do, it’s important to recognise it’s many limitations. Let’s also remember that the human brain has had an evolutionary head-start of some 6 million years. And the fact that we’re having this discussion suggests that computers have reached some approximation of parity in about 60 odd years. So we shouldn’t be under any illusions about how this will play out going forward. But I wrote this piece to point the out that even as of today, there are so many parameters along which brain already lags behind its silicon and wire based equivalent. A last cautionary point – the various cognitive functions of the brain peak at different points of our lives – some as early as in our 20s and some later. But peak they do, and then we’re on our way down!

Fortunately, for most industries, there should be a significant phase of overlap during which computers are actually used to improve our own functioning. Our window of opportunity for the next decade is to become experts at exploiting this help.

Why Are We Suddenly So Bad At Predicting the Future?

Imagine that a monkey got into the control room of the universe and spent the year clicking random buttons. Imagine him hopping about on the ‘one musician less’ button, stomping on the ‘auto-destruct’ lever and gurgling while he thumped repeatedly on the ‘introduce chaos’ switch. Close your eyes and picture him dropping a giant poo on the bright red panel marked ‘do not touch under any circumstances’. That my friends is the only way to think about 2016 – after all, it was the year of the monkey in the Chinese zodiac. It was the year when rational thinking took a beating, when meritocracy became a bad word, when liberalism escaped from the battlefield to a cave in the mountains to lick its wounds. And not surprisingly, a year when projections, predictions and polls made as much sense in the real world as an episode of Game of Thrones on steroids.
Given much of our lives are spent in productively engaging with the future and making decisions based on big and small decisions about the possible future, this last point is more important than just the schadenfreude of laughing at pollsters and would be intellectuals. The present passes too quickly, so really every decision you’ve ever made in your life is counting on future events to turn out in ways that are favourable. Getting this wrong is therefore injurious to health, to put it mildly. And yet our ability to predict the future has never been under such a cloud in living memory. Why is this so?

Fundamentally, we’re wired to think linearly in time, space and even line of sight. We are taught compound interest but we get it intellectually rather than viscerally. When you first encounter the classic rice grains and chessboard problem, as a smart person, you know that it’ll be a big number, but hand on heart, can you say you got the order of magnitude right? i.e. the total amount of rice on the chessboard would be 10x the world’s rice production of 2010? Approximately 461,168,602,000 metric tons? This problem of compounding of effects is incredibly hard to truly appreciate, even before you start to factor in all the myriad issues that will bump the rate of change up or down, or when the curve hits a point of inflexion. The Bill Gates quote  – ‘we over-estimate the impact of technology in 2 years, and under-estimate the impact over 10’ – is a direct reframing of this inability to think in a compound manner.

Then there’s the matter of space and line of sight. The way the future unfolds is dramatically shaped by network effects. The progress of an idea depends on it’s cross fertilisation across fields, geographies and disciplines, across any number of people, networks and collaborations. These collaborations can be engineered to a point or are the result of fortuitous clustering of minds. In his book ‘Linked’ – Ablert-Lazlo Barabasi talks about the mathematician Erdos who spent his life nomadically, travelling from one associates’ home to another discussing mathematics and ironically, network theory. Not surprisingly, a lifestyle also practiced for many years by a young Bob Dylan, if you substitute mathematics for music. Or consider the story of the serial entrepreneur in Rhineland in the 1400s, as told by Steven Johnson, in ‘Where Good Ideas Come From’. Having failed with a business in mirrors, he was working in the wine industry, where the mechanical pressing of grapes had transformed the economics of winemaking. He took the wine press, and married it with a Chinese invention – movable type, to create the worlds first printing press. His name of course, was Johannes Gutenberg. This kind of leap is not easy to predict, not just for the kind of discontinuity they represent (more on that later), but also because of these networked effects. Our education system blinkers us into compartmentalised thinking which stays with us through our lives. Long ago, a student of my mothers once answered a question about the boiling point of water by saying “in Chemistry, it’s a 100 degrees Centigrade, but in Physics, I’m not sure”. We are trained to be specialists, becoming more and more narrow as we progress through our academic career, ending up more or less as stereotypes of our profession. Yet human progress is driven by thousands of these networked, collaborative, and often serendipitous examples. And we live in a world today with ever expanding connections, so it’s not surprising that we have fallen behind significantly in our ability to understand how the network effects play out.

If you want to study the way we typically make predictions, you should look no further than sport. In the UK, football is a year round sport, so there are games every weekend for 9 months and also mid week for half the year. And with gambling being legal, there is an entire industry around football gambling. Yet, the average punter, fan or journalist makes predictions which are at best wilfully lazy. There is an apocryphal story about our two favourite fictitious sardars – Santa Singh and Banta Singh, who decide to fly a plane. Santa, the pilot, asks Banta, the co-pilot to check if the indicators are working. Banta looks out over the wing and says “yes they are, no they aren’t, yes they are, no they aren’t…” – this is how a lot of predictions are made in the world of premier league football today. Any team that loses 3 games is immediately in a ‘crisis’ while a team that wins a couple of games are deemed to be on their way to glory. Alan Hansen, an otherwise insightful pundit and former great player, will always be remembered for his one comment “You can’t win anything with Kids” – which he made after watching a young Manchester United side lose to Aston Villa in the 1995-96 season. Manchester United of course went on to win the season and dominate the league for the next decade and a half. Nobody predicted a Leicester City win in 2016 of course, but win they did. The continuous and vertiginous increase in TV income for football clubs has led to a relatively more equal playing field when it comes to global scouting networks, so a great player can pop up in any team and surprise the league. Yet we find it hard to ignore all the underlying trends and often find ourselves guilty of treating incidents as trends.

The opposite, is amazingly, also true. We are so caught up with trends that we don’t factor in the kinks in the curve. Or to use Steve Jobs’ phrase – the ding in the universe. You can say that an iPhone like device was sure to come along sooner or later. But given the state of the market – with Nokia’s dominance and 40% global market share, you would have bet your house on Nokia producing the next breakthrough device eventually. Nobody saw the iPhone coming, but when it did it created a discontinuous change that rippled across almost every industry over the next decade. The thing is, we like trends. Trends are rational and they form a kind of reassuring continuity so that events can fit our narratives, which in turn reaffirm our world view. And unless we’re close to the event, or perennial change seekers and nomads ourselves, it’s hard to think of countercyclical events. It’s now easy to see how in 2016 we were so caught up in the narrative of progressive liberalisation and unstoppable path to globalisation, we failed to spot those counter-cyclical events and cues that were right there in our path.

In fact there are any number of cognitive biases we are guilty of – on an everyday basis. This article just lists a dozen of them. My favourites in this list are the confirmation bias and the negativity bias. Both of these are exacerbated by social media and digital media. While social media has led us to the echo-chambers – the hallmarks of 2016, our projection bias is also accentuated by our ability to choose any media we want to consume, in the digital world, where access is the easy part. Similarly, bad news spreads faster on social networks and digital media today than at any time before in history. Is it possible that despite knowing and guarding against these biases in the past, we’ve been caught out by the spikes in the impact and incidence of a couple of these, in the digital environment we live in today?
To be fair, not everybody got everything wrong. Plenty of people I know called the Donald Trump victory early in the game. And amongst others, John Batelle got more than his share of predictions right. There is no reason to believe that 2017 will be any less volatile or unpredictable than 2016, but will our ability to deal with that volatility improve? One of the more cynical tricks of the prediction game is to make lots of predictions at many different occasions. People won’t remember all your bad calls, but you can pick out the ones you got right, at leisure! This is your chance, then, to make your predictions for 2017. Be bold, be counter-cyclical. And shout it out! Don’t be demure. The monkey is history, after all. This is the year of the rooster!

The Future of Retail: How Will You Fight Amazon?

What to do when the elephant in the room is a 600-pound gorilla?


Once upon a time, there were 4 high street electronic retailers. Now, they are one. Dixons Carphone, which also includes PCWorld and Currys, now employs some 42,000 people and manages 17 brands across Europe. Yet, while the company continues to innovate and do a lot of the things you would expect from a leading retailer, they are fighting a very different kind of opponent. Like the movie Predator, this is an almost invisible creature, capable of superhuman strength, focus, and accuracy. This is Amazon.

It’s not just retail, the story is repeating itself in other segments too. In some cases, the commercial model has changed as well – for high street music retailers, see Apple and Spotify. For Blockbusters, it’s Netflix. But for most categories, such as for Book chains like Borders, its still Amazon. And given Amazon’s relentless strategy of growth and customer intimacy before profits, its the question every retailer must ask – how to compete with Amazon?

Everyone knows a few legends about Amazon. Many are about the maniacal customer focus – how Jeff Bezos and his family spent Christmas packing gifts by hand. Or how, when asked about why Analysts weren’t buying his stock, he said that as long as customers were buying his products, he didn’t care if analysts bought the stock. The fact is that Amazon is the 600-pound gorilla in the retail business. In 2015. 50% of all e-commerce growth in the US went to Amazon.

What lies behind Amazon’s relentless growth? A combination of the obvious and perhaps less obvious. Global distribution centres, world leading warehouse automation, customer experience par excellence, recommendation engines, one-click purchasing. Kindle readers, prime membership, all you can eat subscriptions. All of this is known and well documented. But there are three key areas where perhaps less attention is paid.

First, Amazon is arguably the worlds most effective innovation company. Its string of relevant and successful innovations from automated warehouses to Amazon Echo, speak for themselves. Second: Amazon deeply understands what it means to be truly committed to an excellent customer experience – and they execute this across payments, site design, offers, delivery, and returns. Third, and most importantly it’s a digital native company. This means that all its core processes are run by software and algorithms, rather than people. Software behaves more consistently, doesn’t suffer fatigue or human errors, and can be improved relatively easily, compared to upskilling humans. Amazon can decide where to introduce human intervention rather than worry about where to automate.

Quite a few brick and mortar businesses have enjoyed success in the past decade, in the UK, through differing strategies. Tesco’s rise and fall with the Dunnhumby data business have been well documented. John Lewis continues to focus on customer service delivered via its partnership model. Halfords focuses on the cycling and travel niche. Each of these businesses will face the same Amazon question and have to figure out how to compete, especially if Amazon decides to open physical stores in future.

So How Should Brick & Mortar Stores Fight Amazon? Here’s a starting 5 point list:

  1. Dominate your segment – make sure that you define a sustainable market (e.g. kitchenware in the UK) and can be the dominant brick and mortar store in that segment, or as consolidation sets in, the last one standing.
  2. Build a strong digital proposition – one that spans the web and mobile, both deeply integrated into your business model. make sure you invest both in digital marketing and in your e-commerce platform. Exploit online communities and design around customer needs.
  3. Build powerful experiences which cannot be created online. Tactile, immersive and human experiences, which can exploit your physical store. You may redesign significant parts of your physical store and even allow customers to comparison shop and complete the purchase online, in some cases.
  4. Bring your physical and digital retail universes together – and ensure that this omnichannel experience becomes a source of data for sharpening your customer experience, in addition to contributing to your sales and profits.
  5. Automate your core processes – from merchandising, offers, check out, payment, delivery and returns, and then focus specifically on where human inputs will improve the process. Invest in developing algorithms that are valuable to your business.

Of course, this is only a beginning and you’ll need to keep investing and building competence in any number of new areas. Some that spring to mind include: trust models, building strong data stewardship, creating a lifetime value of customers, providing technical support for the increasingly smart products you’re likely to be stocking, creating new commercial models – perhaps around the idea of leasing or renting rather than outright purchase, understanding immediacy and real-time business models, advanced security modelling, designing of smart experiences, and deep supply chain visibility – these are just some of the areas you will want to ensure you understand well.

We should also expect to see other market patterns emerge – for example, corner shops/ convenience stores could be pulled together with a common platform which allows them to run independently but provide a shared platform for online & mobile ordering, stocking, supply chain and even leasing drones for delivery. After all, you would go to your corner shop when you need something quickly – when your sugar runs out, in the middle of making tea, for example. What better way for them to deliver to these urgent needs by having drones drop a packet of sugar to your doorstep even before you finish making the tea?

Because of course, if corner shops won’t do it, and the high street groceries dilly dally, this is something Amazon are already planning to do. And frankly, as a consumer, I’ll go to whoever meets my needs in the most painless way.

The Missing Pieces of Innovation

I realised something very important while I was a the innovation event organised by EditorEye at the General Assembly recently. It became clearer to me why, despite spending a lot of time and effort on innovation and hiring some excellent people, organisations are still struggling to get the results out of innovation. The speakers at the event, by the way, were all very good and were all covered the topic extensively. But there are some aspects of innovation which are simply not talked about, while others get a lot of focus, as I’ll show you later in this piece.


But the problem, I believe starts at the root. What is innovation? You can get as many definitions as there are people in the room. Is it new product development? Is it new ideas? Is it creativity? Is it bringing ideas to market? My definition is simple – it’s doing more with less. If it takes 100 units of resources to solve a problem, and you figure out a way of doing it with 80, that’s innovation. We can debate this later, but let me ask you a different question. Is all product development innovation?

Let’s suppose you are an insurance company and you figure out that increasingly there are older people whose lives depend extensively on technology. You survey the market, and you create a new product which looks at a comprehensive technology devices cover for this audience. You create an app and website for it, which is designed to be used by older people – simpler interfaces, larger fonts, etc. You spend time with prospective customers to understand their specific behaviour and problems and design your product to deliver their key unarticulated needs. Your product is a success. Was this an example of innovation? Let’s assume this product was prototyped in your ‘innovation lab’ which has been set up for bringing such new products to market. Still, is it innovation? Which part of this is innovative, given that all these processes are now standard practice for product development? It may be an excellent example of product development, but I repeat, is it really innovation?

Similarly, you might argue, is every successful advertising campaign ‘innovative’? Is every market research that delivers customer insight an act of innovation? What about a new business, or start up? The reality is that you’re perfectly justified in saying yes to all of these questions. And if your innovation lab delivers successful products, then surely it’s justified, irrespective of what you call it. And I’m not disagreeing with that.

However, to miss the point of innovation is also to walk away from a lot of value. Let’s suppose the typical new insurance product costs £2m to develop and test and £10m to market, hypothetically speaking. Let’s say that the numbers here represent the comprehensive resource cost, and not just actual cash outflows. Now if your new product took about the same, and was averagely successful, you’re at par. But what if you could deliver similar success with 20% investment? Or much better returns? And what if you could build a methodology for doing this consistently? That’s innovation! That means you can deliver more for less and on a consistent basis.

Most organisations look at innovation as a way of delivering new products, or new businesses. It’s also common to look at it as a process that follows a standard path: brainstorm ideas, prototype in a lab and then scale through the organisation. I challenge both of these premises. First, looking at innovation as an idea to market process limits our thinking. Innovation needs to be seen as a problem solving methodology. And specifically, a methodology that looks to improve on the expected resource requirement for solving the problem. And to the second, if everybody is replicating this model, then it stops being innovative. Not that it becomes less valuable if the new products work. But the reality is that most innovation initiatives in most companies don’t lead to success. And wouldn’t it be great if we could increase the success rate?

My model of innovation therefore starts earlier, with Problem Definition. If you’re thinking of a new product, why? Is it to ensure coverage of the market? is it opportunistic? Do you believe that there will decline in current products & revenues? Is it a strategic response to competition? Do you feel you underserve the market? Is the problem financial – your return on capital is too low and you’d like to improve this? Are you missing out on more profitable customers or a growing segment? As you can see there are many, many ways of framing your new product development effort and the problem you’re trying to solve may vary significantly. 3M for example, has a commitment to drive 30% of revenues from ‘new products’ – i.e. those built in the last 4 years. For a pharmaceutical company a new product that better addresses a disease or a family of problems, is a protectable revenue stream that can run for over a decade, even as older revenues decline through patent expiration. Whereas Google (Alphabet) just wants to solve bigger problems. That allows it to be mission driven but even there, for example there are specific problems. Arthur Levinson, ex CEO of Genentech leads a platform in Alphabet to combat ageing. Whereas in the new famous example of British Cycling, the marginal innovations are aimed at driving higher performance, and nothing to do with a product at all. This is where we step away from product development and recognise that innovation is a methodology for solving any problem in a ‘do more for less’ way, not just product development. To do this well, try approaching this problem as at least two out of a CFO, a CEO and a head of Operations, or Marketing. Or apply design thinking principles to see how the people impacted by this problem think about it. This award winning redesign of the ambulance started by looking at the ambulance as an extension of the hospital, and the start of the medical process, rather than just a form of transportation to the hospital. Kees Dorst’s book ‘Frame Innovation’ is a good starting point for thinking about problem framing.

The next step is the Research and Baselining phase, so we know what benchmark we’re trying to beat. It is likely that your ambitions are not at the same scale as Google’s (or Alphabet’s), Amazon or Elon Musk’s. In fact you may just be looking to solve an punctuality problem in your department in a way that nobody in your company has done yet. If you define your context as your company, department, industry or the world, you can accordingly set your benchmarks. This is critical because what’s innovative for a government agency (say agile development) could be very old hat for a Silicon Valley company. But this my second key point. Innovation is surely about being different, and not replicating a tried and tested method. So you need to clearly set out what you’re going to do differently (better!) from other similar efforts, before you start. It’s worth noting that of the 5 stages, this gets the least amount of attention because it’s probably the least sexy part of innovation. But it could save you a lot of effort and also dramatically sharpen the subsequent phases. In fact, often, you will be able to find a lot of examples workable ideas in other industries and organisations. No better example of this than the Great Ormond Street Hospital for children learning from Formula 1 pit stops, about swift handovers from surgery to intensive care. This represents a huge reduction of risk, in the innovation journey.

It’s only once you’ve done stages 1 and 2 that you should then get to the Creative Spark stage. For most organisations, this equates to a brainstorming exercise. One of the biggest mistakes in the area of innovation is that people jump into brainstorming with a loosely defined problem and no benchmark. To make it worse, you then get a lot of people with very little knowledge of the context of the problem state coming up with ideas many of which clearly won’t work. I know of a company which was keen to ‘pitch’ ideas to Transport for London. They ran a competition internally, generated hundreds of ideas, evaluated them and drew up a short list of 10. But the brainstorm was run with a global team, not based in London, and consequently many of the ideas, such as mobile app based solutions for contactless ticketing did not factor in the actual challenges of rush hour volumes, or the speed of response required. Besides, many of the ideas were already at play at TFL, which hadn’t been researched well enough. Running pure-play brainstorms is also of limited use if the team doesn’t have enough context of the problem. You can’t brainstorm ways of improving care pathways in the NHS, or supply chains for broadcast equipment, if you don’t know much about them, or the problems they face. There are, however, plenty of techniques for running more effective brainstorms and idea sessions. Additionally, there are other ways apart from brainstorming for the creative spark phase. Best results are often achieved through having creative people in the mix along with experts, or building unpredictability into the process. Tim Harford’s book ‘Messy’ suggests some excellent ways in which this happens.

Once you have ideas you want to take forward, you can then push them through the Innovation Lab stage. Of the 5 phases of this methodology, this is the one most organisations have invested in already and are doing with a lot of focus. Setting up a lab environment, running ‘Google sprints’, ensuring that small teams turn around quick prototypes, building design thinking into the mix and fusing the efforts of creative technologists with deep experts, a lot of companies are able to do a reasonably good job of taking new ideas through a laboratory process to an MVP stage. When I was working at Cognizant, in 2015 we conducted a quick research of ‘innovation labs’ and were not surprised to find that an overwhelming majority of leading banks and retailers already had an innovation lab of some kind. If you haven’t yet been exposed to or been a part of an exercise like this, grab hold of ‘Sprint’ by Jake Knap et al.

But even that is not enough because a lot of initiatives can fail even after lab success. Be they new products or internally facing solutions. Scaling innovation is fraught with risk, and even Google is famous for the number of initiative it has killed after promising starts. This is the key reason that many organisations prefer to buy in the finished product rather than try to build it in house. What the newly created and lab-tested idea needs is not just organisational support, but often a network in which to flourish. The best results are created when the new idea has a life of it’s own and is allowed to grow and morph independently, not simply scale to a larger replica of it’s initial form. The perfect baby needs to grow into a healthy human adult, not a full sized replica of the baby. Most businesses are unable to provide this kind of sustaining network. Steven Johnson’s excellent book ‘Where Good Ideas Come From’ beautifully elucidates this idea of a sustaining networks. When GE set up it’s fledgling IOT business in Silicon Valley, it was not just allowing it to flourish outside of the corporate headquarters, but also allowing it to sustain and nourish itself in a high tech network. In organisations such as Google and 3M, there isn’t a small and tightly defined number of ideas being pipelined to the market, there is a huge internal innovation network, where dozens or even hundreds of ideas feed off each other, combine and morph on their way to a handful becoming successful products. If it’s new product and new business development that defines innovation for you, then you could do well to keep at hand the Innovators Solution, by Clayton Christiansen.


This is just the tip of the iceberg, in a way. Innovation is hard work, and much of it is done away from the public eye, and the adulation of success. But more importantly, innovation is a methodology, which when applied, dramatically improves your ability to problem solve in a way that is ahead of the competition.