(In this first installment of a two-part conversation with Stefan Thomke, Professor of Business Administration, Harvard Business School,we learn why building an "experiment-with-everything" business culture can improve customer experience, accelerate new product development and unleash the power of innovation in your organization. Read part 2 here.)
When shift happens, when industries are turned upside down by disruption, many executives retreat to the comfort zone of "best guess" innovation based on experience. But even the most experienced shot callers often get it wrong. The problem with "best-guess" innovation is that it just doesn't cut it anymore.
So says Harvard Business School Professor Stefan Thomke, a leading authority on business experimentation and author of the forthcoming book: Experimentation Works: The Surprising Power of Business Experiments.
For starters, says Thomke, the first question a company should ask when evaluating new technology or an innovative idea is whether it has staying power. Figure out if it's ready for prime time and whether it creates business value. If the answer is yes and you ignore it, it could be lethal to your business.
This is why digital leaders such as Amazon, Booking.com, Facebook, and Google conduct more than 10,000 online controlled experiments annually, with many tests engaging millions of users. Legacy brands such as Walmart and Hertz and Singapore Airlines also run experiments on the regular.
So, what do all of these organizations have in common? Experimentation is a function of their success. Which confirms that an "experiment-with-everything" approach can have surprisingly large payoffs.
Thomke says that experimentation has helped Microsoft's Bing, for instance, identify numerous changes which have collectively increased Bing's revenue per search by 10% to 25% per year.
Other experiments have increased user satisfaction, boosted profitability and more than doubled Bing's share of PC-based searches in the U.S. alone.
In this insightful Digital Masters interview, Thomke takes us to school on how to turn business experimentation into a powerful competitive advantage. Read and enjoy.
Appian: So, one of the topics you've written extensively about is the importance of experimentation in business. What impact has digital transformation had on the urgency of experimenting to find innovative ways to create customer value?
Thomke: This is already a big deal and it's going to be an even bigger deal going forward. If you don't mind, let me take a step back and talk about why we should experiment to begin with. And then we can talk about how digital is changing the way this is actually being done. So, there's a revolution underway in innovation.
Appian:You mentioned innovation and there's a certain amount of ambiguity associated with that. How do you define innovation?
Thomke: When I say innovation, I mean it in the broadest sense: developing new products and services, customer experience, business models and so on.
And one of the fundamental problems in innovation is uncertainty. Uncertainty is the reason that companies often don't get it (innovation) right. And the odds are stacked against them.We have lots of data that shows that you're much more likely to get it wrong than right.
So, in a sense, innovation has turned into a bit of a guessing game. And what do executives do when they face uncertainty? They often fall back to their experience.
https://twitter.com/hbsexeced/status/788380919778385920
Appian: Because experience feels safe?
Thomke: Yes, but experience can really get in the way, especially when things are changing around you. Then you have the argument of well, we've got lots of data and can't we just look at the data.' The problem is that there's less data around innovative things. Otherwise, it wouldn't be very innovative. So that's one big issue. And the second problem is that context matters.
That is, sometimes data in one context or experience doesn't really transfer to another context as we've seen when Ron Johnson who ran the retail model at Apple, switched over to JC Penney and tried to replicate it over there. But they later they did a U-turn and Johnson was out of a job.
And finally, the third problem is and I really want your readers to understand this is the difference between correlation and causation. When you're a manager, you want to know when an action leads to a desired outcome. When we do data analysis, we get correlations but that doesn't necessarily mean causation.
Appian: Can you break that down some more?
Thomke: You can show, for example that ice cream sales correlate very strongly with drowning deaths. Or you can talk about the correlation between palm (hand) size and life expectancy and the fact that women have smaller palms and tend to live longer than men, and so on and so on. The point is that correlation is not causation.
So, when you go back to traditional ways of dealing with uncertainty, the solution is to run rigorous experiments because experiments allow you to get at answers to many of these questions. And if they're done rigorously, they will actually tell you something about cause and effect.
It (rigorous experimentation) changes decision making and it turns innovation from a guessing game into a scientific approach to management.
Appian: Which brings us back to digital transformation and why it's such a hot topic these days.
Thomke: I've been studying this for more than 20 years now. When you look at digital and online, companies now have this amazing power at their disposal. They can take this online capability and run massive numbers of experiments. Employees can go online and actually improve user experience.
But instead of guessing about how to do it, they can actually go online and instantly test an idea on millions of people within hours and get a fairly reliable result within days. And that kind of agility fundamentally turns management on its head.
We like to think that there's a hierarchy to this, right?
People come up with ideas, then we run it through committees and decide whether it's worthwhile pursuing. And then executives use their wisdom to decide whether this is something we're going to do. But the reality is that even executives are wrong most of the time.
https://vimeo.com/248600321
Appian: What can companies do to build this experimentation capability?
Thomke: So, there are a number of options available. It starts with the tools, the infrastructure the platform, right? Because you need a platform to do this in the online world. What some leading companies have done in the last five to ten years is invest enormous amounts of time and money in building internal platforms.
Whether you look at an Amazon, Facebook, Google, Microsoft, Netflix or Bookings.com and other companies that operate in the online world, they've invested massively in platforms.
And many of these companies run more than 10,000 experiments per year. Bing, which is part of Microsoft, runs more than 15,000 experiments per year. And Microsoft runs way more.
Appian: So, what do you do if you don't have the resources to do massive amounts of experimentation?
Thomke: Well, there are third-party tools out there that provide platforms to their customers which means you don't have to build your own infrastructure. So that's first, the platform itself is necessary. The second thing that's important is the management side of it. And perhaps even more important, you now have to build a culture and a governance structure in your company that allows this to happen.
If you have a complicated governance structure where an experiment has to go to a committee, then a legal department and so on, there's no way you're going to unleash this kind of innovation.
So, you have to think about adapting your governance to accelerate innovation. It's all about the velocity of your experiments that really matters.
(Be sure to tune in next week for the final installment of this two-part series on how to survive and thrive in the pressure cooker of digital transformation when experience is not enough.)
