We’ve long associated innovation breakthroughs with science and technology coming out of R&D labs, e.g., the transistor, penicillin, DNA sequencing, TCP/IP protocols, and so on. Such major lab-based breakthroughs are at one end of the innovation spectrum. At the other end are market-facing innovations, whose purpose is to create appealing and intuitive user experiences, new business models, and compelling market-based strategies.
Lab-based innovations were generally born when scientists, mathematicians or engineers developed new theories, technologies, algorithms or programs in an R&D lab. Over time, often years, the innovations found their way to the marketplace. Since technology and markets advanced at a relatively slow pace, there was little pressure to reduce the transition times from lab to market. This was the prevailing innovation model through most of the 20th century.
It all started to change in the 1980s as the rate and pace of technology advances significantly accelerated. The hand-offs and elapsed times to take an innovation from lab to market were no longer competitive, especially with products based on fast changing digital technologies. Start-up companies significantly shortened the time-to-market for new products and services, putting huge pressure on companies still operating under the old rules.
These competitive pressures, were further exacerbated by the explosive growth of the Internet in the 1990s, as I personally learned when becoming general manager of the newly established IBM Internet Division in December of 1995. A lot was starting to happen around the Internet, but it was not clear where things were heading, and in particular what the implications would be to the world of business. With the Internet, there was no one technology or product you could work on in the labs that would make you a success in the marketplace. This time around, the strategy itself had to come from the marketplace, not the labs.
Embracing the Internet turned out to be much more than a technology change. It had a very big impact on the overall culture of IBM and many other companies, paving the way for a more outside-in approach to innovation based on continuous marketplace experimentation.
In his 2020 book Experimentation Works: The Surprising Power of Business Experiments, Harvard professor Stefan Thomke explains the major changes that’ve been taking place in innovation and experimentation over the past 20 - 25 years.
“Innovation is important because it drives profitable growth and creates shareholder value,” wrote Thomke. “But here is the dilemma: despite being awash in information coming from every direction, today’s managers operate in an uncertain world where they lack the right data to inform strategic and tactical decisions. Consequently, for better or worse, our actions tend to rely on experience, intuition, and beliefs. But this all too often doesn’t work. And all too often, we discover that ideas that are truly innovative go against our experience and assumptions, or the conventional wisdom. Whether it’s improving customer experiences, trying out new business models, or developing new products and services, even the most experienced managers are often wrong, whether they like it or not.”
The answer to this dilemma is continuous experimentation. In the 21st century digital economy it’s possible for companies to design and conduct a large variety of experiments swiftly, inexpensively and at scale. But, learning how to do so effectively is still a work in progress. As is the case with lab-based experiments, there’s no guarantee that each market-facing experiment will be a success, especially on the first go-round. “Indeed, the failure rate of experiments can be 90 percent or higher - whether they are conducted by a single scientist, a world-famous laboratory, a marketing department, or business strategists.”
However, as has long been the case with the tried-and-true scientific method, every experiment generates valuable information, whether the hypotheses and predictions being tested turned out to be true or not. If we didn’t get the result we were hoping for, we can analyze why the experiment didn’t work as expected, what assumptions we made that turned out not to be valid, and most important, what did we learn for the next set of experiment. Most progress, especially in market-facing innovation, is achieved through the cumulative impact of many relatively minor experiments.
What constitutes a good experiment? To help address this question, the book recommends that companies ask themselves a series of interrelated questions that are not so simple to answer. These include:
- Does the experiment have a testable hypothesis? Formulating the proper question or hypothesis is one of the most important aspects in the scientific method.
- Is there a commitment to abide by the results? That is, what kind of experimental results would cause the organization to change its mind, if any?
- How can we ensure that the results are reliable? Does the organization have the proper talent and skills to conduct an effective and reliable experiment whose results can be trusted?
- Do we understand cause and effect? As we’re often reminded, correlation does not imply causation. This is an area where skills and experience are very important.
- Finally, has the organization truly embraced experimentation? Are important decisions being driven by the work of experimentation?
Principles of experimentation. Thomke offers a set of essential principles for successful experimentation learned from his experience working with companies over the years.
- Test everything that can be tested. Remember that the failure rate of experiments can be 90% or higher, so it’s important to conduct a wide variety of experiments that test different ideas and hypotheses.
- Often, small innovations can be very valuable. While we generally glorify highly disruptive ideas, a series of seemingly minor, incremental changes can end up having a big impact.
- Trust in the experimentation system. Often, no matter how convincing the results, there’s no guarantee that everyone will accept them unless they truly trust the integrity of the system.
- Results must be easily understood. Simplicity and rigor are fundamental, so that everyone can easily understand what the experiment is about and how it was conducted.
The role of culture. “If testing is so valuable, why don’t companies do it more?,” asked Thomke in “Building a Culture of Experimentation,” a related Harvard Business Review article published shortly after the book. “After examining this question for several years, I can tell you that the central reason is culture. As companies try to scale up their online experimentation capacity, they often find that the obstacles are not tools and technology but shared behaviors, beliefs, and values. For every experiment that succeeds, nearly 10 don’t - and in the eyes of many organizations that emphasize efficiency, predictability, and ‘winning,’ those failures are wasteful.”
The article discusses several key characteristics of a successful experimentation culture, including cultivating curiosity, insisting that data trumps opinion, democratizing experimentation across the organization, being ethically sensitive, and embracing a leadership model that will follow test results wherever they lead. In the book, Thomke further adds that management plays a critical role. “One takeaway from examples and research is the perhaps unsurprising idea that management counts; that is, when managers actively encourage experimentation, the culture invites experiments. And when ‘failure’ is understood as contributing to learning (i.e., not punished), experimentation is encouraged as well.”
Finally, Thomke makes some tentative predictions about the future of business experimentation. “The epilogue is a heads-up: the future will be exciting and deeply challenging. Combining large-scale testing capabilities with advances in artificial intelligence, big data (which we will have learned how to judiciously use), and evolutionary algorithms may just kick things up another level. The result may be a closed loop process where the generation, testing, and analysis of business hypotheses becomes fully automated.”
Comments