A few weeks ago, McKinsey published The age of analytics: Competing in a data-driven world, a comprehensive report on the state of big data, and in particular, on the challenges and opportunities a company faces as it strives to become a data-driven business.
Big data,- including analytics, data science, artificial intelligence and related information-based technologies - are now everywhere. “Is big data all hype?,” asks McKinsey in the report’s introduction. “To the contrary: earlier research may have given only a partial view of the ultimate impact… the range of applications and opportunities has grown and will continue to expand. Given rapid technological advances, the question for companies now is how to integrate new capabilities into their operations and strategies - and position themselves in a world where analytics can upend entire industries.”
It’s very hard to anticipate the consequences of disruptive innovations. Even harder is predicting when the impact will be felt across the economy and society. It’s often been said that truly transformative innovations are overhyped in the short term but under-hyped in the long term. Think of electricity and automobiles, the Internet more recently and now big data.
I have personally lived through the rise of the Internet. In the Fall of 1995, IBM made the decision to embrace the rapidly growing Internet and created an Internet Division with me as general manager. A lot was happening at the time. It was all very exciting, but it wasn’t clear where things were heading, and in particular, the implications to the world of business. A major part of our job was figuring out the business value of the Internet for our clients as well as for IBM, that is, why should every business embrace the Internet and become what we called an e-business.
The dot-com era was famous for its many innovations, but also for its hype. Many believed that an Internet-based new economy was imminent, - with eyeballs as the key business metric instead of revenue, profit and cash, and where physical assets were no longer relevant, giving dot-com startups an inherent advantage over legacy companies.
Twenty years later we’ve indeed seen the emergence of an Internet-based digital economy, more powerful than anything we envisioned in the mid-late 1990s. The universal reach and connectivity of the Internet has been disrupting and transforming every single industry, especially since the advent of smartphones, IoT devices, cloud computing and broadband wireless networks over the past decade.
Big data has been rising rapidly for the simple reason that there’s so much more digital information than ever before. In 2000, only one-quarter of the world’s stored information was digital and thus subject to search and analysis. Since then, the amount of digital data has been doubling roughly every three years. By now only a small amount of all stored information isn’t digital, around 1% or so. This could not have possibly happened without the digital revolution, which has drastically lowered the costs of collecting, storing, analyzing and sharing the oceans of information we now have access to.
But big data should not only be framed as part of the digital revolution of the past few decades. It’s also a major part of the scientific revolution of the past few centuries. Scientific revolutions are launched when new tools make possible all kinds of new measurements and observations, e.g., the telescope, the microscope, spectrometers, DNA sequencers. Our new big data tools now have the potential to usher an information-based scientific revolution across many disciplines.
“Ultimately, big data marks the moment when the information society finally fulfills the promise implied by its name,” wrote Kenneth Cukier and Viktor Mayer-Schönberger in their 2013 article The Rise of Big Data: How It's Changing the Way We Think About the World. “The data take center stage. All those digital bits that have been gathered can now be harnessed in novel ways to serve new purposes and unlock new forms of value. But this requires a new way of thinking and will challenge institutions and identities.”
As was the case with the Internet 20 years ago, it will take time for companies to learn how to best leverage big data for business value. The journey to become a true data-driven-business is likely even harder than the e-business journey, given that leading technologies like machine learning are advancing so rapidly, and the required talent is still relatively scarce.
In an earlier 2011 report, McKinsey made a number of predictions on the transformational potential of big data. Their new 2016 report takes an in-depth look at the progress made in the intervening years. “Five years later, we remain convinced that this potential has not been overhyped. In fact, we now believe that our 2011 analyses gave only a partial view. The range of applications and opportunities has grown even larger today. The convergence of several technology trends is accelerating progress. The volume of data continues to double every three years as information pours in from digital platforms, wireless sensors, and billions of mobile phones. Data storage capacity has increased, while its cost has plummeted. Data scientists now have unprecedented computing power at their disposal, and they are devising ever more sophisticated algorithms.”
Let me briefly summarize the report’s key findings.
Most companies are capturing only a fraction of the potential value of data and analytics. “Turning a world full of data into a data-driven world is an idea that many companies have found difficult to pull off in practice.” Adoption is particularly lagging in the public and healthcare sectors, due to incentive problems and regulatory issues.
Legacy companies have to overcome hurdles to accelerate their analytics transformation. A number of companies have responded to competitive pressures by making large technology investments in data and analytics. Yet, they’ve failed to make the organizational changes required to realize the full value of these investments, - from clearly articulating the business value of data and analytics to leveraging data-driven insights to improve decision-making.
There is a continuing shortage of analytics talent. “Across the board, companies report that finding the right talent is the biggest hurdle they face in trying to integrate data and analytics into their existing operations… Data scientists, in particular, are in high demand.”
Analytics leaders are changing the nature of competition and consolidating big advantages. “There are now major disparities in performance between a small group of technology leaders and the average company - in some cases creating winner-take-most dynamics.”
The value of data depends on how it will be ultimately used, - and by whom. “A piece of data may yield nothing, or it may yield the key to launching a new product line or cracking a scientific question. It might affect only a small percentage of a company’s revenue today, but it could be a driver of growth in the future.”
Six disruptive data-driven models and capabilities are reshaping some industries - and could soon transform many more:
- Business models enabled by new types of data sets to supplement those already in use;
- Hyperscale platforms that use data and analytics to connect sellers and buyers in real time and on an unprecedented scale;
- Micro-segmenting a population based on individuals’ characteristics as revealed by data and analytics;
- Bringing together and integrating all the relevant data to address a given problem;
- Fueling discovery and innovation; and
- Leveraging advanced algorithms to support and enhance human decision-making.
The frontiers of machine learning, including deep learning, have wide-ranging potential to help solve problems in every industry. “Conventional software programs are hard-coded by humans with specific instructions on the tasks they need to execute. By contrast, it is possible to create algorithms that learn from data without being explicitly programmed…. In short, these systems are trained rather than programmed.”
“Data and analytics are already shaking up multiple industries, and the effects will only become more pronounced as adoption reaches critical mass - and as machines gain unprecedented capabilities to solve problems and understand language. Organizations that can harness these capabilities effectively will be able to create significant value and differentiate themselves, while others will find themselves increasingly at a disadvantage.”