Irving Wladawsky-Berger

Data-Driven Decision Making: Promises and Limits

Decision making, - how individuals, groups and organizations go about selecting a course of action among several alternative scenarios, - has long been a subject of study.  Given the explosive growth of big data over the past decade, it’s not surprising that data-driven decision making is one of the most promising applications in the emerging discipline of data science. 

In a recently published article, Data Science and its Relationship to Big Data and Data-Driven Decision Making, Foster Provost and Tom Fawcett succinctly define data-driven decision making as “the practice of basing decisions on the analysis of data rather than purely on intuition.”  Equally succinctly, they view data science “as the connective tissue between data-processing technologies (including those for big data) and data-driven decision making.”

Looking deeper at data-driven decision making, it’s important to understand not only the promise but also the limits.  When can we embed decisions into well understood, automated processes?  When does automation run into limits, and we should view data-driven decision making as a tool to help people make smarter, more effective decisions?  And what are the prospects for its future, as technology, big data and data science continue to advance?

The Firm as a Large, Complex, Extended Family

Ronald Coase, the eminent British economist and University of Chicago professor emeritus, passed away on September 2 at the age of 102.  Professor Coase was one of the best known economists of the past 80 years.  He lived a long and extraordinarily productive life until the very end.  He is best known for two articles, The Nature of the Firm published in 1937, and The Problem of Social Cost published in 1960.  These along with many other achievements earned him the 1991 Nobel Prize in economics. 

A recent Economist article, One of the Giants, summed up Professor Coase life’s work in his own words:  “I have made no innovations in high theory. . .  My contribution to economics has been to urge the inclusion…of features of the economic system so obvious that…they have tended to be overlooked.”  “Attention to the overlooked helped Mr Coase transform both law and economics,” the article added. 

A second article in the same September, 2013 Economist issue, The man who showed why firms exist, said:

“The job of clever people is to ask difficult questions.  The job of very clever people is to ask deceptively simple ones.  Eighty years ago a young British economist wondered: why do companies exist?  The answer that he gave remains as fascinating today as it was back then. . . Most economists had been content to treat firms as black boxes.  Mr Coase wondered what the black boxes were doing there in the first place.  He used a scholarship that he won as an undergraduate to visit leading American firms such as Ford and General Motors.  He summed up his thinking in his 1937 essay, The Nature of the Firm, which at first attracted no attention whatsoever, but continues to be cited to this day.”

Reflections on the Education of Applied Data Scientists

About a year ago, an article in the Harvard Business Review called data scientists the sexiest job of the 21st century.  Its authors, Tom Davenport and D. J. Patil, succinctly defined data scientist as “a high-ranking professional with the training and curiosity to make discoveries in the world of big data.”

For over ten years, we’ve been using the term big data to refer to the fast growing volumes and varieties of digital data being collected, much of it in real-time.  Over the years, we’ve made considerable progress in the storage and management of these big data sets, helped along by major advances in computer science, math, statistics and related disciplines. 

But all that progress is of limited value without people with the skill sets to extract important insights out of all that data.  The promise of data science is that big data will lead to significantly better decisions and predictions, to the smart management of social organizations like cities, companies and economies, and to research breakthroughs in the social sciences, medicine and a number of other disciplines.  And, on that front, most everyone agrees that people with the necessary skills are scarce.  Demand for data scientists has raced ahead of supply, primarily because there have been few university programs training students in this emerging discipline. 

But, the situation is rapidly changing.  A recent NY Times article observed that  “Universities can hardly turn out data scientists fast enough. . . Because data science is so new, universities are scrambling to define it and develop curriculums.  As an academic field, it cuts across disciplines, with courses in statistics, analytics, computer science and math, coupled with the specialty a student wants to analyze, from patterns in marine life to historical texts.”

A Kind of Digital “Perfect Storm”

Every day brings new stories about the impact of digital technologies on every aspect of the world around us, including their transformative impact on companies in just about every industry.  Some of the articles are about the creation of innovative new technologies and applications, while others are about the creative destruction such innovations leave in their wake.  Lately, the latter kind of stories seem to predominate.

For a while now, companies have been racing to try to keep with the relentless advances of digital technologies.  But, the race keeps getting tougher every year.  We seem to be heading toward a kind of digital perfect storm, where four major IT trends are each gathering speed while interacting with and amplifying each other: mobile devices, cloud computing, social networks and big data and analytics.

These widely discussed technologies are still in the early stages of deployment to a greater or lesser degree. Mobile is furthest along while cloud adoption is taking off.  Data science applications are starting to get deployed, although we are just beginning to appreciate their profound implications.  And while consumer social networks have been thriving for years, their use in business has been slower than expected. 

The disruptive impact of this digital perfect storm is already being felt by technology companies as well as by those in industries whose transformation is already underway.  And perhaps most important, we know that even greater disruptions are in store in the not too distant future.  The resulting creative destruction makes for interesting stories about the changing fortunes of companies as well as their leaders.

Becoming a Digital Business: Progress and Obstacles

A few weeks ago, McKinsey released Bullish on Digital, a short report on how companies are faring with digital technologies and strategies based on a recent online survey of 850 CEOs and other senior executives.  The survey asked the executives about their companies progress and obstacles in embracing five major digital trends: big data and advanced analytics, digital engagement of customers, digital engagement of employees and external partners, automation, and digital innovation.  

Adapting to the ongoing digitization of the economy, and of society in general, is arguably the most challenging transformation every business is facing, so it’s quite interesting to see what the McKinsey survey uncovered. 

All-in-all, executives feel that their companies are, at best, about one-quarter of the way toward their digital business objectives.  Organizational alignment and leadership are the critical factors in the success or failure of their digital strategies.  But, despite the organizational and leadership challenges, they are generally optimistic that their companies are making progress and more fully embracing digital capabilities.