Irving Wladawsky-Berger

Data Science: from Half-Baked Ideas to Data-Driven Insights

In a Big Data presentation last year, MIT professor Erik Brynjolfsson pointed out that throughout history new tools beget revolutions.  Scientific revolutions are launched when new tools make possible all kinds of new measurements and observations.

Big data is leading to such a measurement-driven revolution, brought about by the new digital tools all around us, including our mobile phones; searches and web links; social media interactions; payments and transactions; and the myriads of smart sensors keeping track of the physical world.  These digital tools are enabling us to collect massive amounts of information on who we are, what we do and how we interact as individuals, communities and institutions.

The explosive growth of big data is in turn giving rise to data science, one of the most exciting new professions and academic disciplines.  Data science is a mashup of several different fields.  Its data part deals with acquiring, ingesting, transforming, storing and retrieving vast volumes and varieties of information.  Its science part seeks to extract insights from the data by applying tried-and-true scientific methods, that is, empirical and measurable evidence subject to testable explanations and predictions. 

One of the most exciting part of data science is that it can be applied to many domains of knowledge, given our newfound ability to gather valuable data on almost any topic. But, doing so effectively requires domain expertise to identify the important problems to solve in a given area, the kinds of questions we should be asking and the kinds of answers we should be looking for, as well as how to best present whatever insights are discovered so they can be understood by domain practitioners in their own terms.

Innovation Hubs in the Global Digital Economy

Last week I wrote about Cloud, Services and the Transformation of Production, based on Part I of Escape from the Commodity Trap: Will the Production Transformation Sustain Productivity, Growth and Jobs?, a paper by UC Berkeley professor John Zysman.  Part II of the paper deals with the economic and policy implication of this ICT-based production transformation. 

“For the advanced industrial democracies to expand the real income of the citizens and sustain growth in employment and productivity, their economies will have to escape from the commodity trap,” writes Zysman in the paper’s abstract.  “The commodity trap is the price-based competition throughout markets for standard goods and services, which puts pressure on wages and profit margins alike.  Clearly, the way out of this trap is to create distinctive high value added products  - both goods and services.  The emerging transformation of the production of goods and services is dramatically altering what is produced, where, how, and who captures the value.  It creates opportunities and challenges.” 

Early in the Industrial Revolution , the so-called Luddites smashed the new textile machines that were threatening their jobs.  Zysman reminds us that while many of the Luddites lost their jobs, the overall English economy prospered and the average standard of living went up. “It is easier to identify the jobs that are destroyed,. . .  than to imagine the ways new jobs will be created.  Of course, the character and location of those jobs may advantage some communities and disadvantage others.” 

Disruptive innovations, - like the power looms allegedly smashed by protesting Luddites, - have been displacing human labor for the past two centuries.  These periods of creative destruction, - when new technology-based innovations led to painful transitions in once-dominant companies and jobs, - eventually worked themselves out.  Over time, these same disruptive innovations rejuvenated the economy, and led to the creation of new industries and jobs.

Cloud, Services and the Transformation of Production

Ever since cloud computing first appeared in the IT world, people have struggled to define what it is and why it’s so important. “There is a clear consensus that there is no real consensus on what cloud computing is,” said the organizer of a 2008 cloud conference in his closing remarks.  Just about everyone agreed that something big and profound was going on, although we were not totally sure what it was yet. 

Is cloud the 21st century version of time sharing, enabling users to get virtual access to different kinds of IT resources and software capabilities without having to own computers?  Is it a return to centralized computing, driven by the rising complexities, management costs and energy inefficiencies of distributed systems?  Is it the evolution to utility computing, where, as with electricity and water, you get your IT from large service providers and pay for it based on usage? 

Or, as The Economist said in an excellent 2008 article, is cloud the next step in the brief history of computers, as the machines “are evaporating altogether and becoming accessible from anywhere”. . . as computing is becoming “more and more disembodied and will be consumed where and when it is needed.” 

“The rise of the cloud is more than just another platform shift that gets geeks excited,” it presciently added.  “It will undoubtedly transform the information technology (IT) industry, but it will also profoundly change the way people work and companies operate.  It will allow digital technology to penetrate every nook and cranny of the economy and of society, creating some tricky political problems along the way.”

Big-Bang Disruptions: The Innovator’s Dilemma in the Digital Economy

I recently read a very interesting article, - Big-Bang Disruption.  The article was written by Larry Downes and Paul Nunes and published in the March, 2013 issue of the Harvard Business Review.  It was later expanded into a book, Big Bang Theory: Strategy in the Age of Devastating Innovation published this past January. 

The authors’ key premise is that innovation is once more undergoing drastic changes, driven by the exponential advances in digital technologies.  Companies are now able to create products that are simultaneously better, cheaper and more appealing than those of competitors.  As a result, “entire product lines and whole markets are now being created or destroyed overnight.”  Examples include Skype, iTunes, the Kindle, Netflix, Facebook, Twitter, and smartphone apps like Google maps and Angry Birds. 

Big-bang disruptions are often unplanned and unintentional.  They are typically discovered through continuous market experimentation.  They upend the conventional thinking on strategy, marketing and innovation, giving rise to a new set of business rules.  “Nearly everything you think you know about strategy and innovation is wrong,” they note.

Some Puzzling Questions about Innovation in the Digital Economy

This semester I am teaching the innovation half of a course on Entrepreneurship and Innovation at NYU’s new Center for Urban Science and Progress (CUSP).  Teaching forces you to take a fresh look at the subjects you are covering, so I find myself revisiting questions I’ve long been thinking about:  What is the essence of innovation in the digital economy and how does it differ from the industrial age innovation of the past two hundred years?

The bulk of the innovations in the industrial economy involved physical objects, physical systems and the physical world around us.  Our biggest scientific advances were in the natural sciences, e.g., astronomy, physics, chemistry, geosciences and biology.  In engineering, we made huge progress in applying science and technology to develop increasingly complex physical objects and infrastructures, such as railroads, cars, airplanes, chemical plants, refineries, power stations, bridges, skyscrapers, computers, microprocessors, and so on.  Ever since the steam engine gave rise to the Industrial Revolution in the late 18th century, we’ve had major waves of innovation every 50-60 years, including railroads, steel, electricity and heavy engineering in the 19th century; and automobiles, airplanes and telecommunications in the 20th century.

The Industrial Revolution had a huge impact on all aspects of the economy and society.  It totally transformed the composition of labor.  In 1840, the vast majority of US jobs where in agriculture, with services accounting for roughly 20% of jobs.  Those figures have been drastically reversing since then.  According to 2009 statistics from the CIA World Factbook, only 0.7% of the US labor force is now involved in agriculture oriented jobs and roughly 20% work in the industrial sector.  The bulk of jobs, almost 80%, are in services, with 3/4 of those service jobs based on processing information in one way or another, - e.g., managerial, professional, technical, sales and office.  Similar changes in labor composition have been taking place in just about every country.      

The emerging 21st century digital economy seems quite different from the industrial economy of the past two centuries.  Dramatic advances in information and communication technologies are now enabling us to apply science, technology and innovation to improve the productivity of information-based service jobs, as well as that of sociotechnical systems, that is, systems in which people play a central role, such as business organizations, healthcare institutions and cities.  Such systems are particularly hard to understand and control because of the complexities inherent in human and organizational behaviors.

The Citi-Imperial Digital Money Readiness Index

On January 29 I attended the 2014 Digital Money Symposium in London, co-sponsored by Citi and Imperial College.  The Symposium convened a group of leaders in their fields to explore the state of adoption of digital money and its economic and societal impacts around the world.  This is the second such Digital Money Symposium, the first one having taken place in London in January of 2013. 

I was a part of the Citi-Imperial team that organized the event.  For this year’s Symposium, we developed a framework that would enable us to discuss the state of adoption of digital money on a more data-driven, scientific basis.  We looked at digital money as a highly complex sociotechnical ecosystem, that is, a technology-intensive system that has major societal, economic and political implications, like cities, healthcare and education.  To help us begin to quantify and understand this ecosystem, we developed the Digital Money Readiness Index, a set of global metrics that enables us to link digital money adoption to various measures of socio-economic development. 

The Index is a multi-functional tool, encompassing publicly available data from 90 countries around the world.  By analyzing its various interdependent components, the team computed a measure of how ready a country is to adopt digital money.  Quantifying the progress being made by each country along the digital money roadmap should help understand the key obstacles the country faces, as well as the potential actions it could take to overcome them.

Overall, the Index aims to help answer two fundamental sets of questions: 

What Matters?: What factors affect the adoption of digital money around world, and how do they vary across different countries and regions? 

 Why Bother?: Does digital money adoption really make a difference, and if so, can we quantify the benefit to governments, companies and individuals?

Big Data, Science and the Humanities

A few weeks ago I participated in an online debate sponsored by The Economist around the question: Are Smart Cities Empty Hype?.  I argued the case against the motion.  In my opening statement I said that I strongly believe that digital technologies and the many data services they are enabling will make cities smarter and help transform them over time.  Later, in my closing remarks, I pointed out that since the qualifier smart essentially means information-based or data-driven, the promise of smart cities is inexorably linked to the general promise of big data and data science, which some have felt are themselves being over-hyped.

Over the holidays, I learned about an intriguing debate that took place this past year in the New Republic.  Billed as Science vs the Humanities, it raised a number of important questions.  To what extent does the promise of data science apply to different kinds of disciplines, including the humanities?  Can big data tools and related methods help the humanities achieve a more scientific understanding of human nature?  Let me make some general comments before getting to the debate itself.

Scientific revolutions are launched when new tools lead to new measurements and observations.  Early in the 17th century, Galileo made major improvements to the recently invented telescope which enabled him to make discoveries that radically changed our view of the universe. Over the centuries we have seen that new tools, measurements and discoveries precede major scientific breakthroughs in physics, chemistry, biology and other natural sciences disciplines.  

Similar breakthroughs have been much harder to achieve beyond the natural sciences, especially when it comes to the study of human nature, that is, the way we feel and behave.  But, the situation has been rapidly changing in the past few decades.  First of all, advances in behavioral science, neuroscience, genomics, evolutionary biology and related disciplines have shed significant new light on such people-oriented studies. And then there is the advent of big data.

Are Smart Cities Empty Hype?

A couple of weeks ago I participated in an online debate sponsored by The Economist.  The debate was built around the question: Are Smart Cities Empty Hype?  Defending the motion was Anthony Townsend, Research Director at the Institute for the Future and adjunct faculty member at NYU’s Wagner School of Public Service.  I took the opposite side, arguing the case against the motion. 

The debate consisted of three phases spread out over roughly ten days.  We each first stated our respective positions in our opening statements, followed a few days later by our rebuttals, and then finally our closing statements.  It was moderated by Ludwig Siegele, online business and finance editor at The Economist, who offered his own remarks in each phase of the debate.  Throughout the process, people were invited to vote on the motion, as well as to post their own comments. 

The debate was inspired, I believe, by The Multiplexed Metropolis, an article Siegele published in the September 7 issue of The Economist which explored the impact of big data on cities.  He wrote that the vast amounts of data generated by the many social interactions taking place in cities might lead to a kind of second electrification, transforming 21st century cities much as electricity did in the past.  “Enthusiasts think that data services can change cities in this century as much as electricity did in the last one,” he noted.  “They are a long way from proving their case.” 

In my opening statement, I said that I strongly believe that digital technologies and the many data services they are enabling will make cities smarter and help transform them over time.  My position is not surprising, given my affiliations with NYU's Center for Urban Science and Progress (CUSP) and Imperial College’s Digital City Exchange, as well as my past involvements with IBM’s Smarter Cities and with Citigroup’s Citi for Cities initiatives.  But, I totally understand why so many, - almost half of those voting and quite a few who left comments, - feel that smart cities are mostly hype.  The case for smart cities is indeed far from proven.

Understanding the Workings of Our Minds

Advances in technology, big data and analytics hold the promise to significantly augment our judgement and expertise and help us make smarter, more effective decisions.  But, as we contemplate these exciting innovations, it’s good to take a step back and ask ourselves a few basic questions: How do we make decisions in the first place? What goes on in our minds when we are making decisions, from the simplest to the most complex? 

I heard a fascinating talk by Daniel Kahneman that directly addressed these questions at the IBM Cognitive Systems Colloquium which I attended last month.  Kahneman is Professor of Psychology Emeritus at Princeton University and Senior Scholar at the Woodrow Wilson School.  In 2002, he was awarded the Nobel Prize in Economics “for having integrated insights from psychological research into economic science, especially concerning human judgment and decision-making under uncertainty.” 

The talk was based on his 2011 bestseller Thinking, Fast and Slow.  The book explains the major discoveries by psychologists and cognitive scientists over the past several decades that have led to our current understanding of judgement and decision-making.  In particular, it describes the pioneering work of Kahneman and his long time collaborator Amos Tversky, who died in 1996.

Humans, Machines and Collective Intelligence

Tom Malone gave a very interesting talk on collective intelligence at the IBM Cognitive Systems Colloquium which I recently attended and wrote about.  Malone is Professor of Management at MIT’s Sloan School and the founding director of the MIT Center for Collective Intelligence (CCI).  His research is primarily driven by this fundamental question: “How can people and computers be connected so that - collectively - they act more intelligently than any individuals, groups, or computers have ever done before?”  This is a very important question to explore to help us understand the impact of our increasingly smart machines on the very nature of work and organizations.  

Malone and his collaborators are conducting research on a number of topics in this area.  Do groups exhibit characteristic levels of intelligence which can be measured and used to predict the group’s performance across a wide variety of cognitive tasks?  If so, can you devise tests to measure the group’s intelligence using methodologies and statistical techniques similar to those that have been applied to measure the IQs of individuals for the past hundred years?

To answer these questions they conducted a number of studies where they randomly assigned individuals to different groups, which then worked on a variety of tasks.  Their research results were published in the October 2010 issue of Science.

Cognitive Systems and Big-Data-driven Applications

In the first week of October I participated in a Cognitive Systems Colloquium hosted by IBM at its Thomas J. Watson Research Center.  IBM defines cognitive systems as “a category of technologies that uses natural language processing and machine learning to enable people and machines to interact more naturally to extend and magnify human expertise and cognition.  These systems will learn and interact to provide expert assistance to scientists, engineers, lawyers, and other professionals in a fraction of the time it now takes.”  

The need for such systems is a result of the explosive growth of data all around us.  Not only are we now able to collect huge amounts of real-time data about people, places and things, but far greater amounts can be derived from the original data through feature extraction and contextual analysis.  One of the key lessons from Watson, - IBM’s question-answering system which in 2011 won the Jeopardy! Challenge against the two best human Jeopardy! players, -  was that the very process of analyzing data increases the amount of data by orders of magnitude.  

This is challenging our ability to store and analyze all that data.  The new generation of cognitive systems will require innovation breakthroughs at every layer of our IT systems, including  technology components, system architectures, software platforms, programming environment and the interfaces between machines and humans.

Smart Cities: A kind of 21st Century “Urban Electrification?”

In The Multiplexed Metropolis, an article in the September 7 issue of The Economist, online business and finance editor Ludwig Siegele explores the impact of big data on cities.  Cities are centers of social interactions, commerce, entertainment and many other human endeavors.  Will the vast amounts of data generated by these activities, - properly collected, analyzed and acted upon, - lead to a kind of second electrification, transforming 21st century cities much as electricity did in the past?

“The power cables that penetrated cities in the late 19th century transformed their shape (there are no tall buildings without lifts), their transit systems, their nightlife, their sewerage (cities need a lot of pumps),” writes Siegele.  “Ubiquitous data services might have impacts as wide-ranging: they could make cities more liveable, more efficient, more sustainable, perhaps more democratic.”  But, he adds a note of caution: “Enthusiasts think that data services can change cities in this century as much as electricity did in the last one.  They are a long way from proving their case.”

In the last few years, a growing number of initiatives have been launched in business, government and academia to help prove the case for data-driven smart cities.  Last year, for example, New York City Mayor Michael Bloomberg announced the formation of NYU’s Center for Urban Science and Progress (CUSP), a partnership between NY City agencies, NYU and other academic institutions, and a number of companies including IBM, Microsoft, Xerox, Cisco and Siemens.  A few weeks ago, mayor Bloomberg personally welcomed the inaugural class in CUSP’s MS in Applied Urban Science and Informatics program.

Data-Driven Decision Making: Promises and Limits

Decision making, - how individuals, groups and organizations go about selecting a course of action among several alternative scenarios, - has long been a subject of study.  Given the explosive growth of big data over the past decade, it’s not surprising that data-driven decision making is one of the most promising applications in the emerging discipline of data science. 

In a recently published article, Data Science and its Relationship to Big Data and Data-Driven Decision Making, Foster Provost and Tom Fawcett succinctly define data-driven decision making as “the practice of basing decisions on the analysis of data rather than purely on intuition.”  Equally succinctly, they view data science “as the connective tissue between data-processing technologies (including those for big data) and data-driven decision making.”

Looking deeper at data-driven decision making, it’s important to understand not only the promise but also the limits.  When can we embed decisions into well understood, automated processes?  When does automation run into limits, and we should view data-driven decision making as a tool to help people make smarter, more effective decisions?  And what are the prospects for its future, as technology, big data and data science continue to advance?

A Kind of Digital “Perfect Storm”

Every day brings new stories about the impact of digital technologies on every aspect of the world around us, including their transformative impact on companies in just about every industry.  Some of the articles are about the creation of innovative new technologies and applications, while others are about the creative destruction such innovations leave in their wake.  Lately, the latter kind of stories seem to predominate.

For a while now, companies have been racing to try to keep with the relentless advances of digital technologies.  But, the race keeps getting tougher every year.  We seem to be heading toward a kind of digital perfect storm, where four major IT trends are each gathering speed while interacting with and amplifying each other: mobile devices, cloud computing, social networks and big data and analytics.

These widely discussed technologies are still in the early stages of deployment to a greater or lesser degree. Mobile is furthest along while cloud adoption is taking off.  Data science applications are starting to get deployed, although we are just beginning to appreciate their profound implications.  And while consumer social networks have been thriving for years, their use in business has been slower than expected. 

The disruptive impact of this digital perfect storm is already being felt by technology companies as well as by those in industries whose transformation is already underway.  And perhaps most important, we know that even greater disruptions are in store in the not too distant future.  The resulting creative destruction makes for interesting stories about the changing fortunes of companies as well as their leaders.

Parallel Supercomputing: Past, Present and Future

From the early days of the industry, supercomputers have been pushing the boundaries of IT, identifying the key barriers to overcome and experimenting with technologies and architectures that are then incorporated into the overall IT market a few years later.  While we generally focus on their computational capabilities as measured in FLOPS, - Floating-point Operations Per Second, -  supercomputers have been at the leading edge in a number of additional dimensions, including the storage and analysis of massive amounts of data; very high bandwidth networks; and highly realistic visualizations.

Through the 1960s, 1970s and 1980s, the fastest supercomputers were based on highly specialized, powerful technologies.  But, by the late 1980s, these complex and expensive technologies ran out of gas and parallel computing became the only realistic alternative to scaling up performance.  

Instead of building machines with a small number of very fast and expensive processors, the early parallel supercomputers ganged together 10s, 100s, and over time 1000s of much less powerful and inexpensive CMOS microprocessors, similar to the micros used in the rapidly growing personal computer and workstation industry.  A similar evolution to microprocessor components and parallel architectures took place a few years later in the mainframes used in commercial applications.

The transition to parallel supercomputing was seismic in nature.  Everything changed, from the underlying computer architecture, to the operating systems, programming tools, mathematical methods and applications.  It took considerable research and experimentation to learn to effectively use these new kinds of machines.  Moreover, there were widely different parallel architecture designs, some coming from universities and others from industry.  It wasn’t clear at all which designs worked well for different kinds of applications and would thus be commercially viable.

The Department of Energy (DOE) national labs have long been among the world’s leading users of advanced supercomputers and played a leading role in the transition to parallel architectures.  In 1983, the DOE’s Argonne National Lab established the Advanced Computing Research Facility (ACRF), an experimental parallel computing lab which brought together computer scientists, applied mathematicians and supercomputer users and vendors to learn how to best use this new generation of parallel machines. 

This past May, Argonne convened a Symposium to mark the 30th anniversary of the ACRF.  The Symposium looked both at the progress made in parallel computing over the past 30 years and the major trends for the future.  I attended the Symposium and led a panel on The Impact of Parallel Computing on the World.

Smart Homes - Building for America's Bravest

In the Fall of 2007, I taught a graduate course in MIT - Technology-based Business Transformation.  The course focused on the major challenges that a business must deal with to successfully bring a disruptive innovation to market, including technical and market strategies as well as organizational and cultural issues.  The course drew heavily on my concrete experiences at IBM, especially what I learned as general manager of the Internet Division in the second half of the 1990s.

Within the limits of a one-semester course, I wanted to give the students a fairly comprehensive picture of what it takes to bring a disruptive innovation to market, including the key role played by marketing and communications.  I wanted the class to hear from a top professional in the field what brand and strategic marketing are all about, so I invited Chris Wall, - Vice Chairman of Ogilvy and Mather, one of the top marketing and advertising agencies in the world, - to be a guest lecturer in the class.  For a number of years, I worked closely with Chris and his O&M colleagues in the development of marketing campaigns for IBM initiatives I was leading.  These included the award-winning e-business campaign launched in 1997, - which explained what we meant by e-business in a series of really clever 30 second vignettes, - and the Linux campaign a few years later which included this incredible ad. 

In his excellent class seminar, Chris explained the difference between trying to establish a brand through mindless jingles or paternalistic assertions that no one believes, versus engaging your audience in a conversation about the future.  To engage in such a conversation, you need to essentially tell a story about your aspirations for the brand and how these aspirations are driving your strategy.  And, the more powerful and complex the messages you are trying to convey, the more important it is to do so by telling a compelling, emotionally resonant and simple story.

Chris and I became friends, and over the years he has shared with me some of the marketing campaigns he’s worked on that he is particularly proud of.  A couple of weeks ago he sent me an e-mail telling me about his latest project with a small foundation, Building for America’s Bravest.  Chris and a few of his Ogilvy colleagues created the integrated marketing campaign for the foundation, including its website, ourbravest.org which has just gone live, and includes a few short video clips along with pictures and stories.

I was frankly left speechless by ourbravest.org.  Compelling and emotionally resonant does not begin to do justice to its content.  The video clips and pictures truly grab you by the throat, or rather, by the heart.  I cannot possibly describe them in words, so please take a look and judge for yourself.

From the Digital Divide to Inclusive Innovation: the Case of Digital Money

On June 26 I participated in a panel discussion on digital money at the Royal Society of Arts (RSA) in London.  The RSA, - whose full name is the Royal Society for the encouragement of Arts, Manufactures and Commerce, - was founded in 1754 to “embolden enterprise, enlarge science, refine art, improve our manufacturers and extend our commerce”.  Today, it calls itself “an enlightenment organisation committed to finding innovative practical solutions to today’s social challenges”. 

The panel was organized to mark the publication of an RSA pamphlet - From the digital divide to inclusive innovation: the case of digital money.  I was a co-author of the pamphlet, along with Imperial College professors David Gann and Gerald George and University of Queensland professor Mark Dodgson.  The four of us were joined in the panel by Citigroup executive Tomasz Smilowicz.

We discussed the profound transformative power of digital money.  Economic transactions are increasingly moving from the physical to the digital realm, democratizing financial services around the world and helping billions join the global digital economy and improve their standard of living.  As we wrote in the pamphlet: 

“Just as communications and publishing have been transformed by digital technologies, so too will financial services.  The progress of digital money will inevitably surprise us and it will develop in unexpected ways, but we believe it is on the cusp of delivering a remarkable transformation in the global economy.  It will end the divide between those who can and those who cannot participate in formal economic transactions.  It may usher in a new era of more inclusive innovation that involves billions of more people around the world in constructing the services that affect their future.”

The Era of Cognitive Computing

Tools have played a central role in human evolution since our ancestors first developed hand axes and similar such stone tools a few million years ago.  Ever since, we’ve been co-evolving right alongside the tools we create.  “We shape our tools and they in turn shape us,” observed noted author and educator Marshal McLuhan in the 1960s. 

The Industrial Revolution led to dramatic improvements in productivity and standard of living over the past two hundred years.  This is due largely to the machines we invented to make up for our physical limitations - the steam engines that enhanced our physical power, the railroads and cars that made up for our slow speed, and the airplanes that gave us the ability to fly. 

Similarly, for the past several decades computers have been augmenting our intelligence and problem solving capabilities.  And, according to IBM’s John Kelly and Steve Hamm, there is much more to come.  In Smart Machines: IBM’s Watson and the Era of Cognitive Computing, Research director John Kelly and writer and strategist Steve Hamm, note that “We are at the dawn of a major shift in the evolution of technology.  The changes that are coming over the next two decades will transform the way people live and work, just as the computing revolution has transformed the human landscape over the past half century.  We call this the era of cognitive computing.”

The Continuing, Transformative Impact of IT

Last month, the McKinsey Global Institute published Ten IT-enabled business trends for the decade ahead.  As is generally the case with McKinsey, this is a well researched, well written report.  There are few surprises.  Any technology expected to have a transformative impact on business over the next decade has to at least be already in the hands of leading edge users.  Disruptive innovations take time to play out.

“Many trends reflect the growing dominance of the Internet as an enabling technology, as well as a model and metaphor for commercial and social interactions,” says the report in its introduction.  “Twenty years into the Internet revolution, businesses and consumers have come to expect that information is a Google search away, friends and associates are always available on social networking sites, and goods and services (including public goods such as education and government services) can be had instantly from an online vendor anywhere in the world at any time of the day.”

The social matrix, the Internet of all things, big data and advanced analytics, and realizing anything as a services are the report’s top four trends.  SMAC - Social, Mobile, Analytics & Cloud, - has become the new plastics, capturing the future of IT in one word, or rather, one acronym.  Just about everyone agrees that these are foundational technologies, like the Internet twenty years ago, that every business must embrace.

New Tools Beget Revolutions: Big Data and the 21st Century Information-based Society

On May 22 I attended the 2013 MIT Sloan CIO Symposium.  This year’s theme was The Transformational CIO: Architecting the Enterprise of the Future.  “The enterprise of the future will be very different from the one we know,” wrote the organizers in the event’s website.  “It will be complex, develop hitherto unforeseen forms, and will have to respond to unforeseen challenges.  All while the windows of opportunity and response will continue to shrink.  There will not only be the need to be agile and adapt to changes as they occur, but to be proactive in shaping the next generation enterprise to be ready for the future.” 

The Symposium included a number of talks and panels on the key issues facing CIOs in our post-digital world, that is, in an era when digital technologies permeate just about every nook and cranny of the business, and where every business is a digital business, the title of one of the talks at the event.  While a number of transformative technologies were discussed, big data was the most prominent.    

MIT professor Erik Brynjolfsson led an all-MIT academic panel of experts on The Reality of Big Data.  In a brief introductory talk, Brynjolfsson pointed out that throughout history new tools beget revolutions.  Scientific revolutions are launched when new tools make possible all kinds of new measurements and observations.  In 1676, for example, Antonie van Leeuwenhoek used a microscope, a relatively recent and rare tool, to discover the existence of microorganisms in a drop of water.  Thus was microbiology born, leading to major discoveries in biology, medicine, public health and food production in subsequent decades and centuries.

Traditional Data Analysis and Data Science

I recently participated in an industry panel on Redefining BI & Analytics in the Cloud at a conference in New York.  Among other questions, moderator Michael Hickins, - a senior editor at the Wall Street Journal and editor of its online CIO Journal, - asked the panel to discuss the difference between the business intelligence and analytics approaches of the past twenty years and the emerging discipline of data science. 

In my opinion, data science should be viewed as a multidisciplinary evolution from business intelligence and analytics.  In addition to having a solid foundation in statistics, math, data engineering and computer science, data scientists must also have expertise in some particular industry or business domain, so they can properly identify the important problems to solve in a given area and the kinds of answers one should be looking for.  Domain expertise is also needed to be able to draw the proper conclusions from their analysis, and to communicate their findings to business leaders in their own terms. 

The emergence of data science is closely intertwined with the explosive growth of big data over the past several years.  Institutions are now wrestling with information coming at them in volumes and varieties never encountered before.  In addition to their multidisciplinary skills, data scientists bring increased breadth and depth to the analysis of these different sources of information compared to traditional analyst roles, as described in this short article in IBM’s website, So what does a data scientist do?: 

“Whereas a traditional data analyst may look only at data from a single source - a CRM system, for example - a data scientist will most likely explore and examine data from multiple disparate sources.  The data scientist will sift through all incoming data with the goal of discovering a previously hidden insight, which in turn can provide a competitive advantage or address a pressing business problem.  A data scientist does not simply collect and report on data, but also looks at it from many angles, determines what it means, then recommends ways to apply the data.”

Reflections on Storytelling

Storytelling has played a central role in human communications since times immemorial.  Storytelling predates writing.  Oral narratives were used by many ancient cultures as a way of passing along their traditions, beliefs and learning from generation to generation.  Over the centuries, the nature of  storytelling has significantly evolved with the advent of writing and the emergence of new technologies that enabled stories to be embodied in a variety of media, including books, films, TV and the Web.

While contemporary storytelling is mostly associated with entertainment, education and culture, it also plays a major role in business.  In particular, storytelling is widely used in the development of brands for products and services as well as companies. 

Establishing a brand requires much more than advertising, mindless jingles or assertions that no one believes.  It is akin to engaging in a conversation with your intended audience, where you talk about your challenges and aspirations for the product and the company in order to create emotional associations and expectations.  In short, you are creating and telling a story about your brand.  And, the more powerful, important and complex the messages you are trying to convey, the more important it is that you do so by telling a compelling and emotionally resonant story. 

Good storytelling is particularly important when introducing a complex and potentially disruptive offering in the marketplace whose value is not well understood.  It’s a good way of explaining what the new innovation is all about in as simple a way as possible.  As a concrete example, let me share my experiences with the development of IBM’s Internet and e-business strategy in the mid-late 1990s.

The Digitization of the Economy

MIT just launched a new Initiative for the Digital Economy to systematically address the impact of digital technologies on business, the economy and society.  The university-wide initiative is being organized by MIT’s Center for Digital Business (CDB), and is led by Erik Brynjolfsson and Andy McAfee, who are respectively the CDB’s director and principal research scientist.

Since the mid 1990s, we’ve been talking about the emergence of an Internet-based digital economy.  It has often been said that powerful disruptive innovations like the Internet are overhyped in the short term but under-hyped in the long term.  This is certainly the case with digital economy.  The term was widely thrown around in the heady dot-com bubble days as people talked about a new economy with eyeballs instead of revenue, profit and cash as key business metrics. 

But, over the past decade the Internet has transitioned from the connected world of PCs, browsers and web servers, to the hyperconnected world of mobile devices, cloud computing and broadband wireless networks.  The physical and the the digital now blend into each other over porous boundaries, as people, institutions and things are all becoming interconnected.  A 21st century digital economy is indeed now emerging in this hyperconnected, inclusive, and increasingly smart world.

The Evolution toward a Global Digital Money Ecosystem

I recently read an excellent article, 2012: The Year in Payments - A Look Back At The News And The Noise In Payments by Karen Webster, CEO of Market Platform Dynamics and founder of PYMNTS.com.  In the article, Webster reviews some of the major developments in digital payments over the last year, and gives her opinion of what is likely to happen in 2013 and beyond.  Beyond her detailed reviews and predictions, the article is a very well written overview of the evolution toward a digital money economy, including the major advances already underway as well as the huge challenges that still lie ahead. 

For several years now I’ve been convinced that the development of a global digital money ecosystem is among the most exciting and important societal challenges in the coming decades.  It’s right up there, in my opinion, with other technology-based transformational innovations, including electricity, the telephone, radio and TV, and the Internet and World Wide Web.

Where are we in digital payments?  Webster believes that 2012 will be viewed in the payments history books as a very good year.  “Payments innovation fueled by the IP-enablement of devices used by consumers and merchants to interact drove payments innovation into high gear.  Players large and small flooded the market with new applications, new business models, and new approaches to transforming the shopping experience.”

Reflections on Big Data, Data Science and Related Subjects

A couple of months ago I was appointed executive-in-residence at the Center for Urban Science and Progress (CUSP), an interdisciplinary applied science research institute led by NYU and NYU-Poly in partnership with academic institutions, global companies and New York City government agencies.  CUSP’s overriding mission is the study of “the grand technical, intellectual, engineering, academic, and human challenges posed by a rapidly urbanizing world.”   It was formally launched last April by New York City’s mayor Michael Bloomberg. 

CUSP’s research and educational programs are centered on urban informatics, - “the acquisition, integration, and analysis of data to understand and improve urban systems and quality of life.”  Big Cities + Big Data and Bringing Urban Data to Life are prominently displayed in its website. This coming September, it will start offering two new programs in Applied Urban Science and Informatics, a 30-credit Master of Science, and a 12-credit Advanced Certificate. 

Given the central role of big data with CUSP and with other initiatives I’m involved with, I’d like to step back and reflect on what this all means.

Hurricane Sandy: a 21st Century Wake-Up Call?

As I was personally experiencing the fury of Hurricane Sandy a few weeks ago, I thought of the Internet Highway System and President Dwight D. Eisenhower.  Sandy brought into sharp relief how dependent so many of us have become on digital infrastructures, and the Internet in particular, for almost every aspect of our life and work.  Sandy also demonstrated how inadequate and fragile our digital infrastructures are given our increasing dependence on them.  If President Eisenhower was around today, I reflected in moments of frustration, I was convinced that he would be a strong advocate for building a world class national broadband infrastructure, just like he championed the Internet Highway System over half a century ago.

The National Interstate and Defense Highways Act was signed into law by President Eisenhower in June of 1956.  Most everyone would agree that it has contributed greatly to the economic success of the US over the past fifty years.  But, in addition to economic development, one of the main reasons President Eisenhower championed the Interstate Highway System was national defense.  While serving as Supreme Commander of the Allied forces in Europe during World War II, he saw first hand the role played by the German Autobahn network in its national defense, and felt that the US needed such a modern highway system to provide key ground transport routes for military supplies and troop deployments in case of an emergency or external threat. 

Getting ready for Sandy, and then feeling its power was very different from watching a far away disaster on television.  I got off fairly easily compared with so many in the New York Metropolitan Area,  including our town in Connecticut, whose homes were flooded or were seriously damaged by falling trees.  We lost a few large trees, one of which damaged part of a fence.  The power in our area was out for almost a week, but, like an increasing number of my neighbors in our woodsy area, we have adjusted to the frequent power losses by installing a standby generator, so our house had electricity and heat during the power outage.  We also lost cable and our cable-based Internet access.  My wireless mobile phone was barely usable other than very early in the morning before most of the town was awake.  For a couple of days, even that did not work. 

Over 22 million people live in the Tri-state New York Metropolitan Area.  The region is arguably the world’s center of commerce and banking, home to almost 20 percent of Fortune 500 companies.  New York City is ranked at the top by just every survey of the world’s most competitive cities, particularly because of its appeal to a wide range of businesses and talented people. 

But, as Sandy proved to us all, our physical infrastructure is antiquated, and our digital infrastructure lags far behind the growing importance of the Internet in the economy and in our personal lives.  With few exceptions, the situation is not much better across the US.  The infrastructures of just about all US metropolitan areas are falling behind the rest of the world.

Reflections on the Nature of Human Organizations

Urbanization is one of the major forces transforming societies all over the world.  The rapid pace of urbanization is succinctly summarized in these sentences from the 2012 World Urbanization Prospects of the United Nations Population Division:  

“Between 2011 and 2050, the world population is expected to increase by 2.3 billion, passing from 7.0 billion to 9.3 billion (United Nations, 2011). At the same time, the population living in urban areas is projected to gain 2.6 billion, passing from 3.6 billion in 2011 to 6.3 billion 2050.  Thus, the urban areas of the world are expected to absorb all the population growth expected over the next four decades while at the same time drawing in some of the rural population.  As a result, the world rural population is projected to start decreasing in about a decade and there will likely be 0.3 billion fewer rural inhabitants in 2050 than today.”

In 1900, less than 15 percent of world’s population lived in cities.  According to the latest UN estimates, 52 percent of the world’s population now lives in urban areas.  That number is estimated to go up to 60 percent by 2030, and near 70 percent by 2050. 

Why are metropolitan areas continuing to grow at such a rapid pace?  After all, back in the 1990s, quite a few people were predicting that the Internet would cause cities to decline.  The Internet would enable people to work from anywhere, so they would choose to live in less populated areas, where lower costs and more space would let them enjoy a better quality of life.  Needless to say, these predictions have not come true.

The Science of Cities

Last April, New York City Mayor Michael Bloomberg and NYU President John Sexton announced the formation of the Center for Urban Science and Progress (CUSP), an applied science research institute led by NYU and NYU-Poly in partnership with academic institutions, global companies and New York City government agencies.  CUSP is focused on the grand technical, intellectual, engineering, academic, and human challenges posed by a rapidly urbanizing world.  Several weeks ago, I was appointed executive-in-residence at CUSP by its Director, Steve Koonin. 

Urban systems are the quintessential example of complex sociotechnical systems, the emerging field studying systems that combine powerful and ubiquitous digital technologies with the people and organizations they are transforming.  Such systems exhibit a level of complexity that, until recently, has often been beyond our ability to understand and control.  Not only do we have to deal with the complexities associated with large scale IT infrastructures, but with the even more complex issues involved in human and organizational behaviors. 

This emerging field aims to go beyond the analysis of cities and similar sociotechnical systems.  It aims to be prescriptive in nature, applying its findings to help us design better performing systems and organizations, as is the case with other engineering and management disciplines.  It is multidisciplinary in nature, and should enable us to address a particular domain, - e.g., transportation, energy, finance, - by bringing together a number of different methodologies, including: systems-oriented techniques like optimization, stochastic systems, systems simulation and systems dynamics; social sciences, management and planning; and engineering capabilities focusing on design, development, testing and operations. 

Reflections on the Longevity of Mainframes

At the end of August, IBM announced the latest member of its mainframe family, the zEnterprise EC12.  As pointed out in the announcement, the new system is a result of more than $1B in R&D investments over the past four years, resulting in major improvements in performance, security, availability and other key enterprise features.  But perhaps, what is most impressive about this announcement, is the longevity of the IBM mainframe, which is now in its 48th year.  Few computer families having major announcements in 2012 could trace their vintage to the 1980s, let alone the 1960s.  There is something pretty unique about the mainframe being not only alive but well after all these years.

I have had a long association with mainframes.  I was a second year college student at the University of Chicago when the System 360 was first announced in April of 1964.  At the time, I was also working part time at the university’s computation center which used IBM computers.  I still remember attending a presentation on the announcement given by a visiting IBM technical executive.  Later on as a physics graduate student, I used high-end S/360 models for my thesis research.  In IBM, I was closely associated with mainframes through major portions of my 37 year career, in particular, the period from 1977-1992 when I was involved in a number of R&D initiatives on the future of the mainframe. 

To a large extent, the mainframe’s longevity is a result of two major architectural innovations first introduced with S/360.  The first was the notion of a family of computers, from low to high performance, all based on the same instruction set which allowed customers to upgrade to larger systems as well as to future system models without having to rewrite their applications.  The second was OS/360, a common operating system that supported the various members of the S/360 family except for the smaller ones which ran DOS/360, a subset with more limited capabilities.  Today’s z/Architecture  and z/OS are direct descendants of the original S/360 and OS/360. 

Through the 1960s, 1970s and 1980s, mainframes were built using highly sophisticated and expensive technologies.  It was important to get the maximum efficiency and price-performance out of each machine, so the hardware architecture and operating system were carefully designed to achieve high processor utilizations and fast response times to high volume transactions.  Over the years, the IBM mainframes have continued to carefully design the hardware and operating systems software together to optimize performance and industrial strength, e.g., security, systems management, availability and other ilities.

Market-facing Innovation Labs

The corporate research lab has radically changed over the past several decades.  Corporate labs reached their peak in the 1960s and 1970s.  As technologies and markets changed, most such labs declined in importance over the next twenty years.  But, they are being reincarnated in our 21st century information economy as market-facing innovation labs, with significantly different and broader scopes than those of the original industrial economy research labs.

Most of these labs were established by large, successful industrial companies in the years after World War II.  Their job was to push the frontiers of knowledge by conducting both basic and applied research, which would hopefully, over time, lead to new commercial technologies and products.

In the postwar years, ATT’s Bell Labs was one of the top research labs in the world, having developed a number of revolutionary technologies, such as the transistor, the laser, information theory, UNIX and C.  Xerox PARC created a number of the key technologies that helped make personal computers so successful, including the Graphical User Interface (GUI), the WYSIWYG text editor and Ethernet.  IBM’s Research labs developed the Dynamic Random Access Memory (DRAM), magnetic disk storage, the Reduced Instruction Set Computer (RISC) architecture, relational databases and fractals as well as helping to establish computer science as a viable academic discipline. 

I joined the computer science department at IBM’s Thomas J Watson Research Center when I finished my university studies in 1970, and worked there for fifteen years.  In those days, R&D were two very distinct activities, the R conducted by scientists in research labs, and the D conducted by engineers in product development labs.  Marketing, was another distinct activity in a product’s life cycle.  These various handoffs resulted in large time gaps between research, product development, and marketing, but since both technology and markets advanced at a relatively slow pace, there was little pressure to reduce the transition times across the gaps.

A very good 2007 Economist article, The Rise and Fall of Corporate R&D - Out of the Dusty Labs observed that this strong separation between R&D&M was common across all industries.  John Seely Brown, former director of Xerox PARC, succinctly captured the prevailing attitude of research labs in those days with this quote in The Economist article:  “When I started out running PARC, I thought 99% of the work was creating the innovation, and then throwing it over the transom for dumb marketers to figure out how to market it.”

The Continuing Evolution of the Internet

A few weeks ago I had an interesting discussion with Bob Kahn.  Along with Vint Cerf, Bob is recognized as one the key fathers of the Internet, for which they both received the 1997 National Medal of Technology and a number of other top prizes.  The occasion was a board meeting of the Corporation for National Research Initiatives (CNRI), a not-for-profit organization that promotes research in the public interest which Bob founded in 1986 and continues to lead.  Earlier this year, I joined CNRI’s Board of Directors.  

 We were talking about the continuing evolution of the Internet, something Bob is uniquely qualified to discuss.  Beyond his role as co-inventor of its key protocols, Bob has played a leading role in shepherding the Internet from its modest beginnings over forty years ago to its well recognized position as the engine of the digital revolution. 

What do we mean by the Internet?  The most concrete answer is that the Internet is the global system of interconnected computer networks based on the TCP/IP protocols co-developed by Cerf and Kahn.  But, beyond the central role that TCP/IP has played, I believe that the success of the Internet is predicated on a few key architectural principles: open, simple and distributed.

Digital Common Law

On May 30 and 31 I attended a meeting in Washington, the Rights and Bytes: Law and Big Data Workshop. The workshop explored progress in developing effective, legally sound, software-based solutions for the management of digital identities and for the protection and sharing of sensitive data.  It was  sponsored by ID³, a recently organized multidisciplinary non-profit institution with close ties to MIT’s Media Lab.  ID³’s principal focus is to develop architectural frameworks and open source software solutions that embody the necessary policy principles for managing critical personal information so that it can be tested, verified, improved and scaled. 

For the past few hundred years, governments around the world have developed laws, policies and practices to help us better get along with each other as we conduct the many transactions and interactions that are part of our everyday lives.  But, as as Internet and digital technologies are increasingly integrated into just about all aspects of our lives, the laws, policies  and institutions we developed for the physical world are way too impractical for our fast moving, fast changing digital world.  We must somehow bring with us and adapt our policies and regulations from the physical to the digital world.

Is this even feasible?  Yes, argues John Clippinger, ID³’s Executive Director and CEO, and policy strategist and writer David Bollier in an excellent working paper Digital Common Law: Brief on Trust Frameworks and Self-Governance which they wrote for the workshop.

The Untethered, Hyperconnected Enterprise

A few weeks ago I attended the MIT Sloan CIO Symposium.  The theme of this year’s Symposium was Piloting the Untethered Enterprise:  “In today's world of mobile, big data and the cloud, how does a CIO successfully pilot his organization towards its goals?” 

After listening to a number of talks and panels throughout the day, I came away with a few key impressions.  Pretty much everyone agrees that we are in transition from the tethered, connected world of PCs, browsers and classic data centers to the untethered, hyperconnected world of smart mobile devices and cloud computing.  The emergence of the untethered enterprise will have major implications for IT, for business and for society in general.  But lots of questions remain as to what those implications might be.  This is to be expected in the early stages of such a major transition. 

Two back-to-back panels explored what it takes to successfully transition to the untethered enterprise, one consisting of academics, and one of CIOs.  Not surprisingly, each of these panels viewed the opportunities and challenges through somewhat different lenses.

Complex Sociotechnical Systems: the Case for a New Field of Study

Digital technologies are enabling us to develop systems with huge numbers of interconnected components and sophisticated software that infuses them with seemingly unlimited capabilities.  They are penetrating just about every nook and cranny of the economy and of society in general.  And, they are profoundly changing the way all organizations operate, as well as our working and personal lives. 

We are increasingly able to pull all these capabilities together and develop sociotechnical systems, which combine the powerful, inexpensive and ubiquitous digital technologies with the people and organizations they are transforming.  But, we have much to learn on how to best design and operate these systems.  They generally exhibit a level of complexity that is often beyond our ability to understand and control.  Not only do they have to deal with the complexities associated with large scale hardware and software infrastructures, but with the even more complex issues involved in human and organizational behaviors. 

A few weeks ago, MIT professor Joseph Sussman addressed these issues in a lecture - Complex Sociotechnical Systems: The Case for a New Field of Study.  Sussman is Professor of Civil and Environmental Engineering and Engineering Systems and interim director of the MIT Engineering Systems Division. 

Complex sociotechnical systems are high-impact, difficult-to-understand, technology-intensive systems that have major societal, political and economic implications.  While this definition can rightfully apply to many of the systems we have been developing in the last few decades, their importance is on the rise now for two major reasons. 

We are increasingly using systems thinking to address some of our most critical contemporary problems including climate change, energy and the environment, cities and metropolitan areas, jobs and productivity, the global economy, financial systems and national security.  At the same time, we now have the technologies, tools, engineering methodologies and scientific principles to be able to better design and manage such systems.

Systems Thinking, Safety and Risk Management

A few weeks ago I participated in a roundtable breakfast on Protecting Financial Markets in the Age of the Cloud.  The roundtable gathered together a group, - consisting primarily of financial experts, with a sprinkle of technologists like me, - to explore how to best regulate financial markets in our fast changing, highly interconnected, complex, digital world. 

Several participants cited the sharp rise in volume and speed of high-frequency trading over the past decade as an example of a current practice that is potentially increasing volatility and systemic risk in financial markets.  They mentioned the flash crash of May 6, 2010 as the kind of incident that is not well understood, and whose recurrence could compromise the stability and general trust needed for financial markets to function smoothly.  At least one of the roundtable participants was convinced that another major event, similar to the global financial crisis of 2008 would inevitably take place in the next few years. 

Toward the end of the roundtable, the moderator asked us to reflect on the kinds of controls needed to make financial systems more stable and less prone to another major crisis.  Can you do so with incremental, evolutionary safeguards, or do you need major new regulations?  Can the industry institute the needed safeguards on its own or do governments have to play a bigger role?

These are very complex and important questions for which there are no simple answers.  Our answers often reflect where we stand in the conservative-liberal spectrum.  My own point of view is shaped by my interests in complex engineering systems, in particular, in systems whose components are interconnected via digital networks; where software plays a major role in their overall design and operations; and, which involve people and information as well as technology. 

Financial systems are an example of such complex engineering systems.  Thus, I am hopeful that much of the research on the design and operation of such systems will shed light on how to better manage financial systems as well.

The Complex Transition to the Age of the Cloud

In June of 2008 I participated in a conference on cloud computing.  After a full day of talks and discussions, the sense of the meeting was that cloud computing has the promise to bring a real paradigm shift to the IT world, although, as the conference organizer succinctly put it in his closing remarks: “There is a clear consensus that there is no real consensus on what cloud computing is.”  In other words, something big and profound seems to be going on, although we are not totally sure what it is yet. 

Where are we now, almost four years later?  Just about everyone agrees that cloud computing is one of the major trends in IT, with important implications not only to IT but to business and society in general.  Many books and articles have been written on the subject.  A number of companies have cloud-based offerings in the marketplace, with more to come.  But, while many of us feel that cloud is even bigger and more profound than we thought back in 2008, lots of questions remain on its intrinsic nature and its value to the business. 

In November of 2011, Andrew McAfee published an article in the Harvard Business Review: What Every CEO Needs to Know about the Cloud.  McAfee is Principal Research Scientist and Associate Director of the MIT Center for Digital Business, where I am a Fellow.  

He writes that cloud computing is “ . . . a sea change - a deep and permanent shift in how computing power is generated and consumed.  It’s as inevitable and irreversible as the shift from steam to electric power in manufacturing, which was gaining momentum in America about a century ago.”

Hot Spots: Benchmarking Global City Competitiveness

A new report, Hot Spots: Benchmarking global city competitiveness, was just released by the Economist Intelligence Unit.  Hot Spots is the result of a comprehensive study that analyzed the competitiveness of 120 of the world’s major cities.  The study evaluated their performance along eight distinct categories, from which it developed an overall Competitiveness Index for each city.  The study was commissioned by Citigroup as part of its Citi for Cities initiative.

 Hot Spots observes that over half of the world’s population now lives in cities, generating over 80% of global GDP.  As a result, companies are planning their global business strategies from a city, rather than from a country perspective.  The 120 urban centers ranked in Hot Spots have a combined population of about 750 million and collectively account for almost 30 percent of the global economy.  They are fiercely competing with each other to attract business investment and top talent that will enable them to grow their economies, create jobs and improve their overall standard of living and quality of life.

What determines a city’s competitiveness and growth potential?  Hot Spots defines competitiveness as a city’s ability to attract capital, business, talents and visitors.  Competitiveness, according to the report, “. . . is a holistic concept.  While economic size and growth are important and necessary, several other factors determine a city’s overall competitiveness, including its business and regulatory environment, the quality of human capital and indeed the quality of life.  These factors not only help a city sustain a high economic growth rate, but also create a stable and harmonious business and social environment.”

The research team assessed each of the 120 cities benchmarked using 31 different indicators.  Working with a number of experts, it then grouped these indicators into eight major categories, assigning a weight to each toward the final overall ranking: economic strength (30%), human capital (15%), institutional effectiveness (15%), financial maturity (10%), global appeal (10%), physical capital (10%), environment and natural hazards (5%), and social and cultural character (5%).

Scale and Complex Systemic Innovation

The year end issue of The Economist includes a provocative column - Big and clever: Why large firms are often more inventive than small ones.  The column addresses a fundamental question that those studying innovation have been wrestling with for years:  “Are big companies the best catalysts of innovation, or are small ones better?”

The column is based on a recent study - Scale and Innovation in Today’s Economy - by Michael Mandel, chief economic strategist of the Progressive Policy Institute. 

“Conventional wisdom these days says that small is better when it comes to innovation and putting new ideas into practice,” writes Mandel at the beginning of the report.  “Large enterprises are typically thought of as hidebound defenders of the status quo, dominating by market power and brute force rather than technological and innovative prowess. . .Yet reality is far more complicated than this simple small versus big distinction. . . After 20 years where startups have rightly dominated the innovation headlines, we will show that the pendulum may be swinging back.  As a result, there are reasons to believe that scale may be a plus for innovation in today’s economy, not a minus.”

There is little question that small, entrepreneurial start-ups are almost always the ones that bring to market highly disruptive innovations.  They excel at creative destruction, the term that economist Joseph Schumpeter  popularized almost seventy years ago to describe the process of transformation that accompanies disruptive innovation. 

Entrepreneurs have sharp incentives to break into a market with a disruptive new idea that attacks the dominance of established companies.  They are highly motivated to bring to market new products which offer significantly improved designs and/or lower costs, and thus gain competitive advantage and market share from the established leaders in their segment of the market. 

But, in what has become known as the Schumpeterian Hypothesis,  Schumpeter also believed that large companies in concentrated markets have more incentives to innovate.  By investing in new products or processes, the company can sell them to their already large customer base and reap greater rewards faster.  Moreover, in a highly competitive market, startups will have trouble holding on to the gains from their innovations because they will be quickly imitated by rivals, while large companies with greater market power will be able to better protect and hold on to the gains from their innovations.  

So, which is it?  Are large companies at a disadvantage because start-ups can move much faster in bringing new ideas to market, or is their scale an incentive for investing in innovation?  Mandel summarizes the economic research evidence in two words: It depends.

Open, Collaborative Trust Frameworks

A couple of weeks ago I attended a workshop on Inventing the Future of Finance cosponsored by ID³ and the MIT Media Lab.  ID³, - the Institute for Institutional Innovation by Data Driven Design, - is a new multidisciplinary non-profit institution formed to develop architectural frameworks, open source software and educational content for the protection, sharing and monetization of sensitive data. 

It was founded by MIT Media Lab Professor Alex “Sandy” Pentland, and by John Clippinger, who is currently a Research Scientists at the Media Lab, and until recently was a Senior Fellow at Harvard’s Berkman Center for Internet and Society.  Clippinger is Executive Director and CEO of ID³, while Pentland serves as its Chief Scientist.  I just joined its Board of Directors.

One of the key missions of ID³ is the creation of trust frameworks, which it defines as:  “A combination of software mechanisms, contracts, and rules for defining, governing and enforcing the sharing and protection of information according to a common and independently verifiable standard of performance.  Whenever possible, such governance mechanisms and contracts should be self-executing and self-correcting.”

A Neural, Digital System for the Economy

Most everyone will agree that the world’s economy is undergoing historical structural changes, mostly caused by the digital technology revolution taking place all around us.  In particular, ever since the explosive growth of the Internet in the mid 1990s, we are seeing the emergence of a global, digital economy.  What does this mean?  How can we best describe the nature of this emerging digital economy?  What are some of its key implications for the overall economy?

There is clearly no simple answer to these questions.  But, I recently read one of the clearest and most elegant explanations of the changes our economy is going through in an article in McKinsey Quarterly by economist W. Brian Arthur, - The Second Economy.  Dr. Arthur is external professor at the Santa Fe Institute and visiting researcher at PARC’s Intelligent Systems Lab.

“Digitization is creating a second economy that’s vast, automatic, and invisible - thereby bringing the biggest change since the Industrial Revolution,” he writes at the very beginning of the article.  He illustrates his point with an analogy to the transformation of the US economy by the railroads in the second half of the 19th century.”

The Emergence of Cognitive Computing

On October 21 I participate in a Colloquium on the Frontiers of IT at IBM’s Thomas J Watson Research Center.  The colloquium, - which was part of IBM’s Centennial celebration, - brought together experts across industry and academia to discuss and debate the direction of four key Grand Challenges at the very leading edge of IT:  nano systems, exascale, big data, and cognitive computing.

John Kelly, senior VP and director of IBM Research, gave an overview of these four areas.  Here is a similar version of his talk given a week earlier at the University of Melbourne.  John explained that these four areas have the potential to transform the IT industry because of their exponential growth, a result of both continual improvements and disruptive innovations.  “Exponential curves,” he said, “will either put you ahead of the competition of kill you.  It’s one of the other.”

Over the next decade, nano-devices are expected to advance by three orders of magnitude, from a billion to a trillion transistors in a chip.  We will be able to design sophisticated, powerful nano systems-on-a-chip that will be totally contained within such a trillion transistor nano device.  To do so, we will have to shift from silicon to other carbon-based materials.  This requires disruptive technologies and innovations at all levels, including new materials, fabrication processes and design tools.  The work is underway.

Unleashing the Next Age of Prosperity

A few weeks ago I met with Carlota Perez in New York.  I first met Carlota, - a friend so I will refer to her by first name, - around the time of the publication of her 2002 book: Technological Revolutions and Financial Capital:  The Dynamics of Bubbles and Golden Ages.  In her book, as well as her many articles and lectures, she frames the turbulent times we are going through into a clear, historical perspective.  Let me briefly summarize her views. 

Over the past couple of centuries, we have had a technology revolution every 40 - 60 years, starting with the Industrial Revolution in 1771, which was characterized by the emergence of machines, factories and canals.  This was followed by the age of steam and coal, iron and railways which started in 1829; steel and heavy engineering (electrical, chemical, civil and naval) starting in 1875; and the age of the automobile in 1908.  Our present information technology and telecommunications age, whose starting point Perez pegs at 1971, is the fifth such major revolution in that span.

Each such technology revolution is composed of two very different periods, each lasting 20 - 30 years.  The installation period is the time of creative destruction, when new technologies emerge from the lab into the marketplace, entrepreneurs start many businesses based on these new technologies, and venture capitalists encourage experimentation with new business models and speculation in new money-making schemes.

Services, Production and Clouds

Last week I wrote about the workshop I recently attended in IBM’s Almaden Research Lab on The Co-Evolution of Future Technologies, Skills, Jobs, and Quality-of-Life.  In particular, I discussed the talk on the IT-based transformation of services given by John Zysman, professor of political science at UC Berkeley and co-director of the Berkeley Roundtable on the International Economy.

Professor Zysman key premise is that the service transformation we are going through is truly an Algorithmic Revolution, that is, the ability to formalize and codify the activities underlying services, express them as computable algorithms and capture them in software.  Given the major advances in the power and costs of information technologies, many more such service activities are now amenable to be transformed into algorithms and embodied in software.  This is profoundly changing the nature of business and just about all other institutions in society, as  they increasingly turn to IT-based services to add value to their offerings.

Over the past several years I have given a number of seminars on technology and innovation in services.  It has been a tough subject to teach.  Services, along with their creation and consumption, have been difficult to define.  So, like many others, I have discussed services and their attributes in contrast to what they are not - physical products that are generally manufactured in factories.  But, while such a definition of services and the service sector of the economy may have worked well in the past, it does not work so well now.      

One of the major consequences of the IT-based transformation of services is the blurring of the lines between manufactured products and products delivered as services, as well as between the industrial and service sectors of the economy.  The distinction between products and services blurs, as physical products are increasingly embedded into service-based solutions, where the clients pay for the services they receive, not the products embodied in the solutions.  The traditional boundaries between the industrial and service sectors of the economy have been breaking down.

The IT-based Transformation of Services

On September 27 I participated in a workshop at IBM’s Almaden Research Lab on The Co-Evolution of Future Technologies, Skills, Jobs, and Quality-of-Life.  The key objective of the workshop was to explore how new technologies are changing the economics of work activities, and to understand the frameworks, theories and models that might help us better deal with these economics changes.

In a series of talks and panel discussions, the workshop first reviewed the historical patterns of technology innovations and economic cycles, and then looked at the future of work economics in three key areas: technologies and societal infrastructure; skills and jobs; and careers and quality-of-life measures.  I gave a presentation on Job Creation, Education and Entrepreneurship in the Emerging Knowledge Economy.

John Zysman, professor of political science at UC Berkeley and co-director of the Berkeley Roundtable on the International Economy, gave an excellent talk on Production, Clouds, and the Transformation of the Global Economy.  He presented a particularly interesting point of view on the ongoing IT-based transformation of services, which he calls an algorithmic revolution.  His discussion of services is largely based on his 2006 paper, The 4th Service Transformation: The Algorithmic Revolution, which was later followed by a more comprehensive 2010 paper, Services with Everything: The ICT-Enabled Digital Transformation of Services co-written with Stuart Feldman, Jonathan Murray, Niels Christian Nielsen and Kenji E. Kushida.

Leadership and Rational Thinking

As part of its Centennial celebrations, IBM held a conference in New York in September - THINK: A Forum on the Future of Leadership.  The two day THINK Forum brought together leaders from government, business and academia  to discuss the future of leadership from a number of angles, including the changing nature of the corporation, the importance of systems thinking, leading in times of deep structural changes and bringing science to leadership.

After the Forum, I spent some time reflecting on what I had heard in the various talks and panels.  Lots of different opinions and points of views were expressed.  But, I believe that an overriding message kept coming through loud and clear: to make progress in addressing the very complex problems that we all face, we must leverage the power of rational thinking.

Several speakers pointed out that the world is no longer just connected via the Internet, as we have been saying for the past fifteen years.  It is now becoming hyperconnected, given the proliferation of smart mobile devices and sensors all around us.  In such a hyperconnected world, changes now propagate and build on each other faster than ever, rendering just about all private and public institutions increasingly unpredictable and harder to manage.

But, a number of speakers also pointed out that our ability to understand the world around us is directly related to the tools at our disposal.  And, we now have an array of highly sophisticated tools that, when properly used, can help us make sense of all that complexity.  In particular, just like centuries ago telescopes enabled us to peer into the sky and better understand the universe and our role in it, information has now become a similar such tool, enabling us to now peer into our highly complex world and help us understand what is going on.

Social Business: e-business 2.0 . . . and Beyond

IBM executive Sandy Carter has just published a new book - Get Bold: Using Social Media to Create a New Type of Social Business.  I know Sandy well, having worked with her at IBM for over a decade in various initiatives.  She is now Vice President of Sales for Social Business and Collaboration Solutions, as well as one of IBM's key Social Business evangelists and strategists.

IBM announced its Social Business initiative in the Fall of 2010.  Its basic definition is simple and practical: A Social Business embraces networks of people to create business value.  When I first learned about Social Business, I thought of it as the natural evolution of IBM’s Internet and e-business strategies.  Let me explain. 

When first announced in 1997, the basic definition of e-business was equally succinct: Leverage the Internet for Business Value.  So, as Web 2.0 technologies and Social Media applications emerged several years later, it is natural that e-business would have now evolved into something like e-business 2.0, continuing to leverage (the more recent advances of) the Internet for business value.  But, as Sandy observes in Get Bold, Social Business goes way beyond its Web 2.0 implications.

The Passing of an Era

The IBM Personal Computer was announced on August 12, 1981.  Having sold its PC business to Lenovo in 2005, IBM itself did not mark the occasion, and as far as I know, neither did Lenovo or the other companies still selling IBM PC’s.  But, the 30th anniversary of this important event in the history of computing was duly noted by IBM Fellow Mark Dean, - a member of the small team that designed the original machine, - in a nostalgic and thoughtful blog.  At the beginning, Mark writes:       

 “It’s amazing to me to think that August 12 marks the 30th anniversary of the IBM Personal Computer.  The announcement helped launch a phenomenon that changed the way we work, play and communicate.  Little did we expect to create an industry that ultimately peaked at more than 300 million unit sales per year.  I’m proud that I was one of a dozen IBM engineers who designed the first machine and was fortunate to have led subsequent IBM PC designs through the 1980s.  It may be odd for me to say this, but I’m also proud IBM decided to leave the personal computer business in 2005, selling our PC division to Lenovo.  While many in the tech industry questioned IBM’s decision to exit the business at the time, it’s now clear that our company was in the vanguard of the post-PC era.”

“I, personally, have moved beyond the PC as well.  My primary computer now is a tablet.  When I helped design the PC, I didn’t think I’d live long enough to witness its decline.  But, while PCs will continue to be much-used devices, they’re no longer at the leading edge of computing.  They’re going the way of the vacuum tube, typewriter, vinyl records, CRT and incandescent light bulbs.”

Innovation and Entrepreneurship in Large Metropolitan Areas

For several years now I have been thinking about the differences between various models of innovation and entrepreneurship.  In particular, I have been trying to understand the key differences between the innovation and entrepreneurship models that have worked so well in Silicon Valley, the Boston-Cambridge area and similar high-tech, VC-based regions, and the models that might be more applicable to large metropolitan areas like New York and London.

The Silicon Valley model has become the most prominent example of what it means to be a successful innovation hub.   Over the last few decades, it has been the leading region in the creation of new high-tech products and companies based on advanced digital technologies.  A number of regions around the world have tried to become the next Silicon Valley, hoping to develop the right ecosystem mixture, - including technologists, entrepreneurs, venture capitalists, - that will establish their region as a world class center of innovation.  Most such efforts have fallen short of their goals, - a number ending in outright failure. 

Why is that?  To begin with, most successful high-tech innovation hubs have been built around the great engineering universities in their midst - Stanford and UC Berkeley for Silicon Valley, and MIT for the Boston area.  Over the years, these regions have brought together and carefully nurtured a large and fluid community of technologists and, entrepreneurs who share ideas as they move from company to company - as founders, employees, managers, board members or investors.  They have achieved a remarkable network effect particularly suitable for high-tech innovation that give them an advantage in scale that is almost impossible to replicate.

Their university-based, high tech innovation model is particularly oriented toward finding The Next Big Thing, - that is, the next big technology, product or service that will take world markets by storm, transform whole industries, and in the process, create considerable wealth for their creators and investors.  They have been at the center of several waves of disruptive innovations and creative destruction over the past few decades, and will continue to do so into the future.

But, beyond a few niche areas, this is not a model of innovation and entrepreneurship generally suitable for a large metropolitan region like New York.  A model that would work for New York has to leverage the key strengths of the region.  Let me briefly discuss three such New York strengths in particular.

The Science of Complex Systems

When I look back over my long, relatively eclectic career, complex systems have been a common theme in all the activities I’ve been involved in.  It started in the 1960s, when I was an undergraduate and graduate student at the University of Chicago majoring in physics, - the study of complex natural systems.  The research for my thesis was focused on the highly complex world of atoms and molecules.  

After finishing my studies, I joined IBM Research in 1970 and switched to computer sciences.  My main research interests were centered on large computer systems, including mainframes, supercomputers and distributed systems.  In the 1990s, my work was focused on the kinds of complex systems and applications made possible by the advent of the Internet, World Wide Web and e-business. 

Then, around ten years ago, I became very interested in the applications of technology to market-facing, people oriented complex systems, especially systems involving services.  Over the past few years, I have also become increasingly interested in economic systems. 

In addition, I started to notice that a number of the concepts and principles I learned many years ago as a physics student seem to apply to these new kind of complex organizational systems I was now interested in.  Let me explain.

The Modernization of the State

I spent the first week of June in Chile, my fourth visit in the last five years.  I was there at the invitation of Fernando Flores, President of the National Innovation Council for Competitiveness, to assist in the Modernization of the State initiative, one of the major projects in the administration of Chile’s President Sebastian Piñera. 
 
The Modernization of the State initiative is being led by Rafael Ariztia under the General Ministry of the Presidency.  It has four major objectives:

• Improve in the overall services that the government provides to all its users
• More effective and efficient government institutions
• Improve the effectiveness of government employees
• Facilitate the decentralization of the country

I first met Fernando Flores during a trip to Chile in August of 2006 to attend an international conference.  He was a Senator then, a position he held from March of 2002 until he was appointed by President Piñera to his present position in March of 2010.  We discovered that we had a lot of common interests and have since become friends.  I continued to meet with him during subsequent visits to Chile.  He knew about my strong interests in cloud computing, and last year I invited him to a briefing on the subject in IBM’s headquarters in Armonk, New York.

Flores has been very involved in the Modernization of the State initiative, both because of his position as president of the Innovation Council as well as his personal expertise in technology and management.  He holds a Ph D from UC Berkeley, has worked extensively in the fields of organizational leadership and human cognition, and written a number of books, including Understanding Computers and Cognition: a New Foundation for Design and Building Trust: in Business, Politics, Relationships and Life.

Flores feels that cloud computing has a major role to play in helping to achieve the objectives of the Modernization initiative.  We talked about his ideas earlier this year, which I thought were right on the mark.  He later invited me to assist in these efforts by coming to Chile for meetings with Rafael Ariztia and the key groups woking on the initiative, as well as to give a seminar explaining cloud computing and its potential to Chilean government officials.