A few recent articles have expressed concerns that big data may be at the peak of inflated expectations in the so-called hype cycle for emerging technologies, and will soon start falling into the trough of disillusionment. This is not uncommon in the early stages of a disruptive technology. The key question is whether the technology will keep falling through the trough and be soon forgotten, or whether it will eventually move on toward the slope of enlightenment on its way to a long life in the plateau of productivity. How can you tell which it is going to be?
In my experience, a disruptive technology will succeed if it can keep attracting serious researchers and analysts, who will, over time, cut through the hype and bring discipline to its development and marketing, coming up with solutions to the many technical obstacles any new innovation encounters, sorting through its unrealistic promises and reframing the scope and timelines of its objectives. The Internet recovered from the hype that led to the dot-com bubble and has gone on to a highly successful future. Cloud computing is now going through a similar period of serious evaluation and development. So is big data.
In The Rise of Big Data: How It's Changing the Way We Think About the World, an article just published in Foreign Affairs, Economist editor Kenneth Cukier and Oxford professor Viktor Mayer-Schönberger do a very nice job in articulating why “big data marks the moment when the information society finally fulfills the promise implied by its name.” The article is adapted from their book Big Data: A Revolution That Will Transform How We Live, Work, and Think published in March, 2013.