For several years now, big data has been one of the hottest topics in the IT industry. But, what do we mean by big data? What are people really excited about? Is it real, hype, or something in between?
A number of recent articles have been sounding the alarm that big data may be at the peak of inflated expectations in Gartner’s hype cycle. For example, Gartner’s Research director Svetlana Sicular recently observed that big data has already reached the peak of the hype cycle, and is now falling into the trough of disillusionment, a necessary step before (hopefully) moving on to the slope of enlightenment. She writes that a number of her most advanced big data clients are starting to get disillusioned:
“These organizations have fascinating ideas, but they are disappointed with the difficulty of figuring out reliable solutions. . . Several days ago, a financial industry client told me that framing a right question to express a game-changing idea is extremely challenging: first, selecting a question from multiple candidates; second, breaking it down to many sub-questions; and, third, answering even one of them reliably. It is hard. Formulating a right question is always hard, but with big data, it is an order of magnitude harder, because you are blazing the trail (not grazing on the green field).”
As is typically the case with major disruptive technologies, many people are waking up to the fact that realizing the value from big data and becoming a data-driven institution is a lot harder and will take longer than they originally anticipated. But, is it worth it? What is this data-driven world all about?
MIT Media Lab Professor Alex “Sandy” Pentland talks about the promise of becoming a data-driven society in a very interesting online conversation, - Reinventing Society in the Wake of Big Data. Pentland is a big data pioneer, whom O’reilly Media founder Tim O’reilly named one of The World’s 7 Most Powerful Data Scientists in Forbes.
“This is the first time in human history that we have the ability to see enough about ourselves that we can hope to actually build social systems that work qualitatively better than the systems we've always had,” says Pentland. “That’s a remarkable change. It’s like the phase transition that happened when writing was developed or when education became ubiquitous, or perhaps when people began being tied together via the Internet.”