Big Data is one of the hottest topics out there. Big data is a foundational element in IT’s quartet of Next Big Things: Social, Mobile, Analytics and Cloud. But, as the real world keeps reminding us, it is possible to make bad predictions and decisions even if you use tons of big data to make them. The 9/11 attacks showed how even highly sophisticated intelligence agencies can fail to pick out highly relevant signals amidst the mountains of data being analyzed. Our recent financial crisis showed how even the best and brightest can fail to detect an approaching catastrophic storm. The failure of so many professional forecasters to accurately predict the 2012 presidential election shows that you can find almost any answer you want in all that big data.
Big data is indeed incredibly useful in all kinds of endeavors, but only in the hands of talented professionals who know what they are doing and are aware of its pitfalls and limitations. What are some of these limitations? In thinking about this question over the last few years, I started to notice that a number of subtle, non-intuitive concepts that I learned many years ago as a physics student seem to apply to the world of big data and information-based predictions in highly complex systems. Let me explain.
Over 300 years ago, Isaac Newton laid down the foundations of classical mechanics with the publication of his Laws of Motion. The elegant mathematical models of Newtonian physics depict a world in which objects exhibit deterministic behaviors, that is, the same objects, subject to the same forces, will always yield the same results. These models make perfect predictions within the accuracy of their human-scale measurements. Classical mechanics works exceptionally well for describing the behavior of objects that are more or less observable to the naked eye. It accurately predicts the motion of planets as well as the flight of a baseball.
But, the idea of scientific determinism, which would in principle enable us to predict the future behavior of any object in the universe, began to fall apart in the early 20th century. Classical mechanics could not explain the counter-intuitive and seemingly absurd behavior of energy and matter at atomic as well as cosmological scales. Once you start dealing with atoms, molecules, exotic subatomic particles, black holes and the Big Bang, you find yourself in a whole different world, with somewhat bizarre behaviors like the tunneling effect which are governed by the laws of quantum mechanics and relativity. The orderly, deterministic world of classical physics gives way to a world of wave functions, probability distributions, uncertainty principles, and wave-particle dualities.