Complex Systems and the “Rashomon Effect”

I recently read a very interesting NY Times column, The Reality of Quantum Weirdness, by UC Berkeley professor Edward Frenkel.  In the column, professor Frenkel discusses a very deep and important question:  Is there such a thing as a true reality, or “is our belief in a definite, objective, observer-independent reality an illusion?”  The article is about the strange world of quantum mechanics, a world that’s very different from our everyday life experiences.  But part of my fascination with the subject is that I often ask myself similar questions when thinking about the equally mysterious world of highly complex emergent systems, that is, systems where the whole can at times be quite different from the sum of their parts.

Frenkel is an author, - Love and Math is his most recent book, - and a filmmaker in addition to being a mathematician.  He uses the so-called Rashomon effect  to illustrate his points, an effect named after Rashomon, a classic 1950 film by Japanese director Akira Kurosawa, one of the most prominent and influential directors of all time.   

The movie is famous for its novel plot device.  Near Kyoto, a samurai has been killed, but it’s not clear why or by whom.  Four different characters tell widely different versions of the same event:  the samurai’s wife who says she was raped by a bandit, subsequently fainted and then awoke and found her husband dead; a bandit who says he seduced the wife and then killed the samurai in an honorable duel; a woodcutter who says he witnessed the rape and murder but did not want to get involved; and the dead samurai, who speaking through a medium said that the shame of the events he witnessed drove him to kill himself.

The film is an exploration of multiple realities, where it’s not at all clear if there is a real truth, let alone what it might be.  The Rashomon effect has thus come to stand for the contradictory interpretations of the same event by different people.

In the deterministic world of classical mechanics, there’s always a real truth.  The same objects, subject to the same forces, will always yield the same results.  Elegant mathematical models can be used to make perfect predictions within the accuracy of their human-scale measurements.  Early in the 19th century, French mathematician and scientist Pierre-Simon Laplace observed that if we knew the precise state of the universe as represented by the position and speed of every one of its particles, classical mechanics would enable us to calculate all past and future states of the universe. 

But, this predictable world began to fall apart in the early 20th century.  Classical mechanics could not explain the counter-intuitive behavior of energy and matter at atomic and subatomic scales.  19th century classical physics was replaced by a kind of 20th century magical mystery tour, now ruled by the new principles of quantum mechanics. 

At the atomic and subatomic scales that quantum mechanics deals with, the world is far from predictable.  The behavior of a particle at this scale is explained by its wave function, which can only tell us the probability that the particle will be at a specific position at a given time.  According to the Heisenberg uncertainty principle, it’s impossible to simultaneously determine the exact position and velocity of an electron, - or any other particle at these scales, - no matter how good your measurement tools are.  The concept of wave-particle duality further adds that the electron sometimes behaves like a wave and sometime like a particle, depending on what we want to know and what we do to find the answer. 

Frenkel writes about a recent experiment aimed at answering a subject that’s been long debated by quantum physicists.  “Is there a fixed reality apart from our various observations of it?  Or is reality nothing more than a kaleidoscope of infinite possibilities?”  The results of the experiment support the latter scenario, namely, “that there is a Rashomon effect not just in our descriptions of nature, but in nature itself… What this research implies is that we are not just hearing different stories about the electron, one of which may be true.  Rather, there is one true story, but it has many facets, seemingly in contradiction, just like in Rashomon.” 

Other recent papers attempt to explain the difference between a subjective, knowledge-based reality and an observer-independent reality using the metaphor of a meteorologist predicting tomorrow’s weather.  Using classical mechanics and all relevant information, the meteorologist should be able to make an exact forecast since predicting the weather is a deterministic process.  The reasons meteorologists give forecasts based on probabilities is because, given how complex weather systems are, they in fact don’t have access to all relevant information.  Two meteorologist might give different forecasts for tomorrow’s weather if each uses a different model and/or different data to make their predictions.  This is an example of a subjective, knowledge-based reality. 

However, unlike a weather forecast, at atomic scales future behaviors cannot be predicted with certainty even if a person had perfect information about the system in question.  At atomic scales, nature itself is inherently random and probabilistic.  “There is really no escape from the mysterious - some might say, mystical - nature of the quantum world,” notes Frenkel. 

This is all difficult to grasp because the weirdness of quantum mechanics dissipates in the world of everyday experiences.  When dealing with large ensembles of particles, such as a table, a car or the flight of a baseball, we are back to being able to describe their behavior by the rules of classical physics. 

But if so, why can’t we gather enough data and use highly sophisticated models and powerful supercomputers to make precise short and long term weather predictions, figure out what the traffic will be like tomorrow in Manhattan, anticipate the ups-and-downs of the economy or come up with better medical treatments for diabetes and other chronic diseases?  As it turns out, the quantum world is not the only one that exhibits counter-intuitive, somewhat bizarre behaviors.  So does the world of highly complex systems, especially those systems whose components and interrelationships are themselves quite complex.  This is the case with meteorology, systems biology and evolution as well as with sociotechnical systems whose main components are people and their varied interactions, such as healthcare, cities, transportation and economies. 

While in principle such systems are deterministic, in practice our ability to predict their future or past behaviors is severely limited by their inherent complexity, the dynamic nature of their components, their intricate interrelationships, and their high sensitivity to initial conditions, a property called chaos.  The sensitivity of chaotic systems to initial conditions is best captured in the so-called butterfly effect, a concept popularized by MIT meteorologist Edward Norton Lorenz in a 1972 paper Does the flap of a butterfly’s wings in Brazil set off a tornado in Texas?, where he wrote: 

“The question which really interests us is whether…  two particular weather situations differing by as little as the immediate influence of a single butterfly will generally after sufficient time evolve into two situations differing by as much as the presence of a tornado.  In more technical language, is the behavior of the atmosphere unstable with respect to perturbations of small amplitude?” 

Their high sensitivity to initial conditions and their inherent instability means that chaotic systems, while deterministic, will in fact exhibit unpredictable, emergent behaviors.  The next time the butterfly flaps its wings, lots of things will likely be different and its impact may well be felt someplace else entirely, or not at all.  This lack of predictability means that in practice, complex systems are inherently probabilistic, as is the case with weather predictions, the medical treatment of many chronic diseases, traffic patterns in large cities, economic decisions, and so on. 

Despite their probabilistic nature, physicists have made huge progress in shedding light on the atomic and sub-atomic world, including the development of nanotechnology applications where quantum effects are important.  Similarly, recent technological advances can help us better understand and manage highly complex systems.  We now have the ability to gather huge amounts of information about the real-time behavior of such systems, which can be quickly analyzed with powerful supercomputers to help us figure out what’s going on, enabling us to make better informed, smarter decisions. 

We are also better able to develop simulation models of these complex systems, so we can understand how they might behave under widely different conditions.  Since these systems are inherently unpredictable, there is no one model that can accurately tell us about the future.  But, reminiscent of quantum mechanics, we can map out the future states of the system, and compute the probabilities of the different states at different points in time.  We do something similar today with hurricane prediction, where we are not able to tell precisely where the incoming hurricane will hit a few days from now, but we can calculate cones of probabilities reflecting the possible paths the hurricane might take based on the different assumptions and models.

In discipline after discipline, we are beginning to learn how to deal with the very messy world of data science and complex systems, and how to best apply our learning to make good decisions and reasonable predictions.  One of the hardest parts of that learning is the need to let go of our preconceived notions of scientific determinism and get used to living in a world of probabilities, uncertainties, limits to how much we can know, - and Rashomon effects.  Our 21st century magical mystery tour promises to be every bit as exciting as the one we embarked on over a century ago. 

Comments

[Mark Montgomery]

Nicely done Irving--an elegant story telling, despite the absence of mentioning the pioneer of complexity with our friends at SFI:) One of the reasons we are located in NM--have engaged with many up the hill since conceived my classic CAS in our lab in mid 90s--then in N AZ, but it's better being on the ground as I can participate regularly with visiting scientists from all over the world. I am constantly attempting to remind those who will listen that we need to be a bit careful about allowing the magical mystery tour to deflect away from the rigor and discipline of objectivity, as bias is ever present in the CAS we call humans.

[Pasquale Di Cesare]

Implied in Irving’s elegant and thought-provoking post is the ontology of quantum mechanics and in particular the reality of the wave function which is a fascinating subject. I feel a bit uncomfortable with Frenkel’s statement that “There is really no escape from the mysterious - some might say, mystical - nature of the quantum world” which I perceive as opting-out from a deeper and objective discussion. Physics has quite abandoned a serious debate with philosophy as was the case in a not too distant past and I feel we are at a point, in fundamental physics, where this would need to be revamped as several scientists are now trying to. Moreover, cantoning to the framework of physics alone, my feeling is that when we take very seriously the essence of what the two physical theories of Quantum Mechanics and General Relativity tell us about our world, when we take both of them to their extreme consequences and follow mathematical consistency leading to Quantum Gravity (e.g. with Loop Quantum Gravity), we might also have a base for clarifying the discussion on what being a “realist” means which is very much related to the reality of the wave function. It is not clear to me why we do not question reality of the macroscopic world while we seem doing it for the microscopic one. It is inspiring here to cite the intuition of Sir. Roger Penrose where a realistic (and testable) view of the wave function “reducing” to the physical state actually observed (what he calls “OR” or Objective Reduction, see his book The Road To Reality) is very profoundly linked to gravity and to other “mysteries” such as time reversal and the entropy law. I feel that in clarifying these connections will lie a lot of new 21st century science and I fully share Inving’s last enthusiastic sentence!

[DKlemitz]

Wave function collapse. Would have been nice to see some mathematics
-
The quantum state of a physical system is described by a wave function (in turn – an element of a projective Hilbert space). This can be expressed in Dirac or bra–ket notation as a vector.

http://en.wikipedia.org/wiki/Wave_function_collapse#cite_note-6