Complex Organizational Systems

The study of complex systems has been a common arc in my career.  It started with physics at the University of Chicago in the 1960s, where I was studying complex natural systems - atomic and molecular physics in particular.  Later on, when I joined IBM Research and became a computer scientist, my main research interests were centered on large computer systems, including mainframes, supercomputers and distributed systems.  In the last twelve years, my work has focused on the kinds of complex systems made possible by the advent of the Internet and the Web.  Then, in the last five years, my interests have gravitated toward market-facing complex systems involving people and services.

What makes such complex systems complex?  I found the most satisfying answer to this seemingly Socratic question in an excellent paper - Complexity and Robustness - by  professors Jean Carlson and John Doyle from UC Santa Barbara and Cal Tech, respectively.

Complex systems, whether natural or engineered, are composed of many parts.  But it is not the mere number of component parts that makes them complex.  After all, a stone or a table is composed of huge numbers of molecules, yet we would not consider them complex.  According to Carlson and Doyle, a truly complex system must consist of many different kinds of parts, intricate organizations and highly different structures at different levels of scale.  Humans, bacteria, advanced microprocessors, modern airplanes, global enterprises, urban environments, national economies and healthcare delivery are all examples of complex systems exhibiting these massive, heterogeneous, intricate characteristics.

But why are these systems so complex?  Whether they were designed by humans or evolved in nature, why aren't they simpler?  What purpose does this complexity serve?

The answer to these questions is both elegant and profound.  You can find much simpler biological organisms in nature, and you can design far simpler objects.  The key ingredient you give up is not their basic functionality, but their robustness - that is, the ability to survive, for biological organisms, or to perform well, for engineered objects - under lots of different conditions, including the failures of individual components.  Robustness implies the ability to adapt and keep going in spite of a changing environment. 

There is a continuing struggle between complexity and robustness in both evolution and human design.  A kind of survival imperative, whether in biology or engineering, requires that simple, fragile systems become more robust.  But the mechanisms to increase robustness will in turn make the system considerably more complex.  Furthermore, that additional complexity brings with it its own unanticipated failure modes, which are corrected over time with additional robust mechanisms, which then further add to the complexity of the system, and so on.  This balancing act between complexity and robustness is never done.

For example, as part of their evolution, biological organisms - from plants to mammals - have developed highly sophisticated control and regulatory mechanisms designed to help them survive in dramatically fluctuating environments.  In humans these control mechanisms form the Autonomic Nervous System, which includes involuntary functions like breathing, digestion, heart rate and perspiration that must be carefully monitored and regulated to keep us alive. 

These control mechanisms are so sophisticated, especially in higher organisms, that they generally bring along their own problems.  One of the most important protection mechanisms, for example, is the immune system, which guards against disease.  But the immune system is subject to its own serious diseases, such as immunodeficiencies when its activity is abnormally slow, and autoimmunities, which are caused by a hyperactive immune system. 

Computers, airplanes and other machines have become increasingly reliable over the years, including the ability to better tolerate individual component failures.  This enables them to use standard, relatively inexpensive components and keep prices down.  Their sophisticated designs also enable them to adapt to their environment and function adequately under a wide variety of conditions.  But as we know, such robust machines are much more complicated than their simpler predecessors, both in their design and their support requirements.  Furthermore, as machines get increasingly complicated, it becomes practically impossible to identify or test for every possible cause of failures.  So, hopefully rarely, unanticipated failures can still cause catastrophic results.

How about complex social organizations - that is, groups of people pursuing collective goals, such as companies, industries, urban environments, economies and governments?  Do the concepts of complexity and robustness as used by Carlson and Doyle apply to complex organizational systems? 

The key new ingredient introduced by organizational systems is the presence of people as one of their key components.  The concept of robustness applied to such systems implies the ability to perform well despite large variations in the performance of their key components - people - as well as rapidly varying conditions in their environment - the marketplace.  People's behavior clearly exhibits far more variations than the components of a machine or a simpler organism.  And the evolution required of organizational systems to be able to adapt to changing market conditions clearly happens on far more compressed time scales than for biological systems. 

Consequently, understanding and managing complex organizations as holistic systems is a very hard problem, indeed.  Many might say that this is far more of an art or craft than a discipline to which you can apply science and engineering principles.  Despite the difficulty of the challenge, though, it is one we must face up to.  The forces of global integration require far more discipline in how we deal with our increasingly tough business, economic and societal issues.

Fortunately, advances in information and communication technologies are enabling us to develop new kinds of tools and methodologies to better deal with such complex organizational systems.  Let me briefly comment on some of these promising opportunities. 

A critical requirement for dealing with a problem in a more scientific way is information – that is, understanding what’s going on.  Whether in medicine or astronomy, the ability to gather and analyze information makes all the difference in our ability to attack and solve new problems.  As we know, we now have the ability to gather huge amounts of information about the real-time behavior of organizations and markets, which we can then analyze and model with powerful supercomputers so we can make better informed, more intelligent decisions.

In engineering, one of the fundamental principles for improving the productivity and quality of a task is to transform the task into a well defined, repeatable process.  Processes have long been used in the design and manufacturing of all kinds of objects, from cars to semiconductors.  Achieving product excellence at competitive costs generally requires a sharp focus on specialization, standardization and technology, and that is what processes are all about. 

Processes are widely used in business.  Companies generally do very well with highly repetitive, back-stage processes such as those involved in manufacturing, logistics and transactions.  However, productivity and quality have significantly lagged in front-stage processes involving people and services - e.g., marketing, sales, accounting and human resources.  The advent of services sciences in the last five years is all about bringing specialization, standardization and technologies to bear on such front-stage, people-intensive processes in business.

But while technology, information and processes are required, the capacity to innovate and differentiate your business, or region depends on unleashing the imaginations and enterprise of people, both as individuals and as part of innovation communities.  Navigating today’s incredibly complex organizational systems requires highly skilled people.  While they must have strong technical talents, they must also be comfortable working on real-world, market-facing problems, and do so as part of multidisciplinary teams.  Collaborative innovation must be an integral part of their culture.

Let's remember that we have been studying biological and engineering systems in depth for many decades.  But, we have barely scratched the surface in our understanding of complex organizational systems, and how to analyze and manage them.  One hopes that with the new technologies, tools and methods now at our disposal, our progress can now accelerate.  This is truly one of the most fascinating challenges in the decades ahead.

Comments

[jliew]

Hi Irving,

What are your thoughts on sites like Elance, Amazon's Mechanical Turk, Odesk where discrete tasks can now be parceled out to a world-wide market for the best price/talent.

Just curious as to your opinion if these are along the lines of what you mentioned (or completely off).

Warmest regards,

[oehajligs hdanrwfke]

pguqdfoz fhkv rnhuaxqyc sqgntlk jbslozyv atulz uvlo