A Kind of Cambrian Explosion

Over the last few years, the complex engineering systems community has been turning to biology as a source of new ideas for dealing with the kinds of systems that we are now increasingly able to develop.  As digital components continue to become ever more powerful, inexpensive and standardized, these complex systems are growing in breadth and capabilities at a dramatic rate.  Moreover, the kinds of problems we are now able to address are significantly different from what they were in the past.

IT systems are "going up the stack."  They are increasingly being used for businesses and societal applications that need to be able to change and adapt to whatever is going on in the marketplace, and those changes are coming at a very rapid pace.  These market-facing, people-oriented complex systems need to be flexible and adaptable.  Biological systems clearly exhibit these kinds of properties - flexible, adaptable, highly complex and continuously evolving, - which is why biology has become such a source of inspiration for the study of dynamic, complex engineered systems.

Looking at biology as a metaphor, - and begging forgiveness for any poetic license I might have taken in a subject that I am far from an expert - one can very roughly speaking identify an era where the cell was perfected and single cell organisms made their appearance several billion years ago.  This was followed by  multi-cell organisms a billion years ago or so.  Then, around half a billion years ago another dramatic change took place, which is known as the Cambrian Explosion.  Evolution essentially took off, accelerating by an order of magnitude, ushering in a highly diverse set of organisms that were far larger and more complicated than anything that existed before.

The IT industry has been going through a similar process.  If we think of digital components as playing the role of cells in biology, for the first several decades of the industry our focus was to improve and perfect these components - microprocessors, memory chips, disks and so on.  We were building computer systems that were mostly based on a relatively small number of processors.  The lack of standards made it difficult to cluster or network large numbers of such processors into an integrated system.

Then about ten years ago or so, the basic digital components started to become good-enough, - powerful, reliable and inexpensive.  At the same time, the Internet brought a culture of standards to the IT industry, not only in IP networks, but across just about all aspects of the infrastructure.  The combination of inexpensive standard components and the ability to now aggregate them together with open standards enabled us to start building far more powerful systems using parallel architectures, clustering methodologies, local area networks and the Internet.

I believe that we are now approaching our own Cambrian stage in the IT industry.  In the intervening ten years, technology advances have accelerated tremendously - from the very small, - which has enabled IT capabilities to be integrated into just about everything from mobile devices to automobiles to RFID tags, to medical equipment - to the very large, including petaflops class supercomputers and virtual, data center clusters with close to a million nodes.  The Internet is now enabling us to build highly sophisticated, globally integrated, IT-based enterprises.  When we look into the future, there seems to be no end in sight. 

What do we need to do to help this Cambrian transition along - to much larger and much more capable IT infrastructures, - in as disciplined and orderly a way as possible?  I think that four attributes are paramount:  integration, flexibility, diversity and self-management.   Let me say a few words about each.

Integration - The value of integration has by now been well established.  The Internet and the Web have demonstrated how much more valuable IT infrastructures are when all the components are connected based on open standards and work well with each other.  Service Oriented Architectures (SOA) are making it possible to now integrate software, applications and services of all sorts.  Over time, we will learn how to integrate just about all processes, information and people in a business, an industry ecosystem or even the whole economy, so we can deal with them as holistic systems.   

Flexibility - The development of flexible architectures, that would make it easier for complex systems to respond and adapt to frequent changes, is a very difficult problem.  We can perhaps learn from the evolution of engineering systems.  It is well accepted that the introduction of standard, interchangeable parts was one of the major innovations of the Industrial Revolution.   Modern engineering practices are built around the concept of standard, modular components and the tools, platforms and processes needed to develop complex objects – e.g., cars, buildings and computers, - using these parts. 

One of the biggest promises of SOA is that it will help introduce standard, modular, interchangeable components to the world of software, thus enabling developers to build composite applications and business systems by drawing upon pre-defined components and services.  This would make it much easier to build the systems in the first place and to modify them as needed so they can better respond to changing market conditions, as well as incorporate new technologies and other capabilities as appropriate.

Diversity - Complex biological systems are composed of a variety of parts with very different characteristics.  Some are absolutely critical to the survival of the organism and thus must be extremely robust - hearts and brains perhaps being the prime examples.  At the other end are parts like nails, hair and skin that are more commodity-like, and are continuously falling off and being replaced.  Then there is the need for some highly specialized parts like those involved in vision and hearing.  Not surprisingly, evolution has come up with a diverse set of such parts which form the major building blocks in biological systems or organisms.

In IT system we have often spent time looking for the one computer architecture that fits all workloads.  But, we have learned from experience that there is no one-size-fits-all solution.  The characteristics of mission critical transaction applications are quite different from those of information analysis and searches.  Some workloads do not partition well, and thus require Symmetric Multiprocessing (SMP) architectures, whereas other workloads do very well with massively parallel architectures.  We have also learned that for certain workloads like those used in game players, specialized architectures like the Cell processor can be much more efficient than general purpose processors in terms of performance, cost and power dissipation.  Just like in biology, diversity is a very important attribute of complex IT infrastructures

Self-Management - Finally, we have management, perhaps the grandest challenge of all, because it is directly related to the survival of the system - whether biological or engineered.  In biology, the role of keeping organisms well functioning and alive is handled by the autonomic nervous system.  These autonomic tasks are primarily performed without conscious control on the part of the organism - including humans.

Several years ago, it became clear to us that the same forces that were driving the explosive growth of IT infrastructures were responsible for their biggest problems, namely the almost exponential growth in complexity, and the rising costs of managing this complexity.  In 2001, IBM launched the Autonomic Computing initiative to help us address this growing complexity with the objective of making IT systems increasingly self-managing, that is, self-healing, self-configuring, self-optimizing and self-protecting.  While lots of progress has been made in the management of complex systems, it goes without saying that the kind of Cambrian transition we see in the horizon will exacerbate the need for major breakthroughs in the management of large, complex IT infrastructures.

These four key attributes, - needed to support the accelerated growth in breadth and capabilities of IT infrastructures - are not new.  In fact, they are very similar to what we said five years ago when IBM launched the On Demand Business initiative in October of 2002, and started to define the On Demand Operating Environment.  It is a sign of how fast the IT industry continues to advance, and how integrated it has become into the fabric of business and society, that what looked like leading edge requirements only five years ago - the need for an open, integrated, flexible, heterogeneous infrastructure with self-managing capabilities, - is now viewed as absolutely critical requirements to enable the IT industry to evolve into the future.

Comments

[Herve]

About the relationship between biology and system engineering, one interetsing reading is John Lovelock's GAIA Hypothesis...

[Rafael Corrales]

Irving -

Your posts must take a long time to write! Anyways, I enjoyed the "using biology as a metaphor" and it certainly does seem like we're approaching this Cambrian stage.

"Critics contend that no matter how big computers get, they can't become intelligent until we know how to emulate the brain's functions in software. Not so, retorts Inman Harvey, a mathematician turned roboticist at Britain's University of Sussex. By mimicking evolution, ''it's possible to create artificial brains without really understanding how they work,'' he says. In other words, they could evolve their own internal programming, just as human brains have." From an old BW article

[Ritutapan]

Irving,
It was really a nice reading for me. Thanks a ton.

Your using of biology as the metaphor was interesting and truly contextual.
Self-managable integration of the complex IT industry with the realms of business and society, i.e. 'Cambrian Explosion' -- seems to be the next big thing happening soon.

-- Tapan