Anatomy of a Complex Realtime System

Anatomy of a Complex System

Over the years, Realtime systems have increased in complexity. As the systemsget complex they get harder and harder to develop as it needs a completelydifferent mindset and tools to develop complex systems. Most tools andtechniques that have served us well for years do not scale very well withincreasing complexity of systems.

In the future complexity of man made systems will increasedramatically. How will these systems work and operate? Some of theseanswerscan be found right now by examining biological systems. Current state ofbiological systems represent system architectures that have evolved overmillions of years. 

Most biological systems are several order of magnitudes more complex than man madesystem. Nature has a few billion yearsof experience in developing complex Realtime systems. In this article we willcover some of the characteristics of biological systems and how they caninfluence development of complex Realtime systems in the future.

Nature vs Man Made Systems

The following table covers different characteristics of systems found innature. The implication of these characteristics on complex Realtime systems arealso covered.

Characteristic	Nature	Implications for Complex Realtime Systems
The Cell	All biological systems are object oriented in design. The most basic object is the Cell. The Cell contains the DNA which is interpreted to perform various actions required by the cell.	State Machine is the most basic building block of complex real-time systems. The code executed to run through the state machine is equivalent of the DNA in a cell.
Cell Aggregation	Cells in the body join together to form tissue. Tissues form different organs like stomach, lungs etc. Organs form systems like Respiratory system, digestive system.	Complex real-time systems would be a sea of state machine objects interacting with each other. There will be low level state machines which would work together to implement high level state machine behavior.
Layered Architecture	The biological systems have a layered architecture and interactions between objects at high level are very well defined. For example the interactions between the stomach and intestines are fairly well defined. However there are hardly any interactions between a cell in the stomach and a cell in the intestine. The interactions take place at clearly defined interface points.	Complex systems should be designed in a layered fashion. The system is divided into sub-systems with clearly defined interfaces between sub-systems. Very few objects in a sub-system should interact with objects in the other sub-system. For example, a telephone switching system would consist of call processing, maintenance, O&M sub-systems. Most objects in the call processing sub-system will interact only with other objects in the same sub-system.
Cell Inheritance	Cells in the human body have the same basic structure with a cell wall, cytoplasm, nucleus, DNA, RNA etc. Individual cells have further refinements to the basic structure to perform the specific function assigned to them.	In complex real-time systems all state machine objects in a sub-system share common characteristics. These characteristics can be abstracted in a common base state machine class. The individual state machines inherit from the base state machine class. For more details, refer to the article on state machine inheritance.
Organism Inheritance	Most of the animal and plant kingdoms are organized as a deep hierarchy. Animals and plants are organized as families. For example, the cat family contains the lion, cat, tiger etc.	Extremely complex systems will probably never get developed from scratch. Such products would be developed by refining and recombining existing, well tested products. All products based on the same basic products would form a product family.
Evolution and Mutation	All biological systems have evolved over a millions of years, with each generation evolving to better adjust to the changing environment. Most changes take place gradually but sometimes there were quite sudden and major changes caused by mutations.	The only way to develop complex software systems is to follow an evolutionary software release development schedule. The first release implements the basic cross-section of the system. Subsequent releases refine the software in small steps. This development lifecycle minimizes development risk by integrating the product at every stage of development. Sometimes sudden changes in business environment, technology or competitive landscape require sudden changes, similar to mutations.
Animal and Plant Kingdoms	Most complex living organisms on Earth fall into animal and plant classifications. From organization point of view, plants are completely distributed system with no centralized control. Animals have centralized control in form of a central nervous system and brain.	Complex Realtime systems can generally be designed with two techniques: Distributed processing with overall centralized control (Animal). In such systems a central processor would keep track of overall operation of the system. Distributed processing with little or no centralized control (Plant).
Asynchronous Communication	Most interactions in nature are asynchronous, i.e. the initiator of the interaction is free to process other interactions while waiting for a response. In many cases the send message and the response are processed quite independently.	Most interactions in Realtime systems are already asynchronous in nature. i.e. sender of a message doesn't block to receive the response. Other messages will be processed by the sender. A response will be processed as and when it is received. This contrasts with communication techniques like CORBA, RPC where the sender blocks till a response is received.
Immune System	Most biological systems are able to recover from a large number of fault conditions. These cases range from re-growing organs, recovering from a cut in the skin to fighting with invading virus and bacteria. Due to these "fault handling" techniques the human body is able to recover from most fault conditions. We need to see a doctor only if the natural defenses fail.	As man made systems increase in complexity, the systems will need to be capable of recovering from all kinds of software and hardware failure conditions. This will become essential as cost of maintaining complex systems would be prohibitive if the operator is expected to fix every little problem in the system.
Load Sharing Redundancy	Nature rarely uses one-for-one redundancy. Most redundant systems are load sharing, e.g. lungs, liver etc.	We have seen an increasing trend towards load sharing over one-for-one redundancy. In general load sharing systems scale better with increasing load. Failure of a load sharing system will never knock out the complete system.
Emotions and Unpredictable Behaviour	Complexity and random, unpredictable behavior go hand in hand. Most higher order organisms exhibit emotions and unpredictable behavior.	As man made systems become more and more complex, systems will sometimes behave in completely unpredictable fashion. This would happen because it is impossible to test a complex system completely. As complexity increases, so will the number of untested scenarios. So be prepared to baby sit the system during sleepless tension filled nights!