Explaining Complex Systems in Organisations — An Indispensable Guide for Software Developers
1. What Is a Complex System?
Complex systems are collections of interdependent components that interact nonlinearly to produce emergent behaviours. These behaviours arise from the interactions between components and are not predictable based on individual components alone.
Complex systems are found in various domains, such as the natural world, human-made structures, and social and economic systems.
Related —
2. Basic Concepts of Complex Systems
The study of complex systems aims to understand the behaviour of these systems and develop methods for managing them. Here are some key concepts in the study of complex systems:
- Emergent behaviour — Refers to collective patterns of behaviours arising from the components’ interactions and the system’s information exchange with its environment. These behaviours cannot be predicted from the individual components.
- Interconnectedness — The components of a complex system are connected through a network of relationships and interactions. This interconnectedness is a factor in emergent behaviour, and the changing nature of these connections (under internal or external influences) makes prediction impossible.
- Unpredictability — This attribute is due to dense networks between components, nonlinear relationships, and feedback loops. Cause and effect are lost to the observer in complex systems, and long-term predictions are impossible. Complex systems are said to have non-deterministic dispositions towards change in specific directions.
- Self-organization — Refers to the process by which a complex system evolves to a stable state by changing the interactions of its components. Developing new structures and behavioural patterns in a complex system often result from self-organization.
Understanding these basic concepts is essential for effectively managing the behaviour of complex systems. In future sections, we will explore how these concepts apply to specific domains, such as software development, business management, and knowledge management.
Related —
3. Characteristics of Complex Systems in Software Development
Software development projects can be complex (rather than ordered or complicated) if they include novel technology, high customer interactions (observed as high business requirements volatility), or increased uncertainty.
Below are some of the most prominent characteristics of such projects.
3.1 Nature of Causality
The relationships between components in complex software systems are often non-linear, meaning that small changes in one component can result in radical changes in another.
Non-linearity and dense networks of interconnections can make predicting the system’s behaviour based on individual components impossible.
For example, changing team structures, tools, or processes to achieve a specified objective with one purpose may create unanticipated side effects (aided by positive feedback loops) that ultimately undermine them.
3.2 Constraints and Boundaries
Software development projects are often subject to constraints and feedback loops that shape their behaviour.
For example, budget and timeline constraints can affect the choices made by the development team.
Systems are also defined by their boundaries, which is another way of thinking about constraints. For example, team boundaries are created by setting rigid team structures, roles, and objectives, which are subsequently reflected in the system’s architecture.
Related —
3.3 Feedback Loops
Negative feedback loops are a regulatory mechanism in which a change in a system triggers a response that opposes the change, thus stabilising the system’s original state. These loops are essential for maintaining homeostasis and preventing excessive deviations from the set point in biological, physical, and social systems.
Positive feedback loops are a regulatory mechanism in which a change in a system triggers a response that amplifies the change, leading to a further deviation from the system’s original state. These loops can create self-reinforcing cycles, often resulting in rapid and dramatic changes. Positive feedback loops are important in growth, development, and physiological responses but can also lead to unstable or chaotic behaviour in some systems.
For example, a software system’s design shift can affect the development timeline, leading to cascading effects on the project schedule and further constraints on the development team.
Related —
3.4 Novel Behaviour
Complex software systems can exhibit new and unexpected behaviour due to internal interactions between components stimulated by external environmental changes.
This novelty can be seen in phenomena such as the creation of organisational culture in a group of individuals. Novelty can also be observed in the evolution of customer requirements as preferences change after interacting with the system.
3.5 Self-Organisation
Complex systems can self-organise to reach a low energy state of equilibrium by forming connections between their components.
For example, in Agile software development, the flexible and adaptive nature of the process can result in self-organisation which can be reflected in the software architecture and design.
Self-organisation also appears in team hierarchies, especially informal ones.
Related —
3.6 Equilibrium States and Strange Attractors
Complex systems can reach a state of equilibrium through self-organisation, where the interactions between components result in stable behaviour.
For example, market supply and demand constantly reshape supplier-vendor relationships, driving the system to new equilibrium states.
However, strange attractors can sometimes emerge, leading to unpredictable behaviour, even in the short term.
For example, a team’s attitude towards quality and responsiveness in software development might shift depending on the current SDLC stage. A typical example would be the (sometimes unexplainable) relaxed observation of processes and best practices when fixing bugs versus implementing new features.
4. Why is it Important to Understand Complex Systems?
Complex systems are inherently different from ordered systems, and understanding the differences is critical for effective management and decision-making. The following are three key reasons why we think this is true.
4.1 The Risk of Mistaking a Complex System for an Ordered One
One of the highest risks in dealing with complex systems is mistaking them for ordered ones. When this happens, efforts to control and manage the system based on traditional approaches will likely fail.
For example, a common mistake in software development is assuming that a software project can be managed like a linear process with a clear beginning, middle, and end. This assumption does not age well and often leads to a rigid Waterfall approach, ill-suited to handle software projects’ complex, adaptive nature.
On the other hand, treating all software development projects as complex and, therefore, requiring (costly) Agile approaches is equally likely to fail.
The key message here is that, just like in software design, context is essential, and solutions have a bounded domain of applicability.
Related —
4.2 Abandoning the Notion of Long-Term Prediction and Control
Another critical aspect of understanding complex systems is recognizing that they are inherently unpredictable in the long term. Efforts to predict the future or control the outcome of a complex system are generally futile.
In software development, for example, it is not possible to predict with certainty the exact outcome of a project or to control all aspects of the development process. This unpredictability is compounded by size, novelty, and uncertainty, for example, in mega-projects.
Instead, an Agile approach combined with a modular design creates a flexible and adaptive process that can evolve as the project progresses. By embracing uncertainty and unpredictability, software development teams can be more effective in delivering successful projects.
4.3 Understanding the Agency of Individuals
The behaviour of complex systems is shaped by the interactions of the individual components within them.
While these components may have some level of agency, the entire system’s behaviour is not necessarily a result of the individual choices of these components.
In software development, individual developers can influence local decisions and outcomes. However, the group behaviour and the project outcome will result from interactions between all the stakeholders.
Understanding how individual roles in complex systems can influence their behaviour is crucial for developing effective strategies not centred around leaders only and that avoid the long-term, centralized, command-and-control paradigm.
5. Approaches to Managing Complex Systems in Organizations and Teams
Complex systems in organisations and teams can be challenging to manage due to the inherent nature of their structure and behaviour. It is vital to consider four fundamental concepts of good management to overcome these challenges:
- Systems Thinking
- Cybernetics
- The role of informal networks
- Dispositions in complex systems
5.1 Systems Thinking
This approach to making sense of organisations is a radical shift from past paradigms that place the individual at the centre of the picture.
Systems thinking focuses on the system rather than individual components by considering the interactions and interdependencies between components.
5.2 Cybernetics
Cybernetics is a field of study that explores the science of control and communication in complex systems. The principles of cybernetics have been applied to various fields, including biology, psychology, engineering, and management.
When applied to organisations and teams, cybernetics provides a framework for understanding the role of feedback and regulation in the functioning of complex systems.
By focusing on the flow of information and control, cybernetics can help organisations and teams better understand the relationships between different system components and develop strategies for managing these relationships.
5.3 Informal Networks
Informal networks are individuals interacting informally (outside formal structures or hierarchies) to exchange information, ideas, and support.
These networks often form naturally within organisations and teams and can play a significant role in managing complex systems. For example, in software development teams, informal networks provide fallback mechanisms when formal processes break down or cannot respond adequately in challenging situations.
These networks facilitate the flow of information and ideas and provide a support system for individuals as they work to manage complex systems.
5.4 Dispositionality
Dispositionality refers to the propensity of a complex system to behave in a certain way based on its internal structure and external environment.
By understanding the dispositionality of the system, managers can work to amplify desired effects and dampen adverse ones. This approach contrasts command-and-control methods that identify a target state and attempt to close the gap.
6. The Role of Processes and Procedures
Processes and procedures play a crucial role in managing complex systems. However, their impact on the system can be positive or negative, and understanding their impact is essential to manage complex systems effectively.
6.1 What Roles Do Processes Play in Managing Complex Systems?
Processes provide instructions for people to follow to achieve specific outcomes. In complex systems, procedures and best practices can guide implementations, ensuring everyone is on the same page and working towards the same goals.
For example, in software development, processes such as the Software Development Lifecycle (SDLC) can ensure that projects are delivered on time, within budget, and to the expected quality standards.
Related —
6.2 How Mature Processes Can Serve as Knowledge Storage Devices
Processes can act as knowledge storage devices, capturing the expertise and experience of the team so that it is not lost when team members change. This characteristic can be vital in complex systems where knowledge is often decentralized and dispersed across the organization.
For example, in software development, a mature Agile process can provide a framework for the team to work within, allowing for flexibility and adaptability while still providing structure and guidance. By capturing the knowledge and experience of the team in the process, future teams can benefit from the hard-won lessons learned by previous groups, reducing the risk of repeating the same mistakes.
6.3 How Rigid Processes Accelerate Complex Systems Failure
However, overly rigid processes can sometimes lead to problems. Complex systems are characterized by change and unpredictability, and rigid processes can hinder the ability of the system to adapt.
Applying best practices designed from past experiences to novel situations is, by definition, flawed and can lead to destructive behaviours as it over-constrains systems that are not easily restrained.
The best example software developers are familiar with is following a rigid, sequential process such as the Waterfall model in projects where requirement volatility (and therefore uncertainty) are high, as the process does not allow for experimentation or short feedback loops.
Overconstrained systems are more prone to catastrophic failures (Black Swans) as risks are hidden and force maintains order.
In addition, rigid processes can create a culture of conformity and stifle creativity and innovation. Complex systems thrive on diversity and creativity, and inflexible processes can limit the ability of the system to generate new solutions to problems.
Related —
7. Final Words
Complex systems are a fascinating aspect of our lives and our world. They encompass everything from software development projects to the structures and behaviours of human organizations.
Understanding complex systems is critical because they differ from ordered systems in ways that significantly affect the designing and managing of interventions effective in producing desired outcomes.
The study of complex systems involves several basic concepts, such as the nature of causality, the role of constraints and feedback, and the emergence of novel behaviour. It also requires a systems thinking approach, a recognition of the boundaries and connectedness, and an appreciation of the importance of self-organization and knowledge management.
Challenges in managing complex systems in organizations and teams are significant. They include misperceptions about their nature, difficulties in predicting and controlling their behaviour, and the need for effective processes and procedures to support knowledge storage and decision-making.
Approaches such as cybernetics and systems thinking have been developed to help organizations better manage complex systems, but each has pros and cons.
The challenges of managing complex systems can be addressed in the software development industry by understanding Agile, Waterfall, and scientific management methods.
These methods offer different approaches to working with uncertain and unpredictable situations, with Agile and Waterfall providing a more flexible approach and scientific management offering a more rigid structure.
The best approach for a given organization will depend on its specific context and the software products and technologies it adopts.
Operational excellence is critical to obtaining a competitive edge and increasing an organisation’s value proposition, as it allows organisations to effectively manage their resources and achieve their goals.
In conclusion, understanding the nature of complex systems, their challenges, and the approaches to managing them is essential for success in the software development industry and other complex systems.
Effective management of complex systems requires a balance of rigidity and flexibility, understanding processes and procedures, and focusing on operational excellence.