Self-Organisation in Agile Teams — How and When Does It Work

Under Construction!

Subscribe to stay informed.

Georges Lteif

Georges Lteif

Software Engineer

Last Updated on February 3, 2023.
Subscribe now to stay posted!
About Us
— min read

1. Overview

I came across the 11th principle of Agile a few years back, and I must admit it hasn’t resonated with my thoughts at the time quite well.

This principle, which you can find in the Agile manifesto, states the following:

The best architectures, requirements, and designs emerge from self-organising teams.
Agile Manifesto

  • First, the relationship between architecture and team structure did not seem obvious or even logical. A bit later, I came across Melvin Conway’s rule, which provided insights into how this relationship might be interpreted. Still, I was not entirely sure, as no explicit reference to this rule is made in the manifesto.
  • Second, team self-organisation seemed counter-intuitive and irrational as it allowed unconstrained and random results–surely not what we would like to happen. In my view, some order was necessary, and a top-down blueprint with explicit processes, roles, and objectives seemed like a sensible approach to building a functioning and effective team.
Complexity, self-organisation, and complicated structures and behaviours are visible everywhere in natural as well as human-created phenomena such as teams, organisations, economies, and financial systems.
Complexity, self-organisation, and complicated structures and behaviours are visible everywhere in natural as well as human-created phenomena such as teams, organisations, economies, and financial systems.

But software architecture and human groups (such as a team of developers) are complex systems for reasons we have thoroughly covered in other articles. As with anything involving complexity, simplistic interpretations like the ones I mentioned above can be safely and promptly discarded without losing meaning or substance.

This article aims to articulate the concepts behind self-organisation, first in nature and then in Agile teams, focusing on their impact on software architecture, design, and requirements, thus covering the 11th principle of Agile.

More precisely, we will define self-organisation in multiple contexts, its manifestations, its domain of applicability, and what we can learn from it regarding organisational theory and Agile practices.

We will examine self-organisation in chaos theorycomplexity sciencecomplex adaptive systems theory, and organisational culture models to obtain a 360 degree perspective.

A note before we proceed. This article focuses solely on the association between Agile teams and architecture, business requirements, and solution design. Still, it provides no introduction to any of these concepts, as we have examined those in comprehensive detail elsewhere.

Therefore, we recommend you first read the two articles in which we have explored the fundamental principles of architecture and design.

Related —

2. Self-Organisation Misinterpretation in the Modern View of Agile Teams

3. Self-Organisation in Nature

3.1 Chaos Theory

The best example to introduce chaos theory can be by examining the job of weather forecasters. Here is what we can say about that:

  • Generating weather forecasts with arbitrary precision requires computing power that can track the motion of every molecule in the atmosphere and much more. Even quantum computers cannot do that today.
  • The weather patterns over short periods (5-10 days) are predictable quantitatively within limits. We can forecast the high and low temperatures, the chance of rain, wind speed, and humidity within an acceptable margin of error.
  • Over longer periods, the weather patterns become unpredictable quantitatively but remain tractable qualitatively. We know what winters and summers look like, and we can safely predict that winters will stay cold and summers hot for at least the next few centuries.
  • No weather predictions (or realisation) will reoccur in precisely the same way other than by pure chance. No storm or rainy day will ever be a precise replica (evolution, wind speed, amount of rainfall and distribution, humidity levels throughout the day) of an older one.

The image below shows an example of a Lorenz attractor where we see the many trajectories running around two attraction centres. No two courses are the same. We know that, ultimately, the path will be around one of the attractors, but we cannot predict exactly which one and the precise shape of the track.

A Lorentz attractor shows irregular but qualitatively-predictable patterns of behaviour.
A Lorenz attractor shows irregular but qualitatively-predictable patterns of behaviour.

You can run the Python code below to see how it works. A (very) technical document on the Lorenz strange attractor is available online.

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

def lorenz_deriv(x, t0, sigma=10, beta=8/3, rho=28):
    x_dot = sigma * (x[1] - x[0])
    y_dot = x[0] * (rho - x[2]) - x[1]
    z_dot = x[0] * x[1] - beta * x[2]
    return np.array([x_dot, y_dot, z_dot])

def lorenz_attractor(x0, t, sigma=10, beta=8/3, rho=28):
    from scipy.integrate import solve_ivp
    sol = solve_ivp(lambda t, x: lorenz_deriv(x, t, sigma, beta, rho), [0, t[-1]], x0, t_eval=t)
    return sol.y

x0 = np.array([1.0, 1.0, 1.0])
t = np.linspace(0, 100, 10000)
xyz = lorenz_attractor(x0, t)

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.plot(xyz[0, :], xyz[1, :], xyz[2, :])
ax.set_xlabel("X Axis")
ax.set_ylabel("Y Axis")
ax.set_zlabel("Z Axis")

Systems presenting chaotic behaviour are typically non-linear and iterative. At equilibrium, they settle into an irregular pattern of behaviour such as the ones described above.

But studies have also shown that under the right conditions, such systems can display other forms of behaviour, called bounded instability far from equilibrium.

Bounded instability can occur when a critical parameter of the system is modified beyond a threshold. Consider, for example, the rising temperature of a body of liquid; beyond a specific value, we start observing convection current, what experts of chaos theory call dissipative structures.

Ilya Prigogine, a physical chemist and Nobel laureate, conducted extensive studies of chaotic systems far from equilibrium. His work showed that dissipative structures result from the internal and spontaneous self-organisation of a chaotic system at critical values of its control parameter (temperature for a liquid).

Convection currents are examples of dissipative structure in chaos theory. They are spontaneous self-organisation patterns displayed when a control parameter of the system (temperature) reaches a critical value.
Convection currents are examples of dissipative structure in chaos theory. They are spontaneous self-organisation patterns displayed when a control parameter of the system (temperature) reaches a critical value.

In this case, spontaneous and unguided self-organisation patterns are fragile and require much energy to maintain their structure.

Drawing parallel conclusions between chaotic systems displaying dissipative structures under conditions of bounded instability far from equilibrium and the behaviour of complex human social groups can be dangerous. It has to do with our ability to influence the future effectively. In Prigogine’s words:

Is the future given, or is it under perpetual construction? One could express the question thus: “Is causality to be understood as formative or […] transformative?”
The End of Certainty – Time, Chaos, and the New Laws of Nature — Prigogine, 1997

Under such conditions, organisation decision-making cannot be effective as causal links between decisions and results break down entirely.

But human systems as inherently complex. If chaos theory is not representative enough, what other model can be examined where self-organisation may occur? In the next section, we investigate one such candidate.

3.2 Complex Adaptive Systems

Complex Adaptive Systems (CAS) are systems composed of many interacting elements capable of self-organisation and adapting to changing conditions. They are often characterised by non-linearity, emergence, and unpredictability. Examples include ecosystems, financial markets, social networks, and the human brain.

A Complex Adaptive System self-organises due to the interactions and feedback between its components. These interactions allow new patterns, behaviours and structures to emerge without needing external control or direction. This self-organisation is driven by the agents in the system, each pursuing their objectives and goals and adapting to the actions of others. The result is a complex, collective behaviour that cannot be reduced to that of individual agents.

External factors can influence the self-organisation of a Complex Adaptive System. These factors can include the environment, external stimuli, and constraints imposed by the surrounding system.

For example, a city’s transportation network organises itself based on its residents’ movement patterns. The availability of public transportation and road infrastructure, government policies, and economic conditions also influences it. The influence of external factors can shape and alter the self-organisation process, leading to different outcomes and behaviours.

The nature of causality in Complex Adaptive Systems is often non-linear and indirect. In traditional cause-and-effect relationships, a specific event or action leads to a predictable outcome. However, in CAS, the relationships between events and results can be much more complex and difficult to predict. The interactions between elements in a CAS can lead to emergent behaviours that cannot be attributed to any single cause but rather to the collective behaviour of the system as a whole. Causality in CAS is often a result of many factors operating together, making it difficult to determine a single cause-and-effect relationship.

The nature of causality in CAS is often influenced by self-organisation in that cause-and-effect relationships can be challenging to identify and predict. This unpredictability is often the result of the collective interactions and feedback between CAS components, making it difficult to determine the source of a particular behaviour.

An example of self-organisation and emerging behaviour in human social groups is forming consensus in group decision-making. In group decision-making, individuals with different perspectives, experiences, and opinions come together to make a collective decision.

The interactions and feedback between individuals, such as sharing information, negotiation, and compromise, drive the self-organisation of group decision-making. The collective behaviour of the group emerges from these interactions, resulting in the emergence of a consensus.

4. Leadership and Self-Organisation in Social Groups

4.1 Formation of Social Groups

4.2 The Role of Leadership

4.3 Informal Networks

4.4 Silos — Good or Bad?

Silos, a phenomenon of compartmentalisation, can emerge in software teams and organisations for several reasons. Some of the common causes include:

  1. Organisational Structure — Silos can be created by how an organisation is structured. For example, if an organisation is divided into functional departments such as development, testing, and operations, teams within these departments may work independently and not collaborate effectively.
  1. Lack of Communication and Collaboration — Silos can also emerge due to poor team communication and collaboration. When teams are not communicating effectively, they may not be aware of what other teams are working on, which can lead to the development of silos.
  1. Territorial Behavior — Siloing can occur when team members become territorial and protective of their areas of expertise, leading to a lack of sharing of information and resources and, therefore, bonding in structured relationships.
  1. Incentives and Rewards — Silos can also be encouraged by how individuals and teams are rewarded and recognised. If rewards are based on individual or team performance rather than overall organisational performance, team members may be less likely to collaborate.
  1. Fear of Failure or Mistakes — When team members fear making mistakes or failing, they group closely and form silos, leading to a lack of trust and collaboration.
  1. Lack of shared goals and vision — The absence of a shared purpose and vision of the project leads to different interpretations of events and actions. People who form similar ideas will create silos.

Pros of Siloing in Agile Software Teams:

  1. Clear lines of responsibility and accountability — This can help ensure tasks are completed on time and to a high standard.
  1. Specialised expertise and knowledge — This fosters within specific project areas and silos.
  1. A sense of identity and pride within a team — This helps build morale and motivation.
  1. Clear hierarchy and chain of command — This can be important in organisations with a solid bureaucratic organisational culture.
  1. Decision-making can be made quickly and efficiently — There is a clear chain of accountability.

Cons of Siloing in Agile Software Teams:

  1. Poor communication and collaboration across different teams, leading to a lack of integration and consistency in the design
  1. Difficulty building trust and developing a sense of shared ownership of the project
  1. Increased risk of inconsistency and gaps in the system’s design
  1. Inflexibility impeding the ability to respond quickly to changes in requirements or design decisions
  1. A lack of cross-functional collaboration can lead to less creative and innovative solutions.

Related —

5. Self-Organising Teams in Agile

5.1 The Importance of Hierarchies

Hierarchies are important; if you don’t institute them formally, the dominant figures in the team will create them anyway.

5.2 Naive Interventionism

5.3 Enabling vs Governing Constraints

Governing and enabling constraints affect the behaviour and performance of Complex Adaptive Systems (CAS).

Both types of constraints (governing and enabling) can also influence the emergence of new patterns (and antipatterns), behaviours (desirable and otherwise), and structures within the system. Understanding their role is essential for directing the system’s evolution.

Governing constraints are limitations or restrictions that affect the system’s behaviour and performance. They can take many forms, such as physical and resource constraints or rules and regulations. Governing constraints can limit the flexibility and adaptability of the system or team and negatively impact performance.

For example:

  • In financial markets, regulatory bodies can affect participants’ behaviour in the market.
  • In an ecosystem, resource constraints, such as the availability of food and water, can determine its inhabitants’ growth and expansion patterns.
  • In human social groups, they can take the forms such as resource limitations, organisational culture and policies, time constraints, or societal norms.

For example, resource constraints (such as time, budget, or management support) can limit a team’s ability to complete a task. Organisational policies, such as communication and decision-making process guidelines, can also act as constraints. On the other hand, resource constraints can drive teams to be more creative and resourceful, while time constraints can drive them to be more focused and efficient.

Understanding and managing team constraints are vital for improving team performance and ensuring the successful completion of tasks.

Enabling constraints, on the other hand, are limitations or restrictions that facilitate the system’s behaviour and can be guidelines, rules, or protocols. Enabling constraints can improve the performance and efficiency of the system or team by providing structure and direction.

The difference between governing and enabling constraints is their impact on the behaviour and performance of the system or team. The former limits behaviour while the latter facilitates it.

The term “enabling constraints” was popularised by the economist and systems thinker Stafford Beer in his book “Brain of the Firm”. He is considered one of the pioneers of cybernetics, whose work focused on using systems thinking and cybernetic principles to improve organisational performance. Beer introduced the concept of enabling constraints, arguing that, when used properly, they could provide guidance and direction for organisations, helping to ensure that they were working towards their goals and objectives.

Enabling constraints can take many forms, such as protocols, guidelines, rules, or best practices. They can be internal to the system or team, such as team processes and procedures, or external, such as industry standards or regulations.

Enabling constraints serve several critical functions in Complex Adaptive Systems and teams:

  1. Clarifying goals and objectives — Enabling constraints can provide clarity and direction to the system or team, helping ensure everyone is working towards the same goals and objectives.
  2. Improving communication — Enabling constraints can improve communication and collaboration within the system or team by establishing clear lines of communication, facilitating information flow, and providing guidelines for interaction.
  3. Enhancing performance — Enabling constraints can improve performance by providing structure and direction to the system in mature processes and best practices accumulated from past experiences.
  4. Facilitating innovation — Enabling constraints can promote innovation by providing a structure for experimentation and learning. They can foster a culture of innovation and continuous improvement by establishing guidelines for exploring new ideas and testing new approaches.

6. Architecture as a Reflection of Organisational Structure

7. Conclusion

Leave a Reply