Self-Organisation in Agile Teams — How and When Is It Effective

1. Overview

I came across the 11th principle of Agile a few years back, and I must admit it hasn’t resonated with my thoughts at the time quite well. This principle, which you can find in the Agile manifesto, states the following:

First, the relationship between architecture and team structure did not seem obvious or logical. A bit later, I came across Melvin Conway’s rule, which provided insights into how this relationship might be interpreted. Still, I was not entirely sure, as no explicit reference to this rule is made in the manifesto.

Second, team self-organisation seemed counter-intuitive and irrational as it allowed unconstrained and random results–surely not what we would like to happen. In my view, some order was necessary, and a top-down blueprint with explicit processes, roles, and objectives seemed like a sensible approach to building a functioning and effective team.

But software architecture and human groups (such as a team of developers) can be modelled as complex adaptive systems for reasons we have thoroughly covered in other articles. As with anything involving complexity, simplistic interpretations like the ones I mentioned above can be safely and promptly discarded in favour of a more rigorous one.

This article aims to articulate the concepts behind self-organisation, first in nature to get some background on the concepts, and then in Agile teams, focusing on its impact on software architecture, design, and requirements, thus covering the 11th principle of Agile.

More precisely, we will define self-organisation in multiple contexts, its manifestations, its domain of applicability, and what we can learn from it regarding organisational theory and Agile practices. We will examine self-organisation in chaos theory, complexity science, complex adaptive systems theory, and organisational culture models to obtain a 360-degree perspective.

A note before we proceed. This article focuses solely on the association between Agile teams and architecture, business requirements, and solution design. Still, it does not introduce these concepts, as we have examined them in comprehensive detail elsewhere. Therefore, we recommend you first read the two articles in which we have explored the fundamental principles of architecture and design.

2. What to Expect

The article will be structured as follows:

1. First, we introduce the problem through a list of potential issues with self-organisation when left unguided.
2. Next, we discuss three concepts that will help us understand self-organization:
   • Complex Adaptive Systems
   • Hierarchy Emergence in Newly-Formed Social Groups
   • Silos as a form of self-organisation.
3. Self-Organising Teams in Agile, with a focus on hierarchies, naive interventionism from above, and how constraints can shape self-organisation
4. Finally, how self-organisation can impact software architecture.

3. Agile Self-Organising Teams – Problem Description

What results can be expected of Agile teams attempting to self-organise? The following scenarios will hopefully illustrate aspects of the problem this article will discuss.

1. Lack of Direction: Without clear leadership or direction, self-organising teams may struggle to prioritize tasks and achieve their goals. This could result in confusion, missed deadlines, and a lack of focus.
2. Groupthink: Self-organising teams may be prone to groupthink, which occurs when people prioritize harmony and conformity over critical thinking and rational decision-making.
3. Resistance to Change: Self-organising teams may resist changes that they perceive as threatening their autonomy or decision-making power.
4. Lack of Accountability: Without clear lines of accountability, self-organising teams may struggle to identify and address problems or inefficiencies within the team.
5. Self-organisation Will Occur Anyway: Without a formal hierarchy, an informal one will be created around the dominant members of the group, regardless of seniority, expertise, or merit.
6. Self-organisation and Software Architecture: Perhaps by creating self-sufficient teams with the right skills and competitors, quality design and code will ensue. The relationship with architecture, as described in the Agile manifesto, remains to be clarified.

While self-organisation is a core principle of agile development, it may not work in all situations or for all teams. It can be challenging to implement and maintain, particularly for larger teams or complex projects. However, when implemented correctly, self-organising teams can be highly effective, promoting collaboration, creativity, and adaptability.

4. Complex Adaptive Systems

4.1 Overview

Complex Adaptive Systems (CAS) are systems composed of many interacting elements capable of self-organisation and adapting to changing conditions. They are often characterised by non-linearity, emergence, and unpredictability. Examples include ecosystems, financial markets, social networks, and the human brain.

4.2 Self-Organisation in Complex Systems

A Complex Adaptive System self-organises due to the interactions and feedback between its components. These interactions allow new patterns, behaviours and structures to emerge without needing external control or direction. The agents in the system drive this self-organisation, each pursuing their objectives and goals and adapting to the actions of others. The result is a complex, collective behaviour that cannot be reduced to that of individual agents.

External factors can influence the self-organisation of a Complex Adaptive System. These factors include the environment, external stimuli, and constraints the surrounding system imposes.

For example, a city’s transportation network organises itself based on its residents’ movement patterns. The availability of public transportation and road infrastructure, government policies, and economic conditions also influences it. The influence of external factors can shape and alter the self-organisation process, leading to different outcomes and behaviours.

4.3 Complex Systems and the Nature of Causality

The nature of causality in Complex Adaptive Systems is often non-linear and indirect. In traditional cause-and-effect relationships, a specific event or action leads to a predictable outcome. However, in CAS, the relationships between events and results can be much more complex and difficult to predict. The interactions between elements in a CAS can lead to emergent behaviours that cannot be attributed to any single cause but rather to the collective behaviour of the system as a whole. Causality in CAS is often a result of many factors operating together, making it difficult to determine a single cause-and-effect relationship.

The nature of causality in CAS is often influenced by self-organisation in that cause-and-effect relationships can be challenging to identify and predict. This unpredictability is often the result of the collective interactions and feedback between CAS components, making it difficult to determine the source of a particular behaviour.

4.3 Examples of Self-Organisation

An example of self-organisation and emerging behaviour in human social groups is forming consensus in group decision-making. In group decision-making, individuals with different perspectives, experiences, and opinions come together to make a collective decision.

The interactions and feedback between individuals, such as sharing information, negotiation, and compromise, drive the self-organisation of group decision-making. The collective behaviour of the group emerges from these interactions, resulting in the emergence of a consensus.

5. Hierarchy Emergence in Newly-Formed Social Groups

5.1 Influence, Expertise, and Social Dominance

In a newly formed group without a formal structure, power hierarchies can emerge through various mechanisms, including informal influence, expertise, and social dominance.

Informal influence is a common way power hierarchies emerge in a new group. Members may naturally gravitate towards those who have charismatic personalities, good communication skills, or display leadership qualities. These individuals may become informal leaders and gain influence over other group members. They may have more sway in decision-making or be looked upon to provide direction or guidance.

Expertise is another mechanism through which power hierarchies can emerge in a new group. Members with specialized knowledge or skills relevant to the group’s task or purpose may be viewed as experts and gain influence over others. Other members may defer to these experts’ opinions or decisions, perceiving them as more knowledgeable or competent.

Social dominance is a third mechanism that can lead to power hierarchies in a new group. Members who are more dominant or assertive may assert their authority and influence over others. This can manifest through behaviours such as interrupting others, speaking more frequently or loudly, or dominating conversations. Over time, these dominant individuals may emerge as the group’s de facto leaders or decision-makers.

It’s worth noting that power hierarchies can positively and negatively affect group dynamics. On the one hand, they can provide direction and coordination, facilitate decision-making, and promote group cohesion. On the other hand, they can also lead to conflict, exclusion, and unequal distribution of resources and opportunities.

Power hierarchies can emerge in a newly formed group without a formal structure through informal influence, expertise, and social dominance. Understanding how these dynamics play out in a particular group can help to manage and mitigate potential adverse effects and promote positive outcomes.

5.2 Documented Studies

Several documented experiments demonstrate the emergence of power hierarchies in newly formed groups without a formal structure. Here are a few examples:

1. The Stanford Prison Experiment: This experiment, conducted by psychologist Philip Zimbardo in 1971, randomly assigned participants to act as prisoners or guards in a simulated prison environment. Within a short period, the guards began exerting authority over the prisoners, using increasingly aggressive tactics and displaying dominance over their charges. The study showed how power dynamics could quickly emerge in a new group, even without formal authority structures.
2. The Chickens and the Pecking Order: In a classic study on social dominance in chickens, researchers observed how a new group of chickens quickly established a pecking order, with some individuals dominating others. The researchers found that the initial order was based on factors such as size and aggression but that other factors, such as social learning and experience, played a role over time.
3. The Group Decision-Making Experiment: In a 2007 study, researchers assigned participants to groups tasked with deciding on various scenarios. The study found that even in groups with no formal leadership or authority structure, individuals with higher confidence, assertiveness, and verbal fluency tended to influence the group’s decision-making process more.

These experiments demonstrate how power hierarchies can emerge in newly formed groups through various mechanisms, including dominance, social learning, and expertise. While these hierarchies can have positive and negative effects, understanding their dynamics can help manage and mitigate their potential adverse outcomes.

6. Silos — Good or Bad?

Creating cross-functional teams is one of the pillars of Agile, and its justification lies in the fact that silos are considered an anti-pattern. In this section, we present both sides of the story.

6.1 Overview

Silos, a phenomenon of compartmentalisation, can emerge in software teams and organisations for several reasons. Some of the common causes include:

6.2 Pros of Siloing in Agile Software Teams

6.3 Cons of Siloing in Agile Software Teams

6.4 Summary

It seems that silos are inevitable if we want growth through specialisation, quick decision-making processes, and creating a sense of identity and coherence in a group. On the other hand, silos can dictate the system’s architecture and are not necessarily based on sound premises.

7. Agile Self-Organising Teams

Let’s summarize the above ideas and derive conclusions to aid our discussion on Agile self-organising teams.

7.1 The Importance of Hierarchies

Hierarchies are inevitable; if you don’t institute them formally, the dominant figures in the team will create them anyway, and not necessarily on sound premises. Even when a formal hierarchical structure has been defined by senior management, it will face challenges in consolidating its position and proving its legitimacy. In extreme cases, alliances within the group might form to disrupt or obstruct its decisions and plans.

Where does that leave us with Agile teams? This author’s views are as follows:

1. Creating, announcing, and supporting formal hierarchies is vital in any team, including Agile ones. The hierarchies can be simple (single reporting lines) or complex (multiple reporting lines). Either way, putting a structure in place conveys how senior management wants to run the organisation and what objectives should be achieved.
2. Leaders within the hierarchy must prove their legitimacy by supporting the team members (by helping them solve technical and organisational challenges) and advancing the organisation’s long-term goals.
3. Hierarchies provide an organisational structure, create and maintain communication channels, and provide a decision-making framework.
4. Hierarchies should be designed to facilitate decision-making and optimize productivity. This makes it a bit more than an engineering exercise since, for complex problems, multiple viable solutions can be engineered.

The Agile manifesto principle related to the self-organisation of Agile teams is pretty short and, I think, provides ample opportunities for interpretation (or misinterpretation). Without overruling the principle, which, in essence, was meant to address issues of siloing by creating cross-functional teams and facilitating interactions and discussions, our view is that something is missing on the implementation side, and the ideas presented above should hopefully fill the gap.

7.2 Naive Interventionism

One argument I constantly hear from promoters of self-organising Agile teams is that with self-organisation comes the freedom of making decisions on specific tasks. This cherished sentiment of empowerment seems vital to people of that school of thought, and my views are very much in sync there and run as follows:

1. It is best if the team has the necessary knowledge and skills to distribute the roles according to the skills and talents of its members.
2. Because a human group is also a complex system, what Taleb calls “Naive Interventionism” in his book Antifragile can be catastrophic. Any action on a complex system will have unintended (and unpredictable) consequences, and a strategy that consists of setting remote, long-term goals followed by an action plan might be futile. A better strategy is to observe the team closely, reinforce positive behavioural patterns through funding, resourcing, and support, and discourage undesirable behavioural patterns.

So what is the role of leadership in this context? Leadership can allocate resources and implement a system of rewards and punishments. These powerful tools can provide the necessary constraints (more on that next) so that chaos collapses and order emerges.

7.3 Enabling vs Governing Constraints

Governing and enabling constraints affect the behaviour and performance of Complex Adaptive Systems (CAS).

Both types of constraints (governing and enabling) can also influence the emergence of new patterns (and antipatterns), behaviours (desirable and otherwise), and structures within the system. Understanding their role is essential for directing the system’s evolution.

7.3.1 Governing Constraints

Governing constraints are limitations or restrictions that affect the system’s behaviour and performance. They can take many forms, such as physical and resource constraints or rules and regulations. Governing constraints can limit the flexibility and adaptability of the system or team and negatively impact performance.

For example:

For example, resource constraints (such as time, budget, or management support) can limit a team’s ability to complete a task. Organisational policies, such as communication and decision-making process guidelines, can also act as constraints. On the other hand, resource constraints can drive teams to be more creative and resourceful, while time constraints can drive them to be more focused and efficient.

Understanding and managing team constraints are vital for improving team performance and ensuring the successful completion of tasks.

7.3.2 Enabling Constraints

Enabling constraints are limitations or restrictions that facilitate the system’s behaviour and can be guidelines, rules, or protocols. Enabling constraints can improve the performance and efficiency of the system or team by providing structure and direction.

The difference between governing and enabling constraints is their impact on the behaviour and performance of the system or team. The former limits behaviour while the latter facilitates it.

The term “enabling constraints” was popularised by the economist and systems thinker Stafford Beer in his book “Brain of the Firm”. He is considered one of the pioneers of cybernetics, whose work focused on using systems thinking and cybernetic principles to improve organisational performance. Beer introduced the concept of enabling constraints, arguing that, when used properly, they could provide guidance and direction for organisations, helping to ensure that they were working towards their goals and objectives.

Enabling constraints can take many forms, including protocols, guidelines, rules, or best practices. They can be internal to the system or team, such as team processes and procedures, or external, such as industry standards or regulations.

Enabling constraints serves several critical functions in Complex Adaptive Systems and teams:

1. Clarifying goals and objectives — Enabling constraints can provide clarity and direction to the system or team, helping ensure everyone is working towards the same goals and objectives.
2. Improving communication — Enabling constraints can improve communication and collaboration within the system or team by establishing clear lines of communication, facilitating information flow, and providing guidelines for interaction.
3. Enhancing performance — Enabling constraints can improve performance by providing structure and direction to the system in mature processes and best practices accumulated from past experiences.
4. Facilitating innovation — Enabling constraints can promote innovation by providing a structure for experimentation and learning. They can foster a culture of innovation and continuous improvement by establishing guidelines for exploring new ideas and testing new approaches.

8. Architecture as a Reflection of Organisational Structure

Self-organisation is just half the story in Agile’s 11th principle: “The best architectures, requirements, and designs emerge from self-organising teams”, so where do requirements, design, and architecture fit into the picture?

We mentioned Conway’s Law at the beginning of this discussion; let’s discuss it in some detail.

8.1 Conway’s Law

Melvin Conway first introduced his law in an article titled “How Do Committees Invent?” published in the April 1968 issue of the journal “Datamation”. The article was based on Conway’s experiences as a programmer and consultant on large software projects. It explored the challenges organisations face when designing and developing complex systems. The article presented Conway’s law as an observation about the relationship between an organization’s structure and the resulting design of the systems it creates. Since then, Conway’s law has become a widely cited principle in software engineering and organizational theory.

Melvin Conway’s famous rule on software architecture states that “organizations which design systems… are constrained to produce designs which are copies of the communication structures of these organizations.”

In other words, Conway’s law states that the architecture of a software system will mirror the structure of the organization that created it. For example, suppose a team is organized into separate sub-teams responsible for different functional areas of the software. The resulting software system will likely comprise separate components responsible for those functional areas.

Conway’s law has important implications for software development teams, as it suggests that to create a well-designed software system, the organizational structure of the development team must also be well-designed. It also suggests that if changes are needed in the software system’s architecture, it may also be necessary to change the organization.

8.2 Implications of Conway’s Law on Agile Team Structure

Agile teams (or squads) are generally small-sized (maybe 10-15 members) and are most likely part of a larger group where their primary job is to develop new product features. In this sense, Conway’s Law may or may not apply to existing and mature products. At this stage, a product’s architecture and design patterns are mature and sophisticated, and influencing them can be challenging.

Also, if every team self-organises differently, getting a consistent design in place wouldn’t be easy. We have argued in other articles on architecture that the latter is never simple in sophisticated products, and complex design patterns will surely emerge as the product evolves. However, I am unsure how self-organising teams can positively influence it.

9. Conclusion

Agile is a powerful framework for thinking and working on software development projects. Agile is now almost two decades old (although its earlier forms predate publishing the Agile Manifesto in 2001). Some of its practices (user stories, sprints, product owner roles, backlogs…) are now mainstream and successfully implemented in many shapes and forms.

Other features of Agile are relatively less clear and sometimes chaotically applied. This is most certainly true for the 11th principle, which formed the central part of this discussion.

Self-organisation is often portrayed to be a key principle of agile software development that can benefit teams and organizations. When implemented effectively (the story goes), it can improve team communication, collaboration, and motivation and increase productivity and creativity. There is some truth in these statements.

However, it is important to note that self-organisation is not a one-size-fits-all solution and may not be suitable for all teams or projects. Success requires a supportive organizational culture, clear goals and objectives, and effective communication and leadership. Ultimately, agile teams that embrace self-organisation can achieve greater autonomy and flexibility, leading to better outcomes for both the team and the organization.