- 1. Overview
- 2. The Hidden Benefits of Agile Design
- 2. Antifragility, Evolution, Innovation and Agile Design
- 2.1 Agile Design — Embracing Adaption, Innovation, and Uncertainty for Survival in a Constantly Evolving World
- 2.2 Hierarchical Structures and Constant Regeneration
- 2.3 Small Cross-Functional Teams Drive Agile Design
- 2.4 A Measure of Good Design
- 2.5 How Unarticulated Needs Direct the Evolutionary Path of Agile Designs
- 2.6 Time-to-market, Minimum Viable Products (MVP), and Technical Debt
- 2.7 Avoiding Architectural Debt
- 2.8 The Untold Benefits of Procrastination
- 2.9 Why Bottom-Up Design is the Key to Success
- 3. Final Words
- 4. Featured Articles
A few years back, I attended a lecture by a renowned software professional and influencer on Agile architecture. The speaker, Mr T, aimed to reconcile the conflicting ideas of fast and frequent delivery, which is at the core of Agile, with careful planning and design. However, as the lecture progressed, it became clear that the task was more challenging than Mr T had anticipated.
Throughout the talk, Mr T struggled to articulate a consistent and coherent framework where Agile design could be viable for delivering better architecture. Though there were faint insights here and there, Mr T’s ideas failed to offer paradigm-shifting solutions.
Fast forward a few years, and after consuming numerous articles, online lectures, and podcasts, I stumbled upon a logical and consistent candidate solution to the Agile design conundrum.
Mr T’s fatal mistake, I realized, was formulating Agile design as a universal approach to building more resilient software architecture. His solution was intended to supersede traditional design methods and work in any context.
However, Mr T did not consider that every methodology, method, theory, or concept must have a natural boundary, a closed domain of applicability, no matter how powerful it is.
Even in science, physical theories like General Relativity or Quantum Mechanics apply on large or tiny scales but not both (although recently, Quantum Loop Gravity and String Theory seem, at least in theory, to bridge that gap).
Agile design can work effectively if applied in the correct settings and to achieve specific objectives. The settings and the objectives will take centre stage in our discussions.
We will attempt to answer the following questions via a slightly unorthodox route.
- What is Agile design?
- What problem is it destined to solve?
- What does Agile design or Agile architecture look like?
- What are its limitation and domain of applicability?
Instead of providing direct answers in an uninteresting approach, we will take the reader along a journey of discovery. By the end of this article, the reader will have formulated a strong impression of what Agile design and architecture must look like if they were to succeed.
2. The Hidden Benefits of Agile Design
2.1 Agile Design — What Are the Ideal Situations for Success and When Can We Apply Them
In Part 1 of this series, we have enumerated the conditions under which Agile design works best. Agile design is particularly well-suited for projects with the following characteristics:
- Complex and rapidly changing requirements — Agile design is well-suited for projects where the requirements are complex, rapidly changing, or not fully understood at the beginning of the project. The iterative development process of Agile allows for requirements to be refined and added as the project progresses.
- A high degree of uncertainty — Agile design is well-suited for projects with high uncertainty, such as research and development projects or projects in emerging markets. The iterative development process allows flexibility and the ability to adapt to changing circumstances.
- Need for customer involvement — Agile design is well-suited for projects where customer involvement is essential, as it emphasises customer collaboration throughout the development process.
- Small and medium-sized projects — Agile design is well-suited for small and medium-sized projects, as it is less formal and bureaucratic than traditional project management methodologies.
It is important to note that Agile design is not always the best fit for every project. It may not be suitable for projects with highly regulated or highly critical requirements, such as projects in the aerospace or medical device industries.
2.2 How Does Agile Design Assist Architects in Handling the Unarticulated Needs of Their Clients?
Agile design methodologies, specifically Scrum and Kanban, have effectively addressed unarticulated business needs through:
- Iterative development, and
- Customer collaboration principles.
They may surface during the validation phase through usability testing or customer feedback sessions. Addressing these unmet needs can enhance the user experience and increase the software’s overall value proposition.
In the agile design process, the development cycle is divided into small, manageable phases, commonly referred to as iterations or sprints in Scrum and Work In Progress (WIP) in Kanban.
At the onset of each iteration, the development team engages in a collaborative effort with the customer, also known as the product owner, to identify and prioritize the most crucial features or user stories for that specific iteration through techniques such as user story mapping.
This iterative approach to development, also referred to as the spiral model or incremental model, allows for the rapid identification and addressing of issues as they arise.
The customer can provide feedback on the functioning software, known as the Minimum Viable Product (MVP). The team can subsequently make necessary adjustments to address emerging needs.
Furthermore, the emphasis on customer collaboration, also known as co-creation, in Agile design plays a significant role in its success.
Agile development teams typically maintain close working relationships with customers throughout the development process through techniques such as pair programming and regular meetings such as daily stand-ups and sprint planning sessions.
This close collaboration allows the team to understand better the customer’s needs and requirements, including any requirements that may not have been identified upfront, also known as emergent requirements.
2. Antifragility, Evolution, Innovation and Agile Design
2.1 Agile Design — Embracing Adaption, Innovation, and Uncertainty for Survival in a Constantly Evolving World
Agile design, evolution in living organisms, and antifragility share similarities in how they approach problem-solving and adapting to change.
Evolution is a process by which organisms adapt to changing environments through:
- Mutations: the gradual accumulation of small changes
- Exaptation: the radical repurposing of an existing behaviour or structure
The ability of an organism to adapt and evolve is critical to its survival and success.
Similarly, Agile design is based on the principle of iterative development, where small, incremental changes are made to the software through a series of iterations, allowing the development team to adapt to changing requirements or unexpected problems quickly.
Incremental updates, however, might not always be enough. Where old methods fail, repurposing existing technology for new needs (through an exaptation process) is vital, and Agile design must facilitate exaptation processes to be effective.
Antifragility is a concept introduced by Nassim Nicholas Taleb in his seminal book Antifragile — Things That Gain From Disorder. It refers to a property of complex systems that allows them to benefit from volatility and disturbances rather than being harmed by them.
Antifragile systems can adapt and become stronger in response to stressors, interpreted here as valuable information about environmental changes. Similarly, Agile design allows teams to incorporate customer feedback into future releases.
Delaying customer feedback will provide stability in the short run (until the customer starts using the software). Nevertheless, it will fail catastrophically in the long term as developers might design an unusable product. In this case, the absence of evidence (for customer dissatisfaction) must not be interpreted as evidence of absence.
In Antifragile systems, heuristics and experimentation can be employed to create a robust system that can adapt to and benefit from volatility and disturbances. A fail-fast, fail-often approach, where the system is designed to identify and recover from failures quickly, can also be employed to increase the system’s resilience to stressors.
Furthermore, a Design for Failure approach, where the system is designed to anticipate and mitigate potential failure scenarios, can increase the system’s antifragility.
The concept of embracing change and uncertainty is also reflected in the use of DevOps practices, prioritising the integration of development and operations teams to enable faster and more efficient release cycles.
Adapting and evolving in software design and living organisms is critical to success and survival.
While an Agile design approach achieves this ability through iterative development and close customer collaboration, antifragility achieves it by embracing change and uncertainty and building redundancy and flexibility.
2.2 Hierarchical Structures and Constant Regeneration
A hierarchical system facilitates evolution and constant regeneration by allowing for modularity and flexibility in its design. The system is divided into several layers, each building upon the one below it. This structure allows individual components within each layer to be modified or replaced without affecting the overall system’s functionality.
In software design, this concept is implemented through Agile architecture. Agile architecture is a method of designing and building software systems that emphasize adaptability and flexibility. It is based on the Agile software development methodology, which emphasizes iterative and incremental development.
One of the fundamental principles of Agile architecture is a modular design, which allows for individual components of the system to be easily modified or replaced without affecting the system’s overall functionality.
Another critical principle is the use of a layered architecture, which allows for the separation of concerns and the ability to evolve the system over time.
This approach allows for continuous evolution and regeneration of the software system, as new features and capabilities can be smoothly added, and old ones can be phased. This design style enables the software to adapt to changing requirements and stay relevant in a constantly evolving technological landscape.
2.3 Small Cross-Functional Teams Drive Agile Design
Distributed large teams can face several challenges when producing Agile architecture. One of the main challenges is communication and coordination. Team members may be located in different locations and time zones and may also have different cultural backgrounds.
These factors make it difficult to have regular meetings, share information, and collaborate on design decisions. Additionally, distributed teams may have difficulty building trust and developing a sense of shared ownership of the project.
Another challenge is the potential for silos to form within the team, where different sub-teams or individuals are working on various aspects of the project without proper coordination or integration. Siloing can lead to inconsistencies and gaps in the design, making it challenging to achieve a cohesive, unified design.
In contrast, cross-functional small teams are more conducive to creating better software architecture. Small groups are more likely to have good communication and collaboration, as team members are more likely to know each other and work closely together. Additionally, cross-functional teams are composed of individuals with different skills and perspectives, which can lead to more creative and innovative solutions.
Small cross-functional teams are also more agile and able to respond quickly to changes in requirements or design decisions, allowing them to adapt to new technologies and changing market conditions more effectively.
2.4 A Measure of Good Design
A measure of good design is how dependent higher structures are on lower ones, and this is closely related to the principle of separation of concerns and cohesion. Cohesion is a measure of how closely the elements of a module or component work together to achieve a single, well-defined purpose.
When a system is well-designed, the higher structures are not overly dependent on lower ones, and the components within the system have high cohesion. This means that changes made to the lower structures do not affect the functionality of the higher structures.
For example, a good design principle in software would be to separate the user interface from the business logic and have high cohesion within each module. This way, changes to the user interface do not affect the business logic and vice versa, allowing for more straightforward modification and maintenance of the system.
On the other hand, in a poorly designed system, the higher structures are tightly coupled with the lower ones, and the components within the system have low cohesion. The coupling of parts means that changes to the lower structures have a cascading effect on the higher structures, and the elements within the system do not have a clear and single well-defined purpose. Coupling makes the system inflexible and difficult to modify.
However, it is essential to note that there are limitations to decoupling software components. It is crucial to consider the limitations and trade-offs involved in the process and to strike a balance between flexibility, simplicity, and performance.
Therefore, in a good design, separation of concerns, decoupling, and cohesion are vital principles leading to a more modular, flexible, and maintainable system.
2.5 How Unarticulated Needs Direct the Evolutionary Path of Agile Designs
The traditional design methods, which assume that developers fully understand the user’s requirements and that the user knows what they want before seeing the product, are often flawed in the real world.
Complex systems, such as software products and their users, constantly evolve and adapt to users’ changing needs and preferences.
As users interact with a software product, they may discover new features and capabilities that they did not know existed before and begin to rely on them more heavily. They may also repurpose some software features to solve different problems. Additionally, users may develop higher expectations and request more advanced or complex features as they become more familiar with the technology.
To address these challenges, agile design practices, such as Scrum, Kanban, and Lean software delivery methodologies, allow for the co-creation of a software product through the cooperation of stakeholders and developers.
Feedback loops, such as sprint retrospectives and user testing, are critical for this process. Feedback not only changes the software but also modifies the developer’s perception of business needs and their understanding of the industry.
The feedback loops, in this way, allow for a complex system and its environment to interact, modifying the dynamics of the system’s evolution.
The strength of the computer entrepreneur Steve Jobs was precisely in distrusting market research and focus groups–those based on asking people what they want–and following his own imagination. His modus was that people don’t know what they want until you provide them with it.
This dynamic interaction between users, developers, and the software product can be observed in today’s world, as social media and mass opinion constantly evolve and shape each other.
We can create software products tailored to users’ needs and preferences through Agile design practices rather than relying on outdated assumptions.
2.6 Time-to-market, Minimum Viable Products (MVP), and Technical Debt
Time-to-market is critical in startups and new ideas because it can significantly impact the success of the product or company. There are several reasons why:
- First-mover advantage — Being the first to market with a new product or idea can give a startup a significant advantage over competitors. It allows the company to establish a foothold in the market and gain mindshare before others.
- Limited resources — Startups typically have limited resources, including time and money. By getting to market quickly, a startup can start generating revenue and gathering customer feedback earlier, which can help the company make better decisions about future development.
- Competitive landscape — The startup landscape is highly competitive, and competitors can quickly copy new ideas. By getting to the market soon, a startup can establish itself and build a loyal customer base before others enter the market.
- Validation — The faster a startup can validate its product idea and business model, the more quickly it can pivot or scale if necessary.
Therefore, startups often prioritise getting to market quickly over other factors, such as building a perfect product or incurring short-term technical debt, thereby increasing their chances of success.
Time-to-market and minimum viable product (MVP) concepts highlight the importance of quickly and efficiently validating and iterating on product ideas. Agile design, a methodology that emphasises flexibility, collaboration, and rapid iteration, is well-suited for this product development category.
In Agile design, teams work in short sprints and prioritise the most important features to be developed first. Value-based prioritisation allows startups to get a Minimal Viable Product (MVP) to market quickly and start gathering feedback from customers.
Additionally, and by definition, Agile design encourages fail-to-safe experimentation early on in the product’s lifecycle, giving the designer quick feedback and, more importantly, options by preventing developers from gambling the startup’s success on a single specific design. This optionality allows startups to quickly pivot or scale if necessary, which is especially important in a competitive landscape.
In the meantime, it is OK to zigzag your way to success by trying simultaneously to keep technical debt under control while also not missing out on reaching your customer base as quickly as possible.
2.7 Avoiding Architectural Debt
Architectural debt refers to the cost of maintaining and updating a software system due to its design or architecture.
It can occur:
- When a system is initially designed and implemented in a rush or with little foresight, leading to problems that become more difficult and expensive to fix as the system grows and evolves.
- When a system is continually modified to meet changing requirements without considering the long-term impact on the overall design.
Managing architectural debt is essential to software development and maintenance, as it can significantly impact a system’s overall cost and quality.
Agile design methodologies can lead to the accumulation of architectural debt, especially if the focus is primarily on delivering new features and functionality in short sprints without sufficient attention to the long-term design and maintainability of the system.
Agile development prioritises flexibility and adaptability, which can make it more challenging to maintain a consistent and coherent architecture over time. However, it is not inherent in Agile development, and there are several ways to mitigate this risk.
Modularity, decoupling, cohesion, and a hierarchical structure are key principles that can help to reduce architectural debt in Agile design by making the system more flexible and adaptable over time.
- Modularity refers to breaking a system into smaller, independent units or modules with well-defined interfaces.
- Decoupling refers to minimising dependencies between different parts of the system.
- Cohesion refers to the degree to which the elements within a module or component work together to perform a single, well-defined function.
- A hierarchical structure allows for a clear separation of concerns and a logical system organisation.
Together, these principles can make it easier to understand, test, and maintain the system and make changes or add new features without affecting the entire system. It also helps to reduce complexity, making the system more manageable and less error-prone.
2.8 The Untold Benefits of Procrastination
Modernity (through globalisation and connectedness) has brought about an abundance of information, which can be both a blessing and a curse in software design. As of 2021, the estimated size of online data is multiple zettabytes.
On the one hand, abundance allows for more diverse perspectives and better design decisions. However, it can also be overwhelming and difficult to differentiate between credible and non-credible sources (signal-to-noise in information theory) or to identify the most relevant information among the vast amount available.
Information overload makes it difficult to focus, process, and make decisions. Imagine a CTO who wishes to update his technology stack every time a new programming language is announced or an architect who responds neurotically to every article in his monthly newsletter by updating the design!
Deferring important decisions in software design can be a way to manage the abundance of information by allowing bad ideas to fade away without compromising agility. A classic Latin adage, “festina lente,” which translates to “make haste slowly”, shows that ancient Romans were aware of the problems of overreaction.
Naive interventionism is a term used by Nassim Nicholas Taleb in Antifragile to describe the tendency of people and organisations to intervene and try to fix problems without fully understanding the systems they are trying to change.
Interventionism can lead to unintended consequences, making systems more fragile rather than more robust through the generous constraining of systems that cannot be naturally constrained. All complex systems respond to overconstraining similarly by catastrophic failure.
Pros of deferring hard-to-change decisions include:
- Allowing for more time to gather information and consider options
- Avoiding committing to a design that could become obsolete or infeasible later on
- Potentially leading to better design decisions as more information becomes available
- Enabling the ability to adapt to changes or new requirements that arise during the development process
However, deferring important decisions in software design can also have adverse effects, such as:
- Increased complexity and technical debt as design decisions are deferred
- Difficulty in making progress on the project as critical decisions are postponed
- Difficulty in maintaining a clear vision for the project
- Difficulty in making decisions quickly when they are needed
- Difficulty keeping the team on the same page, as different members may have other ideas about what the final design should look like.
In this context, finding a balance between gathering information, considering options, and making decisions is crucial to Agile design. We need effective strategies for filtering, evaluating, and making sense of the vast information available in the modern world.
2.9 Why Bottom-Up Design is the Key to Success
Top-down design is a system design method in which its overall architecture is defined first. Then the individual components are designed and implemented to fit within that architecture.
A top-down design approach can result in a monolithic architecture (large, self-contained software programs composed of a single, unified codebase) if the system’s overall architecture is not broken down into smaller, more manageable components. A monolithic architecture is characterized by tightly coupled components and a lack of clear separation of concerns.
One of the main characteristics of monolithic software applications is that they are typically built as a single unit and are difficult to break apart or modularize. Because of this, changes to one part of the application often require changes to other parts, making it hard to update or maintain.
Additionally, a monolithic application can be hard to scale, test or deploy as a whole since the entire application needs to be deployed and started together, making it hard to add new features without affecting the whole system.
Most of the characteristics of monolithic applications are not desirable in a fast-moving, Agile-driven design approach.
However, it is possible to use a top-down design approach and still create a modular, non-monolithic architecture with well-defined interfaces if you know in advance what the optimal system design should be at any point in the future — an impossible task under conditions of uncertainty.
[…] about half of all embryos undergo a spontaneous abortion–easier to do so than design the perfect baby by blueprint.
How do we efficiently design a system without full visibility of the client’s future requirements or how their preferences will evolve? In such situations, a bottom-up design can offer many advantages.
A bottom-up design approach starts with designing individual, low-level components and gradually building them up to form a larger system.
Over time, more components will be added, and without due care on how the parts will fit together, the system can become increasingly complex and difficult to maintain and can also be hard to scale, test or deploy.
Suppose the focus when designing systems in a bottom-up fashion is not solely on the functionality of the individual components but also on how they will fit together in the larger system. In that case, the target design will be optimal.
Bottom-up design can be observed in many natural systems, where individual components come together to form larger, more complex structures. Some examples include:
- Biological organisms — Living organisms are composed of cells, which are the basic building blocks of life. Cells come together to form tissues, which come together to form organs until the organism as a whole is formed.
- Ecosystems are formed by interactions between individual organisms and their environment. These interactions can be considered a bottom-up process, where individual organisms come together to form populations, which in turn come together to form communities until the ecosystem as a whole is formed.
- Social systems — Ant colonies, beehives and even human societies are examples of bottom-up design. Individuals come together to form groups and communities to achieve common goals.
Nature understands that building on an imperfect but successful design is much more efficient than restarting from scratch with every new challenge.
3. Final Words
After all the articulate arguments we have provided in favour of Agile design, it is hard to avoid adopting what appears to be a powerful and flexible strategy for designing software products in every situation at every opportunity.
Therefore, it’s worthwhile reiterating the principal idea of Part 1 on the over-generalisation of methodologies and the contextualisation of projects. The message is the following: Agile in general, and Agile design in particular, scale poorly outside areas of high uncertainty, high customer interaction, and somewhat small scales. Imagine delivering an infrastructure mega-project worth millions of dollars with that kind of flexibility and volatility; no organisation (not even with any “right” mindset) can tolerate that.
This bounded domain of applicability tells us that agility comes at a cost, an overhead acceptable in some situations but not all. The overhead of applying Agile compared to Waterfall can vary depending on the specific organisation and project. However, some of the potential overhead associated with transitioning from Waterfall to Agile include the following:
- Increased communication and collaboration — Agile development methodologies emphasise communication and collaboration between team members and stakeholders, which can require additional time and resources.
- More frequent meetings — Agile development methodologies often involve more frequent discussions, such as daily stand-ups, sprint planning, and retrospectives, which can be time-consuming and require additional resources.
- Greater flexibility and adaptability — Agile development methodologies require greater flexibility and adaptability from team members and stakeholders, which can be difficult for some individuals and require additional resources to manage.
- Change management — Agile development requires a significant cultural shift, which can be difficult and costly. Teams may need to be re-trained and re-organised, and processes may need to be redesigned.
- Team structure and roles — Agile development requires a specific team structure and roles that may differ from traditional development models. This can require re-organising teams and redistributing responsibilities, which can be disruptive and require additional resources.
In summary, Agile is a powerful tool, and Agile design is equally good if applied under the right conditions. It allows startups or new ideas to be tested quickly and efficiently and facilitates the vital evolution of the design.