Part 3: Agile Design Techniques for Creating Strong and Scalable Solutions
1. Overview
A few years back, I attended a lecture on Agile architecture by a renowned software professional and influencer. The speaker, Mr T, aimed to reconcile the conflicting ideas of fast and frequent delivery, which is at the core of Agile, with careful planning and design. However, as the lecture progressed, it became clear that the task was more challenging than Mr T had anticipated.
Throughout the talk, Mr T struggled to articulate a consistent and coherent framework where Agile design could be viable for delivering better architecture. Though there were faint insights here and there, Mr T’s ideas failed to offer paradigm-shifting solutions.
Fast forward a few years, and after consuming numerous articles, online lectures, and podcasts, I stumbled upon a logical and consistent candidate solution to the Agile design conundrum.
Mr T’s fatal mistake, I realized, was formulating Agile design as a universal approach to building more resilient software architecture. His solution was intended to supersede traditional design methods and work in any context.
However, Mr T did not consider that every methodology, method, theory, or concept must have a natural boundary, a closed domain of applicability, no matter how powerful it is.
Even in science, physical theories like General Relativity or Quantum Mechanics apply on large or tiny scales but not both (although recently, Quantum Loop Gravity and String Theory seem, at least in theory, to bridge that gap).
Agile design can work effectively if applied in the correct settings and to achieve specific objectives. The settings and the objectives will take centre stage in our discussions.
We will attempt to answer the following questions via a slightly unorthodox route.
Instead of providing direct answers in an uninteresting approach, we will take the reader along a journey of discovery. By the end of this article, the reader will have formulated a strong impression of what Agile design and architecture must look like to succeed.
2. What Does the Solution Design Series Cover?
This article belongs to a series that discusses solution design and will cover the topics below:
3. The Hidden Benefits of Agile Design
3.1 Agile Design — What Are the Ideal Situations for Success and When Can We Apply Them
In Part 1 of this series, we have enumerated the conditions under which Agile design works best. Agile design is particularly well-suited for projects with the following characteristics:
It is important to note that Agile design is not always the best fit for every project. It may not be suitable for projects with highly regulated or highly critical requirements, such as projects in the aerospace or medical device industries.
3.2 How Does Agile Design Assist Architects in Handling the Unarticulated Needs of Their Clients?
Agile design methodologies, specifically Scrum and Kanban, have effectively addressed unarticulated business needs through:
- Iterative development, and
- Customer collaboration principles.
Unarticulated business or customer needs refer to requirements or user stories that stakeholders or end-users have not explicitly identified or communicated.
These latent requirements may not be immediately discernible during the software development life cycle (SDLC) requirements gathering phase.
They may surface during the validation phase through usability testing or customer feedback sessions. Addressing these unmet needs can enhance the user experience and increase the software’s overall value proposition.
The agile design process divides the development cycle into small, manageable phases, commonly called iterations or sprints in Scrum and Work In Progress (WIP) in Kanban.
At the onset of each iteration, the development team collaborates with the customer, the product owner, to identify and prioritize the most crucial features or user stories for that specific iteration through techniques such as user story mapping.
Subsequently, the team develops, tests, and delivers software, utilizing test-driven development (TDD) and continuous integration (CI) that address these identified requirements.
This iterative approach to development also called the spiral or incremental model, allows for the rapid identification and addressing of issues as they arise.
The customer can provide feedback on the functioning software, known as the Minimum Viable Product (MVP). The team can subsequently make necessary adjustments to address emerging needs.
Furthermore, the emphasis on customer collaboration, also known as co-creation, in Agile design plays a significant role in its success.
Agile development teams typically maintain close working relationships with customers throughout the development process through techniques such as pair programming and regular meetings such as daily stand-ups and sprint planning sessions.
This close collaboration allows the team to understand better the customer’s needs and requirements, including any requirements that may not have been identified upfront, also known as emergent requirements.
4. Antifragility, Evolution, Innovation and Agile Design
4.1 Agile Design — Embracing Adaption, Innovation, and Uncertainty for Survival in a Constantly Evolving World
Agile design, evolution in living organisms, and antifragility share similarities in how they approach problem-solving and adapting to change.
Evolution is a process by which organisms adapt to changing environments through:
- Mutations: the gradual accumulation of small changes
- Exaptation: the radical repurposing of an existing behaviour or structure
The ability of an organism to adapt and evolve is critical to its survival and success.
Similarly, Agile design is based on the principle of iterative development. Small, incremental changes are made to the software through a series of iterations, allowing the development team to adapt to changing requirements or unexpected problems quickly.
Incremental updates, however, might not always be enough. Where old methods fail, repurposing existing technology for new needs (through an exaptation process) is vital, and Agile design must facilitate exaptation processes to be effective.
Antifragility is a concept introduced by Nassim Nicholas Taleb in his seminal book Antifragile — Things That Gain From Disorder. It refers to a property of complex systems that allows them to benefit from volatility and disturbances rather than being harmed by them.
Antifragile systems can adapt and become stronger in response to stressors, interpreted here as valuable information about environmental changes. Similarly, Agile design allows teams to incorporate customer feedback into future releases.
Delaying customer feedback will provide stability in the short run (until the customer starts using the software). Nevertheless, it will fail catastrophically in the long term as developers might design an unusable product. In this case, the absence of evidence (for customer dissatisfaction) must not be interpreted as evidence of absence.
In Antifragile systems, heuristics and experimentation can be employed to create a robust system that can adapt to and benefit from volatility and disturbances. A fail-fast, fail-often approach, where the system is designed to identify and recover from failures quickly, can also be employed to increase the system’s resilience to stressors.
Furthermore, a Design for Failure approach, where the system is designed to anticipate and mitigate potential failure scenarios, can increase the system’s antifragility.
The concept of embracing change and uncertainty is also reflected in DevOps practices, prioritising the integration of development and operations teams to enable faster and more efficient release cycles.
Adapting and evolving in software design and living organisms is critical to success and survival.
While an Agile design approach achieves this ability through iterative development and close customer collaboration, antifragility achieves it by embracing change and uncertainty and building redundancy and flexibility.
4.2 Hierarchical Structures and Constant Regeneration
A hierarchical system facilitates evolution and constant regeneration by allowing for modularity and flexibility in its design. The system is divided into several layers, each building upon the one below it. This structure allows individual components within each layer to be modified or replaced without affecting the overall system’s functionality.
In software design, this concept is implemented through Agile architecture. Agile architecture is a method of designing and building software systems that emphasize adaptability and flexibility. It is based on the Agile software development methodology, which emphasizes iterative and incremental development.
One of the fundamental principles of Agile architecture is a modular design, which allows for individual components of the system to be easily modified or replaced without affecting the system’s overall functionality.
Another critical principle is the use of a layered architecture, which separates concerns and evolves the system over time.
This approach allows for continuous evolution and regeneration of the software system, as new features and capabilities can be smoothly added, and old ones can be phased. This design style enables the software to adapt to changing requirements and stay relevant in a constantly evolving technological landscape.
4.3 Small Cross-Functional Teams Drive Agile Design
Distributed large teams can face several challenges when producing Agile architecture. One of the main challenges is communication and coordination. Team members may be located in different locations and time zones and have different cultural backgrounds.
These factors make having regular meetings, sharing information, and collaborating on design decisions difficult. Additionally, distributed teams may have difficulty building trust and developing a sense of shared ownership of the project.
Another challenge is the potential for silos within the team, where different sub-teams or individuals are working on various aspects of the project without proper coordination or integration. Siloing can lead to inconsistencies and gaps in the design, making achieving a cohesive, unified design challenging.
In contrast, cross-functional small teams are more conducive to creating better software architecture. Small groups are more likely to have good communication and collaboration, as team members are more likely to know each other and work closely together. Additionally, cross-functional teams are composed of individuals with different skills and perspectives, which can lead to more creative and innovative solutions.
Small cross-functional teams are also more agile and able to respond quickly to changes in requirements or design decisions, allowing them to adapt to new technologies and changing market conditions more effectively.
4.4 A Measure of Good Design
A measure of good design is how dependent higher structures are on lower ones, and this is closely related to the principle of separation of concerns and cohesion. Cohesion is a measure of how closely the elements of a module or component work together to achieve a single, well-defined purpose.
When a system is well-designed, the higher structures are not overly dependent on lower ones, and the components within the system have high cohesion. This means that changes made to the lower structures do not affect the functionality of the higher structures.
For example, a good design principle in software would be to separate the user interface from the business logic and have high cohesion within each module. This way, changes to the user interface do not affect the business logic and vice versa, allowing for more straightforward modification and maintenance of the system.
On the other hand, in a poorly designed system, the higher structures are tightly coupled with the lower ones, and the components within the system have low cohesion. The coupling of parts means that changes to the lower structures have a cascading effect on the higher structures, and the elements within the system do not have a clear and single, well-defined purpose. Coupling makes the system inflexible and difficult to modify.
However, it is essential to note that there are limitations to decoupling software components. It is crucial to consider the limitations and trade-offs involved in the process and to strike a balance between flexibility, simplicity, and performance.
Therefore, in a good design, separation of concerns, decoupling, and cohesion are vital principles leading to a more modular, flexible, and maintainable system.
4.5 How Unarticulated Needs Direct the Evolutionary Path of Agile Designs
The traditional design methods, which assume that developers fully understand the user’s requirements and that the user knows what they want before seeing the product, are often flawed in the real world.
Complex systems, such as software products and their users, constantly evolve and adapt to users’ changing needs and preferences.
As users interact with a software product, they may discover new features and capabilities they did not know existed before and rely more heavily on them. They may also repurpose some software features to solve different problems. Additionally, users may develop higher expectations and request more advanced or complex features as they become more familiar with the technology.
To address these challenges, agile design practices, such as Scrum, Kanban, and Lean software delivery methodologies, allow for the creation of a software product through the cooperation of stakeholders and developers.
Feedback loops, such as sprint retrospectives and user testing, are critical for this process. Feedback changes the software and modifies the developer’s perception of business needs and their understanding of the industry.
In this way, the feedback loops allow for a complex system and its environment to interact, modifying the dynamics of the system’s evolution.
The strength of the computer entrepreneur Steve Jobs was precisely in distrusting market research and focus groups–those based on asking people what they want–and following his own imagination. His modus was that people don’t know what they want until you provide them with it.
— Antifragile: Thing That Gain From Disorder — Nassim N. Taleb
This dynamic interaction between users, developers, and the software product can be observed today as social media and mass opinion constantly evolve and shape each other.
We can create software products tailored to users’ needs and preferences through Agile design practices rather than relying on outdated assumptions.
4.6 Time-to-market, Minimum Viable Products (MVP), and Technical Debt
Time-to-market is critical in startups and new ideas because it can significantly impact the success of the product or company. There are several reasons why:
Therefore, startups often prioritise getting to market quickly over other factors, such as building a perfect product or incurring short-term technical debt, thereby increasing their chances of success.
Time-to-market and minimum viable product (MVP) concepts highlight the importance of quickly and efficiently validating and iterating on product ideas. Agile design, a methodology that emphasises flexibility, collaboration, and rapid iteration, is well-suited for this product development category.
In Agile design, teams work in short sprints and prioritise the most important features to be developed first. Value-based prioritisation allows startups to get a Minimal Viable Product (MVP) to market quickly and start gathering feedback from customers.
Additionally, and by definition, Agile design encourages safe-to-fail experimentation early on in the product’s lifecycle, giving the designer quick feedback and, more importantly, options by preventing developers from gambling the startup’s success on a single specific design. This optionality allows startups to quickly pivot or scale if necessary, which is especially important in a competitive landscape.
In the meantime, it is OK to zigzag your way to success by trying simultaneously to keep technical debt under control while also not missing out on reaching your customer base as quickly as possible.
4.7 Avoiding Architectural Debt
Architectural debt is the cost of maintaining and updating a software system due to its design or architecture.
It can occur:
Managing architectural debt is essential to software development and maintenance, as it can significantly impact a system’s overall cost and quality.
Agile design methodologies can lead to the accumulation of architectural debt, especially if the focus is primarily on delivering new features and functionality in short sprints without sufficient attention to the long-term design and maintainability of the system.
Agile development prioritises flexibility and adaptability, which can make it more challenging to maintain a consistent and coherent architecture over time. However, it is not inherent in Agile development, and several ways exist to mitigate this risk.
Modularity, decoupling, cohesion, and a hierarchical structure are key principles that can help reduce architectural debt in Agile design by making the system more flexible and adaptable.
Together, these principles can make it easier to understand, test, and maintain the system and make changes or add new features without affecting the entire system. It also helps to reduce complexity, making the system more manageable and less error-prone.
4.8 The Untold Benefits of Procrastination
Modernity (through globalisation and connectedness) has brought about abundant information, which can be both a blessing and a curse in software design. As of 2021, the estimated size of online data is multiple zettabytes.
On the one hand, abundance allows for more diverse perspectives and better design decisions. However, it can also be overwhelming and difficult to differentiate between credible and non-credible sources (signal-to-noise in information theory) or to identify the most relevant information among the vast amount available.
Information overload makes it difficult to focus, process, and make decisions. Imagine a CTO who wishes to update his technology stack every time a new programming language is announced or an architect who responds neurotically to every article in his monthly newsletter by updating the design!
Deferring important decisions in software design can be a way to manage the abundance of information by allowing bad ideas to fade away without compromising agility. A classic Latin adage, “festina lente,” which translates to “make haste slowly”, shows that ancient Romans were aware of the problems of overreaction.
Naive interventionism is a term used by Nassim Nicholas Taleb in Antifragile to describe the tendency of people and organisations to intervene and try to fix problems without fully understanding the systems they are trying to change.
Interventionism can lead to unintended consequences, making systems more fragile rather than more robust through the generous constraining of systems that cannot be naturally constrained. All complex systems respond to overconstraining similarly by catastrophic failure.
Pros of deferring hard-to-change decisions include:
However, deferring important decisions in software design can also have adverse effects, such as:
In this context, balancing gathering information, considering options, and making decisions is crucial to Agile design. We need effective strategies for filtering, evaluating, and making sense of the vast information in the modern world.
4.9 Why Bottom-Up Design is the Key to Success
Top-down design is a system design method that defines its overall architecture first. Then, the individual components are designed and implemented to fit within that architecture.
A top-down design approach can result in a monolithic architecture (large, self-contained software programs composed of a single, unified codebase) if the system’s overall architecture is not broken down into smaller, more manageable components. A monolithic architecture is characterized by tightly coupled components and a lack of clear separation of concerns.
One of the main characteristics of monolithic software applications is that they are typically built as a single unit and are difficult to break apart or modularize. Because of this, changes to one part of the application often require changes to other parts, making it hard to update or maintain.
Additionally, a monolithic application can be hard to scale, test or deploy as a whole since the entire application needs to be deployed and started together, making it hard to add new features without affecting the whole system.
Most of the characteristics of monolithic applications are not desirable in a fast-moving, Agile-driven design approach.
However, it is possible to use a top-down design approach and still create a modular, non-monolithic architecture with well-defined interfaces if you know in advance what the optimal system design should be at any point in the future — an impossible task under conditions of uncertainty.
“[…] about half of all embryos undergo a spontaneous abortion–easier to do so than design the perfect baby by blueprint.”
— Nassim N. Taleb, Antifragile: Things That Gain From Disorder
How do we efficiently design a system without full visibility of the client’s future requirements or how their preferences will evolve? In such situations, a bottom-up design can offer many advantages.
A bottom-up design approach starts with designing individual, low-level components and gradually building them to form a larger system.
Over time, more components will be added, and without due care on how the parts will fit together, the system can become increasingly complex and difficult to maintain and can also be hard to scale, test or deploy.
Suppose the focus when designing systems in a bottom-up fashion is not solely on the functionality of the individual components but also on how they will fit together in the larger system. In that case, the target design will be optimal.
Bottom-up design can be observed in many natural systems, where individual components come together to form larger, more complex structures. Some examples include:
Nature understands that building on an imperfect but successful design is much more efficient than restarting from scratch with every new challenge.
3. Final Words
After all the articulate arguments we have provided in favour of Agile design, it is hard to avoid adopting what appears to be a powerful and flexible strategy for designing software products in every situation at every opportunity.
Therefore, it’s worthwhile reiterating the principal idea of Part 1 on the over-generalisation of methodologies and the contextualisation of projects. The message is Agile in general, and Agile design in particular, scale poorly outside areas of high uncertainty, high customer interaction, and somewhat small scales. Imagine delivering an infrastructure mega-project worth millions of dollars with that kind of flexibility and volatility; no organisation (not even with any “right” mindset) can tolerate that.
This bounded domain of applicability tells us that agility comes at a cost, an overhead acceptable in some situations but not all. The overhead of applying Agile compared to Waterfall can vary depending on the specific organisation and project. However, some of the potential overhead associated with transitioning from Waterfall to Agile include the following:
In summary, Agile is a powerful tool, and Agile design is equally good if applied under the right conditions. It allows startups or new ideas to be tested quickly and efficiently and facilitates the vital evolution of the design.