1. The Agile Approach: A New Way of Thinking about Project Development
In an introduction to a talk on Clean Agile, Robert Martin (Uncle Bob), one of the 17 people who wrote the Agile Manifesto, describes the Agile approach as:
A small idea about the small problem of small programming teams doing small things.
Dave Snowden, a leading expert on complexity in organisational management, defines Agile as a method that can be applied in complex situations where planning, prediction, and causal links between cause and effect break down.
Still a hot topic?
Agile Today
Agile is not new; it’s at least 20+ years old.
So why is the Agile methodology discussion still a hot topic, why is it not dominant as the best project development methodology, and why is there a need to write another article about it?
The reasons for writing another article on Agile methods are threefold.
Agile is not easy: Perhaps the most common issue with Agile is that it is not asintuitive and self-explanatory as we often think. Some mental effort is required to appreciate its strengths, weaknesses, advantages, and challenges. This will be the first objective of the present article.
There is a need to return to the first principles: Agile and its various implementations (Scrum, Kanban, Extreme Programming (XP), Lean) have benefited from much refinement and enhancement while accumulating (sometimes unnecessary rituals), poor interpretations, and poor practices over the last two decades. Today, the term Agile has almost lost its meaning. Understanding the primary principles of Agile and the context in which they came about is the second objective of this article.
Waterfall vs Agile vs DevOps: The final objective of this article is to understand Agile’s place vis-a-vis other project development methodologies and the context in which it best applies.
In all our discussions, the central theme is always the same: How can a software organisation achieve Operational Excellence in its various departments? In this article, we will focus on software project development, specifically the Agile approach, as implementing Agile practices is one of the six principles of operational excellence that we advocate.
The Agile System Development methodologies (XP, Scrum, Kanban, etc.) were independently created in the 1990s by separate groups. The declaration of the Agile Manifesto in 2001 was only the last step in a series of pre-histories culminating in the Utah meeting.
To fully appreciate its nature and methods, we must examine its history and the context in which it was conceived. This inevitably leads us back to Waterfall, Winston Royce’s paper on managing large software development projects, and the state of software development and its industry in the 1990s.
In the 1990s, professionals believed in what Fowler calls plan-driven processes.
This understanding of software development and delivery was predicated on three principles:
First, projects can be planned to (almost) arbitrary precision. The end product can be designed, planned, and developed with high precision regarding quality, schedule and cost. The element of uncertainty was not accounted for.
Second, the result is guaranteed as long as the process is followed to the letter. There was little room for adapting to changing circumstances.
Third, the project’s scale didn’t matter. Large projects could be managed as efficiently and effectively as small ones.
These three principles were firmly established in the 1970s. They can be neatly observed in Royce’s paper, where he elaborated his views on how large software projects can be managed and what potential benefits the existing Waterfall methodology (the term “Waterfall” was not in use then) could accumulate if some minor changes like prototyping and iterations were incorporated into the Waterfall techniques.
Royce did not see that Waterfall was doomed; on the contrary, it seemed like the best project development method available at the time. However, prototyping and iterations were required to address potential design flaws before they became too costly. The passage below shows precisely what Royce was thinking at the time.
In this case, a very special kind of broad competence is required by the personnel involved. They must have an intuitive feel for analysis, coding, and program design. They must quickly sense the trouble spots in the design, model them, model their alternatives, forget the straightforward aspects of the design which aren’t worth studying at this early point, and finally arrive at an error-free program.
— Dr Winston Royce, Managing the Development of Large Software Systems
While these are brilliant improvements for small teams. I believe that dynamic, iterative, and evolving software development techniques become hard to implement in large teams. Problems of coordination and governance quickly arise, and the process becomes unmanageable. In the 1990s, teams were getting bigger, and software was getting more complex. The need to plan, predict, and commit was insurmountable.
4. Agile Development Methods: A Very Brief History
The Agile movement was started in 2001 when a group of software developers gathered in Utah to discuss new ways to approach software development. This meeting, known as the Agile Manifesto, created a set of principles and values to shape software development.
The historical events that led to the creation of Agile can be traced back to the 1990s when the waterfall methodology characterized software development. This approach involved a linear planning, design, development, testing, and deployment process. The problem with this approach was that it often resulted in lengthy development cycles, cost overruns, and failed projects. This led to a search for new and better ways to approach software development.
The main contributors to Agile are a group of 17 software developers who met in Utah in 2001. This group, which included Kent Beck, Martin Fowler, and Jeff Sutherland, came together to discuss new ways to approach software development. They created the Agile Manifesto, which outlined principles and values guiding Agile development.
When Agile was announced, it was formulated as a manifesto, which outlined four values and 12 principles that would guide Agile development. The four values were individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation, and responding to change over following a plan. The 12 principles included delivering working software frequently, working closely with customers, and promoting sustainable development.
Initially
Agile was received with scepticism by many in the software development community.
Some saw it as a fad or a buzzword, while others were concerned that it lacked the rigour and structure of traditional methodologies.
However, Agile has gained widespread acceptance over time as a highly effective approach to software development.
Today
Agile is widely used by organizations of all sizes and has been credited with improving software quality, reducing costs, and speeding up time to market.
While some still prefer traditional methodologies, Agile has become the de facto standard for software development in many industries.
Today, Agile system development consists of recognizable features refined over the last 2-3 decades. It is important to recognize that applying these methods and techniques does not make you Agile. For example, one-year sprints, extensive and archaic documentation techniques, and rigid design-focused plan-driven processes are more Waterfall than Agile.
A team or organisation is Agile if it can handle evolving requirements and deliver high-quality features faster into production through smaller and more frequent release cycles.
The Agile methodology continues to be widely adopted and practised in the software development industry. Here are some key aspects regarding the state of Agile in software development:
The Ubiquity of Agile Practices:
Development teams worldwide commonly employ agile practices such as Scrum, Kanban, and Extreme Programming (XP).
In cases where these specific methodologies are not used, a hybrid one is employed instead. Hybrid models combine advantages from both worlds and make Agile available to teams that may not be ready to become fully Agile.
Scaling Agile:
The need to scale Agile to larger projects or organizations has led to the development of frameworks like SAFe (Scaled Agile Framework), LeSS (Large Scale Scrum), and Nexus.
These frameworks guide applying Agile principles effectively in larger and more complex environments. Whether or not they have worked in practice is an open question.
The integration of Agile with DevOps practices has gained prominence. This integration aims to streamline the entire software development lifecycle, from code development to deployment and operations.
Many organisations have a stronger awareness of the need for continuous improvement.
This awareness is partly due to Agile practices such as retrospectives and test automation, where the standards for quality deliveries have steadily increased over the years.
Challenges and Criticisms:
While Agile has proven successful in many scenarios, challenges persist, such as the difficulty of cultural transformations, resistance to change, and the potential for misinterpretation or misapplication of Agile principles.
There are also inherent challenges within Agile itself, such as its inability to scale to larger teams or move across different boundaries. To paraphrase Dave Snowden, imagine an organisation making strategic plans on two-week sprints!
6. What Are the Four Basic Principles Underlying the Agile Methodology?
6.1 The Agile Manifesto
Of the twelve principles in the Agile Manifesto, I believe in the primacy of the four below as they beautifully summarize the Agile approach and its most radical implications.
Principle 1
Individuals and interactions over processes and tools
Principle 2
Working software over comprehensive documentation
Principle 3
Customer collaboration over contract negotiation
Principle 4
Responding to change over following a plan
The four principles above revolve around the following ideas:
Traditional software development methodologies portray processes as rigid frameworks that respond poorly to change. On the other hand, personal interaction and collaboration with peers allow challenges posed by changing requirements to be overcome. In such a context, change is welcome, and ambiguity is not as challenging to work with.
Working software that fits the purpose is more valuable than following specifications that were outlined when the business requirements were not mature enough.
Customer collaboration over contract negotiation places customer feedback and involvement above signed-off requirements and plans in the implementation journey.
Responding to change over following a plan means the software development team and their development practices should be flexible and responsive to changing requirements, even if they occur late in the project’s development.
Let’s see why these principles allow project development to run more efficiently and how they help shape a better product that will fill the client’s business needs.
6.2 Smaller, More Frequent Releases Are Better
Imagine you’re working on a project where you need to build ten new features, which you estimate will take 12 months to complete. You have two options: the traditional Waterfall approach or the more modern Agile approach.
Feature delivery in Agile and Waterfall. Notice how early new features are released into production in Agile. Also, note that Agile might be a bit slower in delivering the entire set of new features, but overall, it has produced more customer value over time.
The figure above shows the features delivered for the same project, one following the Agile approach and the other using Waterfall. We can make the following observations:
The Waterfall approach:
With Waterfall, no features are deployed on production during the implementation phase. This is represented by the four short orange bars showing that little or no business value is generated in that period.
Assuming everything goes as planned, you’ll have all ten features released in 12 months.
But what if there’s a delay in the project or a requirement change? Waterfall’s rigid, linear approach makes it difficult to accommodate changes mid-project.
Also, the entire set of features will be delayed in case of delays, not just the feature in question. This is due to the batch-and-queue sequential nature of the Waterfall process.
The Agile approach:
In contrast, with Agile, you can release each new feature as soon as it’s completed.
So, assuming each feature takes an average of one month to complete, you’ll have one new feature in production every month, two features in two months, and so on.
Agile methods allow you to deliver business value continuously rather than waiting for the end of the project.
In case of delays, the impact will be restricted to the features in the backlog rather than holding off the entire project.
A fair comparison between the two approaches would be to count the number of features in production at any given time and multiply it by how long they have been released. By doing so, we can see that Agile outperforms Waterfall regarding business value delivered over time.
Of course, there are caveats to the Agile approach.
Increased overhead:
First, as seen in the above graph, the project sign-off in the Agile case lags behind the Waterfall one. This is because breaking big releases into smaller ones will introduce management overhead. However, the benefits of early feature releases outweigh this overhead.
Feature breakdown:
The second caveat is that not all features can be isolated and independently released. Many features have interdependencies and require coordination to deliver. In real-world scenarios, this can make the picture less clear-cut.
Despite these limitations, it’s crucial to inject agility into any project. By embracing Agile, you can be more responsive to changes and deliver business value continuously, rather than waiting until the end of a project. Agile brings you closer to Operational Excellence in most cases.
6.3 Iterations over Extensive Planning and Design
In software development, unclear or incomplete business requirements can be a major hurdle as they often delay the project due to redesigns and reworks. Traditional Waterfall methodologies attempt to overcome this issue by requiring stakeholders to sign off on every project artifact before proceeding to the next stage. However, this approach often leads to finger-pointing and a final product with poor customer value.
Studies have shown that Agile methodologies can lead to higher customer satisfaction, improved project performance, and increased team collaboration and productivity.
— “The Impact of Agile Methodologies on Software Development Teams” by K. Petersen et al.
Agile, on the other hand, proposes a different approach. Instead of a fixed set of requirements, Agile product owners maintain a prioritized backlog of user stories. The stories are typically small enough so that their business value can be relatively easily gauged. Short user stories also benefit from being less prone to change. Furthermore, the backlog is constantly updated and re-ordered based on customer value and uncertainty around each user story.
Focusing on customer value
User stories with high customer value and low uncertainty are prioritized over low customer value and high uncertainty items.
Top priority stories are, therefore, delivered first, while stories with more uncertainty are updated as more information becomes available.
Adaptability in the face of change
This Agile approach allows flexibility and adaptability when responding to changing requirements, ensuring that the final product delivers real value to customers.
This flexibility comes at a price which outweighs the cost of delivering an unusable product.
Real-world examples of successful Agile implementations are numerous and include Spotify, which uses Agile to iterate and continuously improve its music streaming service, and Salesforce, which uses Agile to rapidly develop and deploy new features and updates to its customer relationship management software.
6.4 Emphasizing the Human Element in Software Development
In our experience, the users are very brilliant people, especially if they are your customers and depend on you for their livelihood. We find that every design phase we go through is based strictly on the users’ reactions to the previous system. The users are the people who do our design once we get started.
Four of the twelve principles in the Agile Manifesto emphasise the human element in software development. These principles focus on collaboration, motivation, communication, and self-organisation.
The first principle highlights the importance of customer and development team collaboration.
The principle states that “business people and developers must work together daily throughout the project.” This principle recognises that involving the customer in the development process is crucial to the project’s success. It sets a high frequency for interactions to underscore the importance of involving the customer throughout the project.
This focus on customer involvement is a significant departure from traditional Waterfall-based approaches, where the customer may not see the product until user acceptance testing (UAT), which may be too late to make any significant changes.
The second principle emphasises the importance of motivation and empowerment in the development team.
The principle states that “build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.“
This principle recognises that motivated and empowered employees are significant influencers in the team’s performance and should be part of any Agile implementation.
A study published in the Journal of Management found that employees’ intrinsic motivation positively relates to job satisfaction and performance.
The third principle stresses the importance of face-to-face communication.
The principle states, “face-to-face conversation is the most efficient and effective method of conveying information to and within a development team.“
This principle recognises that face-to-face discussions are preferred over documentation as they add one or two dimensions to the conversation channels.
Facial cues help communicate the interlocutor’s feelings about a specific idea, while kindness, empathy, and good overall communication skills increase the effectiveness and efficiency of a debate.
A study published in the Journal of Business and Psychology found that employees who communicate effectively with their colleagues are more likely to experience job satisfaction.
The fourth principle recognises the importance of self-organisation in teams.
The principle states that “the best architectures, requirements, and designs emerge from self-organising teams.“
This principle recognises that self-organisation is a property of complex systems, and human groups are profoundly complex.
Self-organisation manifests, for example, through the specialisation of team members in specific product areas or by honing specific skills. The relationship, however, that the authors envisaged between this complexity and software architecture is not evident.
These principles recognise that involving the customer in the development process, empowering employees, using face-to-face communication, and enabling self-organisation in teams are crucial to the success of agile software development.
7. Agile Project Development Terminology
7.1 The Rich Terminology of the Agile Vocabulary
Part of understanding Agile software development requires understanding the rich vocabulary of phrases and terms associated with this method.
Many professionals use different terms and phrases when discussing Agile system development. At first, it can be confusing because some words make sense while others don’t. There are two reasons for that.
Context dependency: Firstly, Agile can mean slightly different things depending on the context (software development team, project delivery, organizational behaviour, etc.).
A problem of semantics: Secondly, the definition of Agile has been extensively expanded (and diluted) over the last two decades. Today, it can almost refer to anything.
We will attempt to address this problem in this section by providing rigorous definitions for the following terms:
As you delve further into the various terminologies, it becomes apparent that there is often a subtle overlap, making the distinctions between them quite nuanced. This is a natural occurrence and should be anticipated. Additionally, you may observe a degree of arbitrariness in the presented definitions, which is also valid.
The four pillars of the Agile approach or philosophy explain the nature of software development.
The Agile approach or philosophy assumes the following about the nature of reality in software development projects:
The iterative model:
The iterative nature of Agile is well-suited to software development projects where the business requirements are constantly evolving and where customer interactions are high.
Iterations are expensive but unavoidable. Each iteration benefits from information collected from the previous one, bringing the product closer to the desired end state.
Users constantly change their preferences as they interact more with the software and understand more what the technology can do for them.
This inherent ambiguity in business requirements is to be expected and accommodated into the Software Development Lifecycle through various techniques involving continuous user feedback.
Software solutions constantly evolve:
Software solutions cannot be fully engineered from the start, and the final product details will depend on the specific implementation path taken.
A perfect design, fit for purpose, can never be created during the early stages of the project and can only be reached iteratively.
Agile team structure, hierarchies, and communication:
Power and status in the team belong to those who bring the most value to the team, regardless of seniority or position.
Hierarchies are cumbersome, and self-organisation is preferred.
Face-to-face communication is superior to written documentation. Communication should focus more on the content than the form.
7.3 What Is the Agile Framework?
The six pillars of the Agile framework.
The Agile framework is the scaffolding on which the Agile methods, practices, and culture are based. This scaffolding consists of the following pillars:
Divergence from the manufacturing model:
Software applications (and their ecosystem, which includes users) are like complex adaptive systems, where principles from complexity theory apply better than those from the Toyota Production System (TPS).
The world of Agile is complex and not just complicated. The emergence of new ideas, technologies, or client preferences is welcomed and easily accommodated.
Documentation is essential,but emphasis must be on working software:
Technical and business documents are less vital to the success of a project. More emphasis is placed on shipping working software that solves business needs.
There is no equivalent to the Waterfall process, where documents must be signed off before moving to the next stage. It is more common to use ticketing systems like JIRA, where statuses can be updated based on progress.
Detailed planning is replaced with a dynamic backlog and story point estimations:
Careful and thorough planning hinders project governance and effective resource and budget allocation.
Agile replaces project plans with constantly updated backlogs to reflect the latest state of knowledge regarding what new feature brings the most value to the business.
Processes, roles, and responsibilities exist to govern the backlog maintenance activities.
Engineering and solution design are not enough; user feedback is vital:
Engineering and solution design requires a detailed understanding of the requirements and a mastery of technology. In Agile projects, this assumption does not hold in the real world and must be replaced with a more adaptive approach.
The evolution of customer preferences with the product and the technology is anticipated and welcomed. Agile methods must include user feedback in their processes.
Cross-functional teams where members bring skills from many domains are more desirable than the old system’s rigid structure, where people were grouped based on their roles.
Work allocation and responsibility distribution are governed by the team’s specific context rather than set by organisational memos.
Cross-functional teams provide less redundancy than role-based ones but allow for faster decision-making.
Value-adding vs supporting tasks:
Like in Waterfall, the core activities where business value is created are Analysis and Development. The rest is necessary because executing only those two stages could never achieve satisfactory outcomes.
Aside from Analysis and Development, all non-value-adding tasks must be minimized in cost and schedule to retain a competitive advantage. This is achieved by deploying Agile practices such as Test-Driven Development, Continuous Integration, and heavy test automation and deployment.
Let’s briefly consider the dynamics of a new software development team. In the initial stages, the team embarks on a journey of exploration and adaptation. Under the guidance of their manager, they embrace various methodologies and approaches, gradually shaping their collective perspective. As this formative period draws to a close, the team has assimilated many experiences, transforming challenges into valuable lessons. These insights become ingrained in their ethos, instigating a sense of identity and belonging. Consequently, any scrutiny or challenge to these established principles is met with resistance, as it is perceived as a threat to their core beliefs and hard-earned cohesion. This phase illustrates the intricate process of team formation and the evolution of a shared mindset within software development.
Since our team focuses on software development, its challenges typically include software development projects. If the group successfully uses Agile as their project management model, we expect them to have acquired an Agile culture.
Teams founded and established on one culture typically find it hard to work with or communicate with teams from other cultures. Much less open would they be to a cultural transformation that moves them from something they are already familiar with to a new and risky adventure.
The Agile culture is what we observe when a team embraces the ideas and notions presented above under the Agile framework and Agile approach sections. It also comprises their methods, practices, and techniques, which are unique to Agile and which we shall discuss shortly.
7.5 What Is the Agile Software Development Methodology?
The Agile methodology is the conceptual framework that includes a set of principles and guidelines governing the team’s methods for delivering specific software development projects.
While a “framework” is a more generic and abstract set of concepts, principles, and guidelines, a methodology applies only to a specific project or project type. For example, you might use Kanban, Scrum, or Extreme Programming (XP), depending on the context and the team’s familiarity with these methods.
Therefore, we can think of an Agile methodology as a set of principles and guidelines inspired by the Agile philosophy, centred on its framework, but customized to suit a specific software project or team type. Below is a brief overview of the most common Agile methodologies.
7.5.1 The Scrum Model
Scrum is an Agile framework that focuses on delivering value to the customer through incremental product development. It is a lightweight framework that allows teams to adapt to changing requirements and deliver high-quality products quickly.
The Scrum framework consists of three roles:
The Product Owner — The Product Owner is responsible for prioritizing the product backlog and ensuring that the team delivers the highest value to the customer.
The Scrum Master — The Scrum Master facilitates the Scrum process and removes any impediments that might prevent the team from achieving its goals.
The Development Team — The Development Team is responsible for creating and delivering the product increment to the customer.
Scrum was developed in the early 1990s by Jeff Sutherland and Ken Schwaber. Scrum was created in response to the need for a more flexible, iterative approach to software development that could handle change more effectively.
Scrum is based on the Agile Manifesto and emphasizes collaboration, flexibility, and rapid feedback. The Scrum framework consists of sprints and short iterations of work that typically last 1-4 weeks.
7.5.2 Kanban
Kanban is a visual management tool developed by Toyota in the 1940s. Kanban is designed to provide a clear visual representation of the work and the work that needs to be done.
In software development, Kanban is often used to manage the flow of work and identify bottlenecks in the development process. Kanban is also based on the Lean philosophy and aims to eliminate waste and improve efficiency.
As an Agile framework, Kanban emphasizes continuous delivery and visualisation of work through the Kanban board. It is a pull-based system that allows teams to work on tasks as they become available rather than waiting for a batch of work to be completed.
The Kanban framework uses a Kanban board to visualize the work that needs to be done. The board consists of columns representing different workflow stages, such as To Do, In Progress, and Done. As work moves through the workflow, it is moved from one column to the next.
7.5.3 The Lean Model
Lean is an Agile framework that emphasizes the elimination of waste and the delivery of value. It is a customer-focused approach that emphasizes continuous improvement and removing non-value-added activities.
Lean was originally conceived as a production management philosophy developed by Taiichi Ohno, a Toyota engineer, in the 1940s. The Lean philosophy emphasizes the elimination of waste and the continuous improvement of processes. In software development, Lean aims to reduce waste by focusing on delivering value to the customer and continuously improving the development process.
The Lean framework has five core principles:
Value — The value principle focuses on delivering value to the customer.
Value stream — The value stream principle emphasizes the elimination of waste
Flow — The flow principle emphasizes the importance of smooth and continuous workflows
Pull — The pull principle emphasizes the importance of responding to customer demand.
Perfection — The perfection principle encourages teams to improve their processes continuously.
7.5.4 Extreme Programming (XP)
Extreme Programming (XP) is an Agile methodology emphasising technical excellence and customer involvement in software development. It originated in the late 1990s and is known for its pragmatic approach to development practices.
Objectives:
Improve software quality through continuous feedback and testing.
Enhance collaboration between developers and customers.
Adapt to changing requirements with a focus on simplicity.
Key Differentiators:
Pair Programming: XP encourages pair programming, where two developers work together on one computer. This promotes knowledge sharing, code review, and improved code quality.
Test-Driven Development (TDD): Developers write automated tests before writing the actual code, ensuring that the code meets the specified requirements and can be easily tested.
Continuous Integration: Frequent integration of code changes allows for early detection of issues and ensures that the software is always working.
Extreme Programming (XP) differentiates from other Agile software development methodologies through:
Practices like pair programming, TDD, continuous integration, and technical excellence.
Placing a strong focus on customer involvement and adaptability.
7.5.5 Large Scale Scrum (LeSS):
Large Scale Scrum (LeSS) is an Agile framework designed to scale Scrum principles and practices to work with larger organizations. LeSS aims to maintain the simplicity and effectiveness of Scrum while addressing the complexities that arise in large-scale projects.
Objectives:
Apply Scrum principles to large, multi-team environments.
Enhance communication and collaboration across multiple teams.
Simplify and streamline processes for large-scale product development.
Key Differentiators:
Single Product Backlog: LeSS maintains a single product backlog for the entire organization, ensuring a unified prioritization of features and reducing dependencies between teams.
Full Scrum Implementation: LeSS retains the core principles of Scrum, including the roles of Scrum Master, Product Owner, and Development Team. This simplifies the adoption of LeSS for organizations already familiar with Scrum.
Feature Teams: LeSS organizes teams around complete features rather than specialized skills, fostering cross-functional collaboration and reducing handovers.
Large Scale Scrum (LeSS) can be distinguished from other Agile methods through the following:
Differentiates by providing a framework for scaling Scrum to large organizations.
Maintains the simplicity of Scrum principles while addressing challenges in large-scale development.
7.5.6 SAFe (Scaled Agile Framework)
SAFe (Scaled Agile Framework) is an Agile framework designed for large-scale enterprises. It is a framework that guides coordinating and aligning multiple Agile teams.
Dean Leffingwell developed SAFe in the early 2000s. SAFe is designed to provide a framework for large-scale Agile development. SAFe addresses the challenges of coordinating multiple Agile teams working on a large-scale project. SAFe provides guidelines and practices for managing work across teams and ensuring that teams are working towards the same goals.
The SAFe framework consists of three levels:
The Team — At the team level, Agile teams use Scrum or Kanban to deliver value to the customer.
The Program — At the Project level, multiple Agile teams work together to deliver larger initiatives.
The Portfolio — At the portfolio level, the organization aligns its strategic goals with the Agile Project.
7.6 What Are Agile Development Methods?
Methods are structured approaches to completing specific tasks. The following list specifies the most common tasks performed in Agile-based methodologies and the most popular Agile development methods in each case.
Project Initiation:
I believe Agile was originally developed for development teams, as Robert Martin stated, and was never meant to work far away from the implementation stages. This means that outside development teams (project managers, business analysts), classical solutions continue to apply.
Conducting workshops is a common method to initiate projects in Agile and Waterfall.
Project managers and sponsors sit together with their counterparts from the client side to agree on the project’s scope, budget, schedule, change management process, and work locations.
Once an agreement is reached, the details are captured in a Project Charter and a Project Initiation Document.
Project Planningand Governance:
Project plans and Gantt charts are replaced with dashboards showing the latest progress on items in the backlog.
Stakeholders wanting to know how much a feature costs and when it can be delivered can derive this information from the epic or story-level estimations and the velocity of the teams working on them.
User stories are fundamental in Agile development, specifically within the Scrum framework. They are concise, informal descriptions of a software feature or functionality from an end user’s perspective. User stories serve as a way to capture requirements and prioritize work in an Agile project.
While requirements documents (BRD) or gap analysis documents can still be useful, the ultimate communication tool between business and technical teams is the user story.
Solution and Software Design:
Because Agile software development breaks down large features into epics and user stories, and with a continuously updated backlog, massive design efforts at the start of the project stop making sense.
Cross-functional teams with senior software engineers create designs per epic or user story and can make decisions quickly.
Because features are added incrementally, major design decisions in Agile can be difficult to make. On the other hand, the design must be flexible enough to quickly accommodate new requests while remaining robust and consistent with the architectural context in place.
The evolution of customer preferences with the product and the technology is anticipated and forces the software teams to keep the design in the best shape to easily accommodate new requests.
Feature Development:
Features are developed by cross-functional teams that work on a user story until completion.
The Definition of Done (DoD) is crucial in standardising development work, ensuring no inconsistencies are present in the deliveries.
Agile development heavily relies on automation tools to automate its build, testing, and deployment.
Developers in an Agile team are now required to understand Agile software development methodology and be proficient in the various tools used in the build, test, or automation pipelines.
Because Agile teams are cross-functional, design decisions can be made quickly on isolated issues, allowing teams to move faster and become self-sufficient.
Showcases are a relatively new addition to the development cycle and are one Agile development method allowing developers to gather early user feedback by showcasing newly developed features to end users and requesting feedback.
Agile emphasises heavy testing within the development cycle and before features are deployed into higher environments.
The ability to run extensive and reliable testing on new features and regression testing on old features is supported through the use of automation tools.
Coverage is a key aspect of testing, and Agile development methods rely on unit tests from TDD and automated System Integration Tests (SIT) to ensure quality is not compromised.
Because Agile focuses on developing small features, testing those specific features is much easier as their sizes are typically smaller, and they can be promptly isolated.
Other types of testing, such as User Acceptance Tests or performance tests, are still relevant and are performed outside development teams. However, Agile development methods focus heavily on shipping quality software and reducing the number of bugs that make it into higher environments.
Support and Maintenance:
In Agile, like Waterfall, support and maintenance are conducted by separate teams with different competencies, cultures, and technical proficiencies in the product.
While this might make sense regarding budgeting, resourcing, and organisational structure and management, it can also be ineffective. However, merging support with development requires the team to work in a DevOps model, the latter being a topic in itself and outside the scope of this discussion.
7.7 What Is the Agile Process?
The Agile process is the systematic application of the Agile methods and framework. It can be understood through the following ideas.
Viewed from an altitude of 30,000 feet, the Agile process is identical to Waterfall with its typical five stages of Define, Build, Test, Deploy, and Monitor, with which we are all familiar.
These five stages constitute the familiar Software Development Lifecycle (SDLC), and it’s hard to imagine software built in another way.
A close-up view of the Agile development process with its four cycles played between business analysts, product owners, developers, and testers.
From the ground, however, the processes cannot be more different. Four key players, engaged in a continuous dialogue, run the following cycles:
The business cycle:
Business Analysts (or BA) are responsible for understanding and translating business needs into requirements. They work closely with stakeholders to define the project’s scope, prioritize features, and create user stories.
The Business Analyst defines the requirements, approves the changes demoed during showcases, and might provide input and feedback during the design phase.
The product cycle:
Product Owners are responsible for the product vision and its development. They work with business analysts to define user stories and prioritize the product backlog.
The product owners have a central and vital role: writing stories, prioritizing user stories in the backlog, and engaging the stakeholders.
Product Owners also work with the development team to ensure the product is delivered on time and meets customer needs.
The development cycle:
Developers create designs, write code, implement user stories, and run showcases.
They work closely with the product owners to understand the requirements and implement the user stories.
They also collaborate with the testers to ensure that the software is tested and meets the acceptance criteria.
The testing cycle:
Testers ensure the software is high quality and meets the acceptance criteria.
They work closely with the developers to identify defects and fix them.
They also collaborate with business analysts and product owners to ensure the software meets customer needs.
We can immediately notice from the diagram above that there are far more connections and interactions in Agile between the various players than in Waterfall, and that’s before adding project managers, scrum masters, architects, and other profiles to the mix.
What does that mean? First, Agile is not intuitive to understand or explain, but more importantly, it doesn’t scale well and is bounded by a strict domain of applicability. Within that domain (see Agile Design), Agile is very powerful.
The Agile process has a main cycle that starts with features defined in epics, followed by epics broken down into user stories. The epics (and their child user stories) are then placed in their appropriate position in the backlog. Next, we have user story development, testing, and deployment into production.
Secondary cycles are allowed within the main cycle, where features, epics, and user stories can be refined through dialogues between product owners, business analysts, and software developers.
At the heart of the Agile work organisation process is the user story, which, in theory, is a single unit of meaningful business value. A user story can be developed and deployed on production in isolation, making smaller, more frequent releases possible.
In Agile, quality is much easier to monitor since issues can be linked (causally and through integration or unit tests) to a particular user story and its associated changes. This is a major advantage over Waterfall, where quality problems are more easily hidden.
Budget and cost are much trickier to control in Agile since requirement clarification and refinement can be ongoing. While effort spent can be easily counted, it cannot be easily predicted.
Resource Allocation
Most Agile teams prefer to use the “pull system”, which, in theory, is more efficient than the push system. The pull system requires developers who run out of work to pick the top priority item in the backlog.
This focus on completing one change before moving on to the next reduces developers’ context-switching time moving between tasks. This makes the pull system more efficient in such situations.
In real-world scenarios, however, there are times when developers must immediately switch to other, more important tasks, such as production issues. In such cases, the pull system doesn’t work well and must be supported by another protocol that dictates when a developer’s current task can be interrupted in favour of a more important one.
Overall, Agile teams typically form pools of resources that can be dynamically allocated based on workload. This can be efficient when the workload is variable but inefficient if tasks need to be interrupted in favour of more critical ones.
A practice is a method, or set of methods, applied repetitively in pursuit of technical proficiency. Agile practices are numerous, of which we distinguish the following:
Backlog management:
Roles, responsibilities, and practices have been developed and refined over the last years to manage a backlog efficiently.
This has been supplemented with powerful tools like JIRA and Confluence that take away much of the manual work that would otherwise need to be done. These tools typically also provide dashboards and other utilities to facilitate backlog management.
User story creation:
Writing user stories is another Agile practice that has been perfected over the years. Powerful and sophisticated guidelines on how to write user stories, specifically around acceptance criteria and design and implementation details, exist in most teams.
One of the most powerful features of Agile user stories is that they can be adjusted to suit a particular team, product, and style.
Product and feature design:
Agile practices such as short user story implementations have placed severe constraints on product design practices (in a good way).
Breaking down large features into small user stories ensures that designs are modular and adhere closely to adopted architectural standards. Quality issues are promptly uncovered, preventing design issues from proliferating.
Gathering feedback:
Gathering feedback information is another area where Agile has excelled.
Agile teams implement many Agile techniques to facilitate user access to prototypes and provide early feedback to developers. Demos, showcases, and prototyping are three of these methods.
Team structure:
Agile teams are cross-functional, combining a rich tapestry of skills and expertise covering programming, testing, database administration, and network security and administration.
Agile teams are self-organized, meaning work allocation and development processes are customized to fit the particular structure and context in which this team operates.
Forming self-organized and cross-functional teams is a standard Agile practice, although not as easy as the other practices mentioned above, as it’s subject to other constraints such as skill availability, team culture, and organisational structures.
Sprint planning and execution:
Sprints are short periods (usually two weeks) during which the team develops features.
Sprints allow the team to focus on specific features and ensure the product is developed iteratively and incrementally.
Many Agile techniques and rituals facilitate sprint planning and execution. These include sprint planning sessions, story-point estimation, team velocity calculation, and sprint retrospectives.
The Scrum Master is a dedicated role created to manage these tasks.
A technique is a method perfected by repetition and which includes art and skill in its execution. The most powerful Agile techniques can be summarized as follows:
7.9.1 Progress Monitoring Techniques
In Agile development, various charts are used to monitor progress and facilitate communication within the team and with stakeholders. Here are some commonly used charts:
Burnup Chart: Similar to a burndown chart, a burnup chart tracks the cumulative work completed over time. It also includes the total scope of work, providing a clearer picture of completed and remaining work.
Velocity Chart: Velocity measures the amount of work a team can accomplish in a sprint. The velocity chart shows the team’s historical velocity over multiple sprints, helping in predicting how much work the team can handle in future sprints.
Cumulative Flow Diagram (CFD): A CFD visualizes the workflow through different stages of the development process. It provides insights into the distribution of work items, highlighting bottlenecks and enabling teams to optimize their workflow.
Release Burndown Chart: Similar to a sprint burndown chart, the release burndown chart tracks the progress of completing all the user stories or features planned for a release. It helps teams ensure they are on track to deliver the entire release scope by the target date.
Lead Time and Cycle Time Charts: Lead time represents the total time taken from creating a work item to its completion, while cycle time is the time taken to complete a work item once actively worked on. These charts help teams understand and optimize their overall process efficiency.
Control Charts: Control charts are used to monitor the stability of a process over time. They can be applied to metrics like burndown rate, cycle time, or lead time, helping teams identify variations and trends in their performance.
Dependency Board: While not a traditional chart, a dependency board visually represents dependencies between user stories or tasks. It helps teams manage and mitigate dependencies, ensuring a smoother workflow.
Quality Metrics Charts: Charts displaying metrics related to code quality, such as code coverage, defect density, or test pass rate, can be valuable in monitoring the overall health of the project.
These charts provide a comprehensive view of a project’s progress, enabling Agile teams to make data-driven decisions, identify areas for improvement, and communicate effectively with stakeholders.
7.9.2 User Story-Writing Techniques
The Agile User Story was first proposed by Kent Beck in 1997 and subsequently formalized and standardized in 2004 by Mike Cohn.
A User Story consists of a logical unit of business requirements viewed from the perspective of the product’s end user. There are several templates in use along the lines of:
As a <role> I can <functionality>, so that I get <business value>.
User Stories are grouped in Epics, which can be regrouped into larger units such as Initiatives.
A critical component of a User Story (and indeed any requirement) is Acceptance Criteria, which is nothing but a list of statements in the form of:
The product [shall, should] support this or that <functionality>.
User Stories typically follow a standard workflow and may be divided into smaller tasks and subtasks. Their progression is controlled by specific rules called Definition-of-Ready (or DoR) and Definition-of-Done (or DoD).
The setup of a user story template, workflow, DoD, DoR, and subtask format is implementation-specific and is acceptable as long as it doesn’t contradict any of the Agile principles.
User Stories are a popular means of capturing business requirements in Agile development because they offer several advantages over other methods:
User Stories are customer-focused — User Stories are written from the user’s perspective, describing what the user needs to accomplish. This helps to keep the focus on the customer, ensuring that the final product meets their needs and expectations.
User Stories are flexible — User Stories are brief and allow for changes and adjustments throughout the development process. This flexibility means that requirements can be refined or changed as the team better understands the customers’ needs.
User Stories are easy to understand. User Stories are written in simple language. This makes them accessible to all team members, including business stakeholders who may not have a technical background.
User Stories encourage incremental delivery — User Stories are often broken down into small, manageable pieces, enabling incremental delivery. This approach allows the team to deliver working software in short iterations, software that customers can test and validate.
Stakeholder management in this context refers to a product owner’s ability to answer questions about the scope, schedule, or both. Examples of such questions can be:
When can we expect to have feature A deployed?
Can we have feature A deployed before DD-MM-YYYY?
Which features can be ready by DD-MM-YYYY?
To answer these questions, a product owner will refer to the story point estimation of each user story and how many “story points” a team can complete within a fixed amount of time, like a week or two, a metric sometimes referred to as velocity).
Effort estimation and story points are subtle notions that require lengthier discussions. They have been discussed in more depth here.
In summary, a story point is a label developers assign to user stories to indicate their implementation difficulty. Story points are finer at small scales and grow coarser with the scale. The Fibonacci sequence is typically used:
Story point set: 1, 2, 3, 5, 8, 13, 21, 34...
Story points of remaining work and team velocity can be placed on a chart to predict when the project will be completed.
In Agile development, technical debt refers to the accumulation of design and development issues that must be addressed. Agile deals with technical debt by emphasizing the importance of technical excellence and continuous improvement.
Here are some ways Agile deals with technical debt:
Continuous Integration and Continuous Delivery — Agile development teams use automated build and deployment processes to deliver the software quickly and frequently. This helps to identify and address technical debt issues as they arise before they become more significant.
Refactoring — Agile development teams continuously refactor their code, restructuring it without changing its external behaviour. This practice helps improve the code’s quality and reduce technical debt.
Sprint Retrospectives — Agile development teams hold sprint retrospectives to reflect on the previous sprint and identify areas for improvement. This allows the team to identify technical debt issues and create action items to address them in the next sprint.
Technical Debt Backlog — Agile development teams maintain a technical debt backlog, a prioritized list of technical issues that must be addressed. This backlog is reviewed regularly during sprint planning, and items are prioritized and addressed as needed.
7.9.5 Reflection and Retrospectives
Agile requires practitioners to reflect regularly on how to become more effective. This strive for continuous improvement is implemented via retrospectives.
Retrospectives work by getting the team in a room every 4-6 weeks and retrospecting on the last period. The person running the retrospective writes down what has and hasn’t worked and, through collaboration with the team, determines what actions can be taken to avoid experiencing the issues in the future.
Retrospectives and similar rituals require a high degree of discipline so that actions are followed up and diligently executed. The team and organisational culture should also be mature enough to conduct the sometimes painful pinpointing of mistakes without assigning blame and finger-pointing.
8. Agile Methodology Advantages over Waterfall or DevOps
The most deadly thing in software is the concept, which almost universally seems to be followed, that you are going to specify what you are going to do, and then do it. And that is where most of our troubles come from. The projects that are called successful have met their specifications. But those specifications were based upon the designers’ ignorance before they started the job.
Waterfall project delivery methodology can be heavily constrained by several factors, as evidenced by studies in project management.
The primary constraint is that business requirements must be well-defined at the outset of the project for the sequential nature of progression in Waterfall to work effectively (Boehm and Turner, 2003).
If the requirements are not well-defined, Waterfall will struggle and may ultimately fail. The lack of iterative processes in any of its stages is another drawback of Waterfall, which makes it less accommodating towards changing business requirements.
Large projects are particularly vulnerable to failure when using the Waterfall methodology because of their prolonged implementation period and the associated delivery delays (Mckenna and Copeland, 2017).
As the duration of a project extends, there is increased pressure to unlock requirements and scope, which can also cause Waterfall to fail.
Furthermore, project delays tend to follow power laws rather than normal distributions, meaning that the larger the project, the higher the expected delay ratio to project duration (Candea et al., 2017).
The drawbacks of lengthy projects and changing requirements in Waterfall-run projects can lead to longer time-to-market, late detection of divergence in customer needs from implemented features, and an irreducible and high cost of change (Browning, 2016).
These challenges highlight the importance of carefully considering the Waterfall methodology’s suitability for a particular project and the need to weigh its advantages and disadvantages.
To mitigate the risks associated with the Waterfall methodology, project managers can consider using other approaches emphasising flexibility and iterative processes, such as Agile or Scrum.
By selecting the right methodology, software teams can minimize the likelihood of project failure and enhance their chances of delivering a successful project.
9. Final Words
The first time I heard of Agile was about a decade ago, and I was entirely sceptical about its performance. Being a thorough believer in the no-free-lunch paradigm, I was convinced that a trade-off was made somewhere and that Agile promoters avoided discussing it.
It took me a bit less than a decade to appreciate Agile’s advantages (and challenges), and this only happened after I had become part of a fully Agile team for the first time in my career.
Although Agile is far superior to Waterfall in every aspect, it is not a silver bullet. Cracks in the Agile edifice can become immediately evident as project size steps beyond certain thresholds or when Agile is applied to specific projects.
Candea, G., Kaushik, R., & Fox, A. (2017). Power laws and the classification of extreme events in rapidly changing networks. Physical Review E, 95(2), 022310.
Mckenna, P., & Copeland, S. (2017). Using project management software in agile projects: A review of the literature. International Journal of Managing Projects in Business, 10(4), 820-837.
