1. Overview
The Agile vs Waterfall discussion remains relevant today even after almost two decades of the announcement of the Agile manifesto.
We know that Agile adoption has been partial and slow, and the 15th State of Agile Report goes into quite some detail on that. If you are still undecided on which methodology is the most suitable, you have come to the right place.
Project delivery can be divided into two categories: lightweight, which includes Agile and Extreme-Programming among others, and heavyweight such as Waterfall.
Although we recommend adopting Agile practices as one of the principles of Operational Excellence in software development, we do, however, first recommend that you gain fluency in the topic.
Agile and Waterfall are built on two different sets of core principles. While Waterfall methodology is pretty intuitive, Agile may not be as much and its fundamental ideas need a bit of unpacking.
The purpose of this article is to do precisely that; give the reader enough information on both methodologies to allow for a wise decision.
We have also published a guide on choosing the best project delivery methodology for your team. We recommend you have a look once you are done with this one.
This article will compare and contrast three models: Agile, Waterfall, and a Hybrid version, and introduce a fourth one (DevOps). We will also explain some of the implementation details of Agile for those looking to migrate.
2. Table Of Contents
3. Selecting a Methodology
3.1 Why It Is Important to Make the Right Choice
It is essential to recognize that Agile is not a silver bullet nor the default choice when unsure. Make a careful choice for the below reasons.
First, choosing an unsuitable delivery method will have the opposite effect of what you desire: it might aggravate your team and introduce unnecessary delays in your projects.
Second, implementing the correct project delivery methodology will help you:
- Deliver faster, higher quality products
- Achieve Operational Excellence
- Provide a solid foundation on which you can design and build efficient production processes
- Speak a suitable, familiar language with partners, suppliers, and customers
Finally, it would be easier to improve on a working model than a broken one.
3.2 Slow Adoption of Agile
According to the 15th State of Agile Report, 94% of the participants reported using Agile practices in their organization. That’s, however, not the whole story. Have a look at the below stats from the same survey. Only 32% however reported using Agile for more than 5 years.
Out of the 94% using Agile, 68% have been using it for less < 5 years. We believe this fact requires some explanation, especially since Agile has been around since 2000! One of the central themes of this article is finding the root causes behind that phenomenon.
4. Waterfall vs Agile FAQ
If you would like some quick answers.
5. Waterfall Methodology
5.1 What is Waterfall
Waterfall was first described in a presentation at the Symposium on Advanced Programming Methods for Digital Computers on 29 June 1956 by Herbert D. Benington (Wikipedia).
Waterfall consists of breaking down all project tasks and activities into a pipeline of sequential stages where each stage depends on the output from its predecessor.
Later on, in 1970, a paper published by Winston Royce on Managing the Development of Large Software Systems gave Waterfall its current fame and stature. The term Waterfall was never used until it appeared in a 1976 paper by Bell and Thayer.
5.2 How Does Waterfall Work
The Waterfall model is a series of sequential stages where each stage of the pipeline specializes in one area of the software development lifecycle.
The stages of Waterfall typically are Initiation, Analysis, Design, Development, Testing, and Deployment.
In a properly setup process chain, each stage in the SDLC has an owner responsible for executing a set of transformations on inputs it has received to produce artifacts, which will serve as inputs to the next stage.
The table below describes the five stages and the deliverables produced in each of them.
| Waterfall Project Phase | Deliverables |
|---|---|
| Project Initiation | Project Initiation Document: defines the project sponsors, pre-requisites, overall scope. Business Requirements Document: details the business requirements to be met. SLA or Service Level Agreement. |
| Analysis | Software Orders of Magnitude: a list of high-level features (or Epics) and a rough order of magnitude of their efforts. High-Level Design: represents a bird’s eye view of what the solution looks like. Solution Design: This is a blueprint of the system to be implemented. Detailed Project Plan: includes all the detailed tasks required for successful delivery and a detailed effort estimation Gap Analysis: usually conducted to understand the differences between what the system offers off-the-shelf and what the end result looks like. Statement of Work: describes the scope, deliverables, timeframes, prerequisites and milestones. Test Strategy: a high-level plan of how the new releases will be tested. More of a guideline to testing than actual test cases at this stage. Discusses test approaches, test environments, and resourcing. |
| Development | Source Code: new version of the source code Test Scripts: the set of test cases to be used for testing the changes. These also include any Test Automation scripts or unit test scripts in case Test-Driven Development is used. User Guides and Reference Manuals: can also be produced at this stage. |
| Testing and QA | Test Cases: an actual test case with summaries, acceptance criteria, steps to produce, and test parameters. Test Report: a report of the test runs with details on failed cases. |
| Deployment and Support | Rollout Plan: a step-by-step plan on how production deployment will happen. Typically with a duration for each step. This is especially important if downtime is involved. Installation, Admin, and Operations Guides: these are the manuals required to run the system in production mode. DevOps Script: if DevOps practices are used for building and deploying applications. |
5.3 Key Principles Behind Waterfall
Waterfall project management, when compared to lightweight methodologies like Agile, presents two very distinct features.
The first feature is massive planning in the form of design and documentation, and the other is using a single sequential pipeline for delivery.
The most significant chunks of work in the delivery pipeline are usually design and development.
Waterfall emphasizes design activities to limit the scope and impact of any changes and therefore reduce the size of testing to the bare minimum. Most experts recommend a ratio of 70/30 for design and development.
The downside of this approach is that planning and design happen only at the beginning of the project, and revisiting the plans at any later stage is very costly.
To mitigate the risk of any rework, Waterfall tries to get it right the first time.
5.4 Advantages of Using Waterfall
Despite its core weaknesses, Waterfall has many advantages:
- Waterfall runs on an intuitive model that is easy to explain and understand. The sequential nature also makes it simple to manage.
- A fixed structure with clear milestones, checkpoints, and deliverables requires a straightforward implementation process with basic management techniques and little hand-shaking and coordination between different teams.
- A fixed project scope reduces the overhead associated with rescoping, redrafting documents and project plans, and acquiring approvals from senior management.
- Generous documentation. No one generally complains about having proper and quality documentation.
These advantages, however, are not enough to justify using Waterfall. A Hybrid model can have all the benefits and very few of the disadvantages of Waterfall. It also combines the good features of Agile to make it even more appealing. We will have a few words to say on Hybrid models later on.
5.5 When Is Waterfall Inappropriate
Waterfall project management works great if the following criteria are met:
- The requirements are well known at project inception. The chances of conditions changing later on in the project are minimal.
- The customer can see the end result and sign it off before development starts so that the chances of developers building unwanted or unusable features are tiny.
- The technology used must be mature and well-understood. Any issues of a technical nature are quickly resolved. Using only mature technology is one of the principles of Operational Excellence.
Points 1 and 2 are usually challenging to satisfy. Here are some reasons why that may be true:
- In some cases, such as building a website, the customer may not know what they want until they see it.
- Using websites again as an example, you may not be able to gauge the user experience until you deploy on production. This weakness is ubiquitous for issues related to the look-and-feel of websites.
- For sufficiently big projects, customer priorities might change. Although not as likely as points 1 and 2, this is still possible.
The below table lists a few real-world examples of scenarios where Waterfall cannot cope.
| Scenario | Example |
|---|---|
| Unknown or Unclear Requirements In these cases, the requirements can never be fully understood at the beginning of a project. | 1) Migrating legacy, poorly-documented systems are an example where requirements can never be fully known before testing. 2) Performance requirements for applications that outperform expectations. 3) Websites where users’ behaviour patterns cannot be predicted. 4) Innovative products yet to be understood. |
| Lengthy Projects / Changing Requirements If the duration of the project is extended, chances are that some requirements might change. | 1) Regulatory updates in financial products are one example of requirements changing during the project. 2) Change in client priorities or business strategy. |
| New, Poorly-Understood Technologies The team is not fluent in the technology used. | 1) Supporting new OS platforms or web framework 2) Deployment mode changes like offering the product as a service |
The below caricature is the best explanation of what usually happens.
6. Agile Methodology
6.1 What is Agile Project Management
In 2001, seventeen software developers convened at a resort in Snowbird, Utah, to discuss lightweight (in contrast to heavyweight, collectively known as Waterfall) development methods.
Together they published the Manifesto for Agile Software Development which consisted of 12 principles that embodied the essence of Agile.
The below list shows five of these principles where the deviation from Waterfall is most prominent:
The experts who convened in Utah acknowledged changing requirements as an integral part of software development.
As such, changing requirements was something that software developers needed to work with rather than try to eliminate.
The question moved from: How do we ensure we have the final requirements to let’s adjust our processes so that changes, even those that come at the later stages of a project, are not too disruptive.
Agile heavily favours shorter delivery timeframes, and for good reasons. Because Agile is designed to handle changing requirements better, the idea of releasing lighter features more frequently allows issues to surface faster.
Also, minor changes are less likely to be buggy and easier to troubleshoot. This philosophy of Agile is much more efficient from a time management perspective as it allows valuable feedback as early as possible.
Agile emphasizes face-to-face interactions over lengthy and time-consuming reports of low-quality information. This suggestion does not say that investing in design documents or other helpful documentation goes against Agile principles.
We have discussed self-organizing teams at length in our article on decision-making processes. Be sure to check it if interested.
For our current discussion, suffice to say that a self-organizing team leverages the synergistic relationships from group interactions. Back-and-forth discussions where valuable opinions contribute to the design and architecture of the software are highly recommended for Agile practitioners.
The natural and organic manner in which self-organizing teams evolve is highly effective in dealing with changing environments.
Continuous improvement is a powerful discipline that helps keep your business afloat in a changing environment.
Agile requires a constant reflection on past sprints or projects in the form of retrospectives, while Waterfall does not address any such requirement.
Continuous improvement, however, is not specific to Agile but is just something to which Agile has given some attention.
6.2 Agile Frameworks
Software development experts have created numerous practices and techniques in the spirit of Agile. Some of these ideas even predate the Agile proclamation:
- Scrum from Ken Schwaber, Jeff Sutherland
- Kanban by Taiichi Ohno
- Behaviour-driven development (BDD) by Dan North, Liz Keogh
- Refactoring by Martin Fowler
- User story by Alistair Cockburn
- Backlogs (Product and Sprint) by Ken Schwaber
Being Agile does not necessarily require the adoption and implementation of all of these techniques. Some of them, however, like User Stories and Backlogs, are so powerful and straightforward that they have become a symbol of Agile.
6.3 What Is the Key Principle of Agile
A robust and straightforward idea lies behind the Agile way of work. Understanding this idea will help you understand what Agile is meant to be and how it should be used for maximum efficiency.
Agile (or any iterative system with a feedback loop) operates by following the below steps:
- Step 1: Determine your current position with respect to your ultimate objective.
- Step 2: Adjust your direction so that it points to your goal.
- Step 3: Make a small incremental step in the new direction.
- Step 4: Repeat from Step 1
This approach to problem-solving is ubiquitous in almost every area of our daily lives.
For example, think of technology and engineering, where small progress is made through massive iterations and continuous improvement on existing models:
- Neural Networks use a similar “algorithm” called back-propagation to learn from historical data (or, more formally, to minimize an error function).
- Moving robots use similar methods to calculate the next thrust of their motors.
- The Product Kata framework in Product Management is centred around feedback loops and iterative product improvement cycles.
6.4 Understanding Cost-of-Change
We cannot honestly conclude any discussion on Agile and Waterfall without explaining the concept of cost-of-change.
Because of how software development works, i.e. in sequential stages where each stage depends on the previous one, redesigning a particular feature usually incurs a high cost. Experts place that cost at about ten orders of magnitude above the original one.
The price you usually pay to redesign and redevelop a feature is called cost-of-change.
The best way to keep the cost-of-change low is to try and get it right the first time. Have a look at the below graph.
Experts believe that cost-of-change rises exponentially with every stage of Waterfall.
Agile and other lightweight methodologies solve this problem by breaking major changes into smaller pieces, shipping features faster into production and relying on early user feedback.
However, there are some types of projects that are high on cost-of-change by nature, regardless of whether lightweight methods are used or not. Think of the construction project of a high-rise building where you cannot prototype the foundations or use iterative procedures to find the best design; you need to get it right the first time.
One example of such projects is those that require certification by regulatory authorities. Often, these certifications are lengthy and expensive, so people try to do less of them.
6.5 When Is Agile Suitable to Use
Agile is appropriate in the following scenarios:
- The cost-of-change is low.
- The requirements are not or cannot be fully known before implementation starts.
- Chances are that the requirements will change during the implementation.
- A lot of the testing has been automated.
6.6 Agile and Test Automation
In a typical Agile sprint, you need to design, develop, and test a new feature in about two weeks.
However, you cannot even commit these changes to your master branch without test automation and decent coverage. At least, not as frequently as you like.
There needs to be a complete and manual regression cycle for every change. Lack of a decent Test Automation Framework will cripple the best Agile practices.
7. The Hybrid Model
7.1 Is There a Third Option?
Hopefully, the Agile discussions in the previous sections have convinced you that Agile is not the silver bullet we are told it is. If this is true, and indeed it is, is Waterfall still an option? Is there a different option?
The answer to this question is Yes. There is a third option that combines some of the great features of Waterfall and Agile and merges them into a Hybrid model.
The Hybrid model will be the subject of our subsequent discussions.
7.2 What Is the Hybrid Model
The Hybrid model is sometimes referred to as Wagile, Waterfall-Agile, or Wet Agile.
The Hybrid model is classic Waterfall retrofitted with powerful and non-obtrusive Agile practices such as relatively more frequent software drops, daily stand-ups, showcases, Kanban boards, story points, and automated testing.
Perhaps the most notable thing to remember is that Hybrid models are Waterfall at their core. Four of the five principles (that is all except Collaboration) that make Agile so different from Waterfall and that have been elucidated earlier remain fundamentally outside the scope of Hybrid models.
7.3 Advantages of the Hybrid Model
Following are some of the advantages of the Hybrid model with which I have personally had some first-hand experience.
This statement means that software developers, already familiar with Waterfall since the 1980s, do not need to undergo a paradigm shift to use the Hybrid model.
You can always start with a classic Waterfall Model and, at a comfortable pace, start adding Agile practices that are compatible with your process flow.
The Hybrid model can be introduced seamlessly into current development processes with little resistance. What happens is, when people experience the prowess of Agile practices, they find it extremely difficult to go back to the old ways.
Agile practices such as TDD, test automation, software drops, and showcasing allow software teams to uncover problems early on and catch up with damaging delays by speeding up the testing.
8. Agile vs Waterfall Comparison Table
The below table presents a comparative analysis of the Agile vs Waterfall vs the Hybrid model we discussed earlier.
The table performs a comparison along six different dimensions: core principles, processes, implementation difficulty, objectives, and pros and cons, and necessary ingredients for success.
| Waterfall | Agile | Hybrid | |
|---|---|---|---|
| Core Principles | Design to development effort ratio of 70/30 | Continuous Improvement, Quality, Collaboration, Changing Requirements | Allows large design efforts, embraces continuous improvement and collaboration |
| Work Process | Sequential stages | Iterative methods with quick feedback. Each cycle (or sprint) has design, development, and testing amalgamated together. | Sequential stages with quick feedback |
| Objective | Deliver complex software projects | Deliver complex projects while allowing for uncertainty | An improved Waterfall-based model with some resilience to uncertainty |
| Pros | Easy to explain. Great for high cost-of-change projects. | Improves efficiency. Allows for requirements to change. | Intuitive. Combines the best of Waterfall and Agile |
| Cons | Introduces high cost-of-change. Is not flexible when it comes to changing requirements | Requires a lot of discipline. Not very intuitive. | Is not a full replacement for Agile |
| Difficult to Apply? | No | Yes | Somehow |
| Cost-of-Change | High | Low | Medium |
| Requires Test Automation | Recommended | Crucial | Mandatory |
9. Waterfall vs Agile vs DevOps
Before safely closing this discussion, we need to say a few words about DevOps.
DevOps is a set of practices (tools and processes) that allows you to achieve continuous delivery, a topic worthy of its own discussion.
For the moment being, suffice to say that DevOps takes Agile two steps further; one step towards fusing Development and Operations, the other towards Continuous Delivery.
DevOps requires a certain level of maturity in automation, not just test automation but also many other levels. It also requires a significant shift in mindset and a big investment in infrastructure, skills, and resources.
There is no harm, though, from looking at DevOps and what it brings to the table.
More importantly, you might want to find out whether there are any low-hanging fruits that you can benefit from immediately. There are indeed some tools and practices that you can apply directly and at a minimal cost, and the most promising of these is build and test automation.
10. Implementing Agile
If you decide that Agile is the right choice for you, remember the following items:
10.1 Topic Fluency
If you have come so far, you must have realized how complex this topic can be. Try not to take things for granted. Verify the details for yourself. These pieces of advice will help you make better decisions and avoid costly mistakes.
10.2 Appreciating the Implications
Understanding the initial setup effort, long-term implications, and the technical skills required to apply each methodology before committing to one path or another is essential.
Appreciating the potential difficulties that may arise from organizational culture resistance to change and lack of sponsorship from senior management is crucial if you want to succeed.
10.3 Publish an Operations Guide
Once you have decided on a project methodology, it’s always a great idea to internally publish an operation guide.
This document will help people understand the big picture, including their roles and responsibilities. It will also give people a solid reference when in doubt and prevent improvisation during implementation.
Most importantly, a published document pushes everybody to comply and sets the processes for future improvements. Without standardization, improvement is impossible.
11. Further Readings
- Great talk by Robert (Uncle Bob) Martin.
- Extreme form of Agile in the intro to this lecture by Allen Holub.
12. Featured Articles
DevOps: Finding Your Path to Successful and Continuous Improvement
1. Overview Looking at the below graph from Google Search Trends, we can see that DevOps has picked up much momentum in the last five years and is steadily increasing, making it interesting to understand more about it. The graph compares DevOps search trends with Agile and Waterfall, both software project delivery methodologies. This categorizationContinue reading “DevOps: Finding Your Path to Successful and Continuous Improvement”
Solution Design and Its Role in Successful Projects
1. Overview From the 1970s onward, IT software projects acquired a notorious amount of complexity due to the advancement of hardware (exponential rise in processing power of the personal computer) and software technologies (proliferation of programming languages and frameworks). These advancements captured the user’s interest and radically changed their preferences toward more modernization and digitalization.Continue reading “Solution Design and Its Role in Successful Projects”
Software Estimation: How to Get It Right the First Time
1. Overview Many IT professionals believe that estimating software with arbitrary precision is near impossible. And that’s, in fact, very true. Difficult as it is, estimating software development tasks is a necessary evil, and you need to get it right, at least within a small, consistent margin of error. In effect, unreliable estimation is cited as oneContinue reading “Software Estimation: How to Get It Right the First Time”
Test and Automation Strategy: A Deep-Dive Into an Essential Solution for Your Daily Agile Practices
1. Overview Software Testing and Quality Assurance is an attractive topic from a software delivery perspective for a very simple reason: it’s an area where you can make substantial gains and improvements in delivery capabilities. Why? Because of the current state of software testing, a topic we have analysed in great detail. In summary, softwareContinue reading “Test and Automation Strategy: A Deep-Dive Into an Essential Solution for Your Daily Agile Practices”
Customer Support: How to Drive Efficiency and Satisfaction
1. Overview Post-production support is the final stage in the value chain of your software delivery processes. Keeping the customer happy can be challenging but its also rewarding as its one step further towards achieving Operational Excellence. In this article, we will go over some practical thoughts on how to improve your day-to-day operations andContinue reading “Customer Support: How to Drive Efficiency and Satisfaction”
FAQ – Software Development and Delivery
Project and Delivery Management What is… Agile Project Management Waterfall Project Management Technical Debt Continuous Delivery Continuous Integration Continuous Deployment DevOps Missing in Agile and Waterfall Self-Organizing Team Cost-of-Change Hybrid Model for Project Management Should you… Start Using DevOps Keep Using Waterfall Use a Hybrid Model How does… Agile Work Waterfall Work How to… ChooseContinue reading “FAQ – Software Development and Delivery”