Waterfall, Agile, and DevOps: A Critique of Current Challenges

1. Overview

We introduced Waterfall, Agile, and DevOps in three previous articles, focusing primarily on their history, advantages, and core principles.

To complete the discussion, this article will examine their relative strengths, challenges, limitations, and adoption in the software community. 

The information we provide is derived from the 15th State of Agile Report and the 2021 State of DevOps Report, supplemented by practical, real-world experience from the field.

Finally, you can check these guidelines if you want to migrate your delivery processes but are still unsure which one best fits your team and organization.

2. Approach

2.1 Assessment Criteria

A few things to note regarding the approach we intend to take:

• The criteria used are insensitive to the variations of each method. We believe the results will remain relevant whether you are using Scrum, Kanban, Extreme Programming, or whichever preferred implementation of Agile you might have.

• The Hybrid model is not in scope for this discussion. The reader, however, will recognize this model’s challenges after carefully considering the difficulties of its roots, Waterfall and Agile.

• The assessment will look at six requirements for each methodology: accessibility, team structure and maturity, infrastructure and tools, scalability (vertical and horizontal), cost of change, software architecture and design.

2.2 Terms and Definitions

The only terms in this article that require additional clarifications are:

• Vertical scalability — refers to the ability of a delivery methodology to work on different levels, for example, on the technical and management level.

• Horizontal scalability — in this context, refers to the ability of a delivery methodology to function across different teams or project streams.

• Cost of change — represents the effort you need to invest in fixing a problem or updating an existing feature. It is the total effort to redesign, reimplement, and restest an updated version of the product.

3. Assessment of Agile, DevOps, and Waterfall

3.1 Before We Continue

We highly recommend reading the introductory Agile, Waterfall, and DevOps articles for background, context, and an understanding of their main concepts.

The below statistics come from the 15th State of Agile Report, and provide us with a snapshot of the current state of Agile in terms of adoption within software communities, organizations, and teams.

The first pie chart with 3% of participants reporting a non-Agile methodology, presents a bleak future for Waterfall. No surprises there.

What is surprising, however, are the numbers in the second chart indicating that only 18% of participants are fully Agile, whereas the rest still use some form of Waterfall.

Why is Waterfall lingering around two decades after the publication of the Agile manifesto? Is it because Waterfall is easier or even better than models? The answer requires some explanation.

We will assess each model based on six criteria in the following sections. This assessment will allow for a fair comparison of all three and perhaps allow us to appreciate more the dynamics of Agile adoption.

3.1 Waterfall

3.1.1 Accessibility

The Waterfall model is the most basic and intuitive software delivery methodology among the three for the below reasons:

• Its progression is sequential and, therefore, easy to follow. Iterations are allowed, but their use and implementation would make sense to anybody.

• Its stages are self-explanatory; we even see the same line-up across many other industries where products or services are created.

• Its roles are distinct, straightforward, and familiar. Rarely would staff need to wear multiple hats or become skilled in areas besides their immediate job.

• Process governance and management are easier than Agile or DevOps due to the simplicity of the setup.

3.1.2 Team Structure and Maturity

Teams of developers, testers, architects, and business analysts using Waterfall are typically compartmentalized and often work in silos.

One of Waterfall’s significant drawbacks is this lack of cross-team interaction and mutual support between the different groups.

Frequent misunderstandings, misalignments, and lack of visibility and empathy across boundaries reduce the team’s productivity.

On the positive side, Waterfall teams typically have one role per member, which leads to faster maturity, easier integration, and a gentle learning curve.

Self-organisation is limited, with power relationships leading to complex and unconstructive shadow hierarchies.

3.1.3 Tools and Infrastructure

Gantt charts, spreadsheets, and a basic source code versioning tool are the necessary and sufficient equipment to implement a solution using Waterfall.

This lack of sophistication makes the model ready to deploy and run but also highly inefficient, primarily due to the lack of automation and the dominance of manual and human intervention leading to all sorts of errors.

3.1.4 Vertical and Horizontal Scalability

The Waterfall model has no issues with vertical scalability. Managers would align expectations with developers based on effort estimations in person-days provided by the latter.

These estimations would then be used to create a schedule and calculate a cost; if these do not align with the stakeholders’ expectations, resource allocation and project constraints are modified.

Efforts across multiple project streams or teams are still required regarding horizontal scalability. It is typically done via workshops, project meetings, steering committees, etc.

3.1.5 Cost of Change

Cost of change in Waterfall is effortless to describe; it has been identified as early as 1970 in Winston Royce’s famous paper, “Managing the Development of Large Software Systems”.

In this paper, Royce points out that cost of change increases dramatically as you move down the Waterfall stages for two reasons; the farther you are in the implementation:

• The fewer alternatives you would have, and
• The farther back you might need to go to fix a problem

Two critical enhancements were applied to remedy these problems.

First, investment in massive design efforts was made to foresee potential challenges in solution architecture and software design before implementation began.

Second, system integration and unit testing were adopted to limit the number of bugs in the software testing phases.

While these measures improved the software quality produced by catching problems as early as possible, they did not have much impact on the cost of change of a bug if it managed to proceed undetected. The problem was only half resolved.

3.1.6 Architecture and Design

Waterfall adopted a holistic approach to the solution architecture and software design of software products. 

A typical project would contain several new features to be implemented. Once the requirements are signed off, architects and software engineers would produce architectural plans and low-level designs for the entire set during the design stage and before implementation.

This method, a holistic top-to-bottom approach, is reliable and easy to manage because architects and engineers would:

• Be working side by side in a small team, and
• All interdependencies are tackled simultaneously.

This method is convenient and reliable but costly; any revision must go through the full feature set to determine any potential redesign effort.

To the latter’s detriment, this setup saw a trade-off between control and growth.

3.2 Agile

3.2.1 Accessibility

Agile brought improvements over Waterfall in many dimensions, and at its core is the concept of smaller, frequent releases. This modification alone requires quite some effort to explain to peers or management since, at first sight, its advantages are not evident.

On top of that, you have paradigm-shifting concepts such as the backlog and its management (called grooming), the cross-functional self-organising teams, effort estimation through story points, Kanban boards, and team velocities.

To make matters worse, several implementations of Agile (like Scrum and Kanban) exist, making the topic seem overwhelming to new joiners. It’s even more difficult for people to decide which Agile version best fits their teams.

Agile is neither intuitive nor cheap, making selling difficult, especially to senior management or teams set in old ways.

These attributes might explain why the top challenges cited in the 15th State of Agile Report (organisational culture, general resistance to change, insufficient leadership support, and inconsistent processes) remain unchanged year after year.

3.2.2 Team Structure and Maturity

Team structure in Agile displays a substantial deviation from its homogenous, static, and self-contained Waterfall counterpart.

Agile teams are supposed to be cross-functional, self-sufficient, self-organising, disciplined, and educated in the Agile way. They usually consist of several profiles like developers, testers, architects, and tech leads.

Agile models have also created new roles, such as Product Owners and Scrum Masters.

This considerable variety of skill sets and diversity of profiles and methods make Agile teams look very different from one organization to the next.

To work together efficiently, the team must be mature, disciplined, and fluent in the Agile methods.

These properties create a complex system of interactions and an atypical hierarchical structure that varies with each group. This ability to self-organize, however, allows teams to find a comfortable setup that best suits their skills, profiles, and expertise while still functioning efficiently.

3.2.3 Tools and Infrastructure

To appreciate the magnitude of change in tools supporting Agile methods, consider the Atlassian suite and how much it has changed in the last decade.

The synergy between tools like JIRA and BitBucket and the greater adoption of Agile was perfect; these applications made Agile easier to implement, while more Agile customers allowed Atlassian to invest and grow them beyond recognition.

Another equally influential set of tools is those that support build and test automation like Maven, Gradle, Jenkins, and Robot.

Without automating many repetitive, time-consuming, and non-value-adding tasks, the overhead that resulted from smaller and more frequent releases would have overtaken any advantages the latter brought.

Conversely, fluency in these tools is now a must for Agile practitioners, thus raising the bar for new joiners and steepening the learning curve.

4.2.4 Vertical and Horizontal Scalability

The challenge with scaling up Agile is a genuine one. As one project manager once told me, Agile tells you that your feature is in the backlog, but it cannot tell you how much it costs or when it can be delivered.

The reader might think this issue has more to do with poor stakeholder management (or effort estimation, velocity calculation) rather than Agile not being scalable.

The truth is that even if better stakeholder management processes were in place, project management could not work in Waterfall while the development uses Agile; they would constantly be out of sync.

Does Agile scale up quickly and comfortably?

The 15th State of Agile Report dedicated only one page to scaling up, which is not much. The report cites Scaled Agile Framework (SAFe®) as the most used method, with 37% of participants applying it within their organizations.

How successful and preponderant is SAFe? That remains an open question.

To efficiently collaborate, the customers, suppliers, project managers, senior managers, and developers must all be Agile. This requirement is much harder to achieve than it sounds since the reality is much more complex and messy than we would like it to be.

What about horizontal scaling?

Agile does not provide specific measures to treat this problem, and the same methods (workshops, meetings) for Waterfall continue to apply here.

Coordinating multiple project streams with interdependencies is a bit more challenging. This difficulty arises from release management in Agile (small frequent releases), where teams may run at different speeds. The team will be blocked if interdependencies exist until the dependency is resolved.

Reducing this interdependency may lead to siloing, another consequence we want to avoid.

4.2.5 Cost of Change

The cost of change in Agile is not as easily described as in Waterfall. In the latter, we stipulated that the closer the project’s final stages, the costlier the changes.

With the introduction of smaller and more frequent releases, less design time is now allowed (more on that in the next section) and significantly more automation, especially around build and quality assurance.

Does this mean that cost of change has gone down proportionately? The answer would have been ‘yes’, except the introduction of so much overhead has muddied the picture quite a bit.

Think of how much time and how many people are involved in writing and implementing a user story and whether these efforts are proportional to the size of the user story.

For example, the time it takes to run a full suite of unit tests does not heavily depend on the size of the change (unless your last user story has doubled the number of these tests!). Small changes, like big ones, have a cost.

What about the design and the short sprints? How has that impacted cost of change?

In the long run, the decentralization and downsizing of the design effort may lead to a sharp increase in architectural and technical debt and, therefore, the cost of change.

4.2.6 Architecture and Design

To manage two-week sprints that resulted in software being released into production, radical changes to the delivery process had to be made.

In addition to running automation in fifth gear, a sacrifice in software design effort (but not necessarily design quality) had to be made.

Something must be changed if the software can now be architected and designed within the development teams (and not in s special guild).

The first change was that architects and senior engineers were now integrated within Agile teams. The second change was abandoning large-scale solution designs and favouring small-scale, feature-level efforts.

Two questions arise:

1. How do we keep different teams working on the same codebase from implementing inconsistent software architecture and design?

1. How do we ensure that an optimum software design is produced if closely related features are not designed and implemented together because they happen to be separated by a few weeks in the backlog?

Cutting down on centralized design efforts has traded control for growth. On the other hand, the architectural and technical debt must be restrained.

This relaxed attitude toward design was counterbalanced by stricter rules and a higher skill level in software architecture. Also, technical debt management measures were instituted with the help of specialized software like SonarQube.

The emergent architecture was complex, arising from simple but strict rules applied iteratively with new features. This dynamic meant a bottom-up approach (think of complex systems like living organisms) instead of a top-down one.

So far, the result seems OK, especially since refactoring exercises (and other maintenance works) could now be budgeted at the management level but managed within an Agile team.

4.3 DevOps

The assessment of the DevOps model is an extension of Agile’s in that both have the same impediments, albeit by varying degrees.

DevOps has added continuous delivery to the original Agile outputs and has brought operations under its umbrella. This supplemental sophistication rendered the complications inherited from Agile more acute.

Despite getting the people who built the software to operate it makes much sense, DevOps is no more intuitive and accessible than Agile. The profiles and skill sets are richer and more complex, and a “DevOps engineer” role has emerged.

The number of tools and services they render has also increased to cover continuous deployment, cloud support, and operations, thus also raising the bar for new joiners.

Every new generation of software project management methodology must accomplish two things. Firstly, it must provide a substantial improvement over its predecessor, and secondly, it must also work with the latest technology, such as cloud deployment, for instance.

5. What Else?

External factors almost always play a role in the success or demise of a new idea. These factors can sometimes be entirely unrelated to the merits or limitations of the concept itself.

The notions behind Agile and DevOps are no different. In addition to their implicit merits and challenges, external, equally influential factors played a decisive role in speeding up or slowing down their adoption and, therefore, their utility.

These factors are organisational culture, process management and governance, and working from home.

• Organisational cultures can work with or against Agile adoption. To understand that, we can try answering the following questions. How does management value time? Do executives look towards the future or the past? Is the objective long-term growth or short-term profit? How is the risk of failure perceived? For instance, if your organisation focuses on short-term gain, you will have difficulty convincing them to invest heavily in Agile or DevOps. This attitude towards change is one of the reasons why Agile has not yet dominated.

• Process management and governance are vital for executing large-scale, high-risk, high-impact change. With inconsistent or poor production processes, it would be challenging to implement Agile with all the strict discipline it requires to succeed. Excellent process governance is another requirement that Agile adopters must satisfy.

• A hybrid working model is a tangible reality that will stick with us in the future. This new model, along with the highly problematic and unforeseen events of the last few years (as of 2022), has placed tremendous pressure on organisations. Not all have emerged victorious on the other end. Those who were resilient enough and could adapt more easily had higher chances of seeing it through. This strive for survival has pushed people to investigate new ways to improve should a new disaster hit.

5. Final Words

If this article were trying to deliver a message, it would read like this. Despite the obsolescence of the Waterfall model, Agile and DevOps are not silver bullets, and the question of which methodology is best suited for your situation is far from evident.

The recommendation, however, is to use Agile practices to achieve Operational Excellence.

We hope this article has shed some light on the potential issues that Agile and DevOps practitioners face and perhaps spurred the discussion on how best to resolve them.

6. Further Readings

• A talk on Clean Agile, Back to Basics — Robert (Uncle Bob) Martin.
• A lecture on Agile architecture — Allen Holub.