Agile vs Waterfall: Finding a Methodology That Works Best for You

1. Overview

Implementing Agile practices is one of the six principles of Operational Excellence we advocate. That, however, would not be without challenges for various reasons.

It suffices to read the 15th State of Agile Report on Agile’s adoption to appreciate some of these blockers, such as “cultural clashes” or “inconsistent processes”.

This article will pull some of these blockers apart to give the reader enough information to assess their situation appropriately and decide which model to adopt.

Our candidate models are the Hybrid model, Agile, and DevOps. We consider Waterfall inadequate to deal with large software projects’ complexities.

3. Approach

The approach we will be using is as follows. We will look at responses that participants in the survey have reported as Agile adoption blockers. We supplement that with our field experience.

Then, we will examine these individually and formulate them as requirements you either satisfy or don’t.

The result would be a scorecard with a few different levels mapped to one of the candidate models above, as well as the level of difficulty of implementing them.

The scorecard may look like this.

Points	Candidate	Implementation
1	Hybrid	Easy
2	Hybrid	Medium
3	Hybrid	Difficult
4	Agile	Easy
5	Agile	Medium
6	Agile	Difficult
7	DevOps	Easy
8	DevOps	Medium
9	DevOps	Difficult

The parameters that we will use in this scorecard are as follows:

• Business Model
• Organisational Culture
  • Resistance to Change
  • Attitude to Failure
  • Risk Appetite
  • Leadership Support
• Time Management
  • Short-term Profit vs Long-term Growth
  • Fire-fighting or Fire Prevention
  • Past, Present, or Future
• Current Software Delivery Model
  • Current Delivery Model
  • Team Structure
  • Test Coverage and Quality
  • Automation
• State of Software
  • Legacy Code and Technical Debt
• Processes
  • Process Examination
  • Process Management and Governance

we examine each criterion in the following sections

4. Business Model

You can safely use Waterfall if one of the below criteria is satisfied:

• Your business model is stable and unlikely to change — In this scenario, new ways of conducting business are very scarce, and your model is immune to environmental changes.
• There is no uncertainty in the business requirements — When the client’s needs are apparent and there is no need for exploration.
• When the ecosystem is orderly — Such an environment does not include any complex system, and client preferences do not change with the introduction of new technologies or natural disasters.

In most of today’s software industry, the above criteria are unlikely to be satisfied. This leads us to consider Agile (or DevOps), and therefore, more conditions we explore next.

5. Organisational Culture

5.1 Resistance to Change

We sometimes confuse resistance to change with laziness, but that’s not always the case; we must look at culture.

Organisational culture serves a purpose, and the beneficiaries of those services would want to keep things unchanged for as long as possible.

From the viewpoint of the beneficiaries, the current assumptions that underlie the culture were formulated in a traumatic learning experience that they are not eager to repeat.

But organisational culture alone cannot explain resistance to change. One more missing component is the potential change in the power distribution among stakeholders.

In such cases, conflicts of interest between the current power brokers and the business would arise, and they might obstruct changes with all their might.

Before proceeding with large-scale changes, you must analyze your stakeholder’s positions and learn how to deal with cultural blockers.

5.2 Attitude to Failure

In some cultures, failure can bring a lot of embarrassment to its owner; therefore, people would avoid it by sticking to the old ways—the more significant the change, the bigger the failure, and the more resistance they will demonstrate.

It is enlightening to know that most technological revolutions (like the steam engine) resulted from many tinkering and failed experiments (Antifragile – Taleb). Theoretical knowledge only came later to explain what was happening.

Returning to the Agile adoption issue, many variations of Agile exist, and Agile practices can be applied to varying degrees.

Every team will be different, and there is no shame in trying things out before settling on a customized method that works for you.

5.3 Risk Appetite

Any change is risky in that it will bring about some disruption to the existing order. In complex systems, the results of this disturbance are usually unpredictable; therefore, people are naturally more cautious around them.

Also, some people are risk-takers, and risk appetite can vary between individuals. For this reason, it is crucial to identify the key decision-makers in your plan.

Decisions can also be made on calculated risks, as the latter can never be eliminated. Risk, however, can be mitigated through contingency planning.

5.4 Leadership Support

In addition to performing adequate stakeholder management to understand the positive and negative forces affecting your plan, it is also vital to have leadership support for the following reasons.

First, implementing Agile, especially starting from a classic Waterfall model with poor processes and weak infrastructure, will require a significant investment of time and resources; this investment must be budgeted accordingly.

Secondly, in such a long-term plan, the results will not be noticeable immediately, and many hurdles must be overcome before you start seeing any returns on your investment.

For this reason, management sponsorship and commitment to long-term strategies are essential.

6. Time Management

When discussing time management issues, we almost always think of to-do lists and how best to prioritise competing tasks. That view does not do justice to the topic.

On the other hand, the time management discussion has cultural significance and will undoubtedly impact your transformation plans and long-term strategy. Let’s see how.

6.1 Short-term Profit vs Long-term Growth

A company’s long-term plans and strategy dictate how much focus and investment will be poured into areas like quality, infrastructure, and R&D. Time is scarce, and time management is knowing where to invest it.

You might hear responses such as “now is not the time” or “let’s focus on finalizing this release”. These responses show how far or short the leadership looks into the future.

Agile is a long-term investment, especially if you start late. The barrier might be too high, and the ROI might not be soon. Therefore, implementing Agile practices must align with the organization’s vision.

For instance, a startup might focus more on getting an MVP out than refining its production processes. Once the product is mature, the company might be acquired by a larger one, and Agile might already be in place.

6.2 Fire-fighting or Fire Prevention

Fire-fighting or fire prevention is another trade-off at the core of time management concerns.

Do you invest time implementing Agile practices at the risk of delaying projects? Conversely, your future deliveries will have better quality, and your production processes will be more Agile. Or do you leave it to the next manager to worry about it?

Both cases involve certain risks and benefits. Here it depends on the personalities involved, the organisational culture, as well as the long-term vision of the company.

In both cases, you need to know where your leadership stands before proceeding with your delivery process migration.

6.3 Past, Present, or Future

Where does your collective consciousness as a group or organisation live? Is it the past, present, or future?

Edgar Schein, one of the leading experts on organisational culture, discusses this topic in his book Organisational Culture and Leadership.

A company’s outlook can severely impact its options. Nostalgic people constantly lamenting a bygone golden age will not find the motivation to persevere for a future vision.

People living under the tyranny of the moment are too occupied with the present to formulate any long-term strategies.

This analysis applies to lengthy projects, including a cultural/intellectual transformation like Agile.

7. Current Software Delivery Mode

The essence of this section and the next is as follows: your starting point will determine how much effort you need to invest to reach your objective and, therefore, the height of the barrier you need to overcome.

7.1 Current Delivery Model

Agile is a multi-dimensional spectrum of software delivery methodologies. Scrum, Kanban, and Extreme Programming all fall under the Agile nomenclature of lightweight models.

You can even concoct a Hybrid model, for instance, 60% Waterfall and 40% Agile, using all elements of Agile like build and test automation, test-driven development, and continuous improvement (via retrospectives) but not frequent releases.

The difference between how agile you are today and where your target operating model lies will determine how much persuasion you need to use to get people on board. It will also determine the duration and cost of the migration.

7.2 Team Structure

You cannot do Agile if you organize your teams in silos since Agile requires fast and quality decision-making methods.

If you already have a siloed team structure with people set in their ways, reorganising them into cross-functional teams might be challenging.

Issues of seniority, position, privileges, knowledge management, and authority might surface and must be dealt with. Equally important, they must be convinced and committed for the transformation to succeed.

7.3 Test Coverage and Quality

Do you have a decent battery of test cases with adequate coverage? Would you be confident enough to push a release into production if it successfully passes all these tests?

If not, then we have a challenge. A natural follow-up question is how much it would take us to have such a decent suite ready. The correct answer is “it depends”.

A properly-architected software should be comparatively easy to test. We talk in relative and not absolute terms because we must consider maturity, size, and the number of features we want to include in our testing scope.

Without robust and automated testing, the case for Agile is lost.

7.4 Automation

Implementing automation tools does not sound as problematic as transforming organisational culture or creating a full test suite for a legacy product.

However, you still need to hire automation experts, agree on the best tools, implement them with the support infrastructure, adjust your production processes to accommodate them, and train the team on how best to use them.

This investment needs a budget, milestones, deadlines, and a project manager. It’s a serious effort.

8. State of Software

8.1 Legacy Code and Technical Debt

Your software’s current state is key to a successful Agile transformation. In this case, you want your software to work with you and not against you.

Legacy code is usually fragile, poorly documented, has little unit testing, is ridden with technical debt, and lacks in all aspects of robust and reliable software architecture and design.

In legacy software, hyper care is needed any time you make a change, even the most minimal, making frequent releases almost impossible. This limitation is magnified with mission-critical software, where mistakes are very costly.

Legacy code is one of the severe obstacles to Agile migration plans;
in such cases, it might be safer, easier, and cheaper to stay with Waterfall or a Hybrid model

9. Processes

9.1 Process Examination, Management, and Governance

Agile philosophy is implemented with Scrum, Kanban, Extreme Programming and similar methodologies.

Numerous variations of each methodology can exist. Each team will have a unique implementation that governs how they groom their backlog, write user stories, organize their teams, and conduct continuous improvement sessions.

Process owners with specific skill sets and Agile know-how must examine and monitor this dynamic and evolving state of production processes were it to succeed.

Not many organizations have sophisticated process management, governance, and improvement processes, and investment in creating such a body of people and skills is another hurdle that needs to be overcome.

10. Final Words

Hopefully, the reader is not discouraged after looking at the long list of criteria they must satisfy before successfully transforming their organisation to Agile.

The reality is even more complex than that. Introducing Agile into development teams is not enough. It only works if both development and management are Agile, which is not easy as Agile seems to become less inaccessible the higher up the chain one goes.

But, as we have mentioned, you don’t need to be fully Agile to reap its benefits and achieve Operational Excellence; in effect, only a small percentage of organisations is fully Agile, as the 15th State of Agile Report tells us.

It is up to every organisation to decide whether incremental improvements are enough or a total transformation is required.