Waterfall Project Management: A Brief History and Introduction

1. Overview

After nine years of developing software packages for space mission programs, Dr Winston Royce decides to publish his “prejudices” in a presentation.

In this presentation, he describes his views on managing large software programs. The paper was duly published in 1970, more than five decades ago.

The paper is not only a joy to read but also contains insights stated with exceptional lucidity. It is not for those insights, however, that the work is famous; it is because it has (mistakenly) propelled Waterfall to the forefront of software project management.

Waterfall was the uncontested software project management framework until the 1990s; by then, serious drawbacks had already been identified, and candidates like Scrum (1995) and extreme programming (1996) were being tested.

The final blow was dealt in 2001 by seventeen software developers meeting at a resort in Utah, US. They published the Agile manifesto with twelve principles on how best software delivery can be achieved in the 21st century.

Despite its clear advantages, Agile had a very slow takeover, as the 15th State of Agile Report still shows. In effect, about 35% of the surveyed companies have less than two years of experience with Agile!

Waterfall lingering for over two decades after Agile’s inception is why these lines are being written now.

But that’s not all. To understand Agile (which is vital for any professional in the software business), we first need to understand Waterfall.

2. Waterfall: A Brief History

Despite the influence of Royce’s paper on Waterfall’s future, the concept of a sequential process for the delivery of large software projects predates the document itself.

In effect, Waterfall was first described in a presentation at the Symposium on Advanced Programming Methods for Digital Computers on 29 June 1956 by Herbert D. Benington (Wikipedia).

Finally, the term Waterfall was never used until it appeared in a 1976 paper by Bell and Thayer.

3. Analysis of Royce’s Paper

3.1 Waterfall: The Sequential Approach

Before we describe Waterfall, it is essential (and enlightening) to have some context on software development in the 1970s. These insights are gleaned from the paper mentioned above.

By that time, and after nine years of creating software for Lockheed Software Technology Center in Austin, Texas, Winston Royce has formulated the following ideas for managing the development of large software systems:

1. The success of a software project resides in “arriving at an operational state, on time, and within costs“.

2. There are only two steps in the software development process that the customer is happy to pay for: analysis and coding. Furthermore, these are the only two steps involving “genuinely creative work” where customer value is created.

3. Analysis and coding on their own are doomed to fail in large projects. Therefore, additional steps (requirements analysis, program design, and testing) are required. These steps create extra work, drive costs up, and add little customer value, making it hard for the management to sell them to the customer and enforce them on the engineers.

4. The following rationale dictates the sequence of the steps. In case of unforeseen difficulties, there is an “iteration with the preceding and succeeding steps but rarely with the more remote steps in the sequence”. This limitation on how far back you need to go makes the process more efficient, thus minimizing the cost of change.

5. However, as Royce grudgingly noted, the reality was that an error uncovered during testing requires at least a redesign and sometimes a rewrite of the specifications. Such was the state of software in the 1960-70s.

Despite the hard reality that Royce has experienced during his assignments, he still believed that the Waterfall approach was “fundamentally sound”, and with five additional features, development risk could be heavily reduced.

Let’s review those enhancements relevant to our present state of technology.

3.2 Waterfall: The Five Enhancements

The five enhancements proposed in Royce’ paper are as follows:

• Step 1: We insert a preliminary design phase between the Software Business Requirements and the Analysis phase. During this phase, you “[…] allocate processing, functions, design the database, define database processing, allocate execution time, define interfaces and processing modes with the operating system, describe input and output processing, and define preliminary operating procedures”. Inspecting these constraints with a lens from the current period shows us a high-level design and architectural work, an exercise of significant impact in large and complex software deliveries.

• Step 2: Ample, clear, and adequate documentation. Royce pinpoints with surgical precision the typical issues addressed with quality documentation:
  • It provides tangible evidence of completion (avoiding the “I am 90% finished syndrome”)
  • If documentation is poor, the design will be lacking, especially in the early phases of the project, where design, specs, and requirements documents are virtually the same.
  • Without good documentation, the software must be operated by those who built it.
  • When system improvements are required, good documentation permits effective redesign and updating.

• Step 3: A Proof-of-Concept (POC) is essential if the concept of operations, technology, or idea is pretty novel or innovative. A POC would help uncover vital flaws or information that can be leveraged during the implementation as early as possible.

• Step 4: Planning and executing a test strategy. “Without question, the biggest user of project resources, whether it be manpower, computer time, or management judgment, is the test phase.” Therefore, it is the riskiest in terms of budget and schedule overruns. Additionally, with the testing being the last phase of the implementation, you quickly run out of alternative options. To overcome these challenges, Royce suggests:
  • Proper documentation like test plans and test results
  • Test planning and execution by test experts not involved in the programming
  • Unit tests (although he does not use the name, he refers to testing every logical program path with “some kind of numerical check”)
  • Code review: Royce believes that visual inspection of the code can uncover small mistakes; there is no mention of style checking, clean code, or adherence to software design rules, though.

• Step 5: Involve the customer throughout the journey. Royce focuses on customer involvement to avoid misinterpretation of requirements, but in today’s software projects, the customer is involved in most project phases.

3.3 A Hint of Agile?

Many sources have hinted at Dr Royce’s paper as more of a critique of Waterfall (some even say it’s harsh criticism). I have not been able to detect that in the text. On the contrary, Royce believed the Waterfall approach “to be fundamentally sound”.

Other sources have cited a hint of Agile in the paper’s second half, where Royce visualizes Waterfall as an iterative process.

Upon closer inspection, the iterative element is not similar in spirit (at all!) to what Agile offers, especially with the smaller, frequent releases idea, which is entirely foreign from Waterfall and Royce’s paper.

In summary, Royce’s paper described the state of Waterfall at the time of writing (1970) and the potential enhancements to eliminate any risk of redesign and rework.

The ideas were not revolutionary. On the contrary, the paper made Waterfall a star for the next decades.

4. Waterfall Today

4.1 Stages of the Waterfall Model

4.1.1 Defining Waterfall

The Waterfall method breaks down all project tasks and activities into a pipeline of sequential stages where each stage depends on the output from its predecessor. Each pipeline stage specializes in one area of the software development lifecycle.

The figure below shows the stages and documents prepared for each stage.

The Waterfall model has not significantly changed since Royce published his paper. The same core stages, analysis and development, remain centre stage while requirement gathering, design, testing, deployment, and operations have become an integral part of the SDLC.

A complex production process with many stages, interdependencies, specialized skill requirements, and iterative subloops can only perform well if:

1. Efficient control processes are in place.
2. The team is motivated and works well together.
3. The procedures are updated whenever they become obsolete or inefficient.

The table below describes the seven stages and the documents produced in each of them.

Waterfall Project Phase	Deliverables
Project Inception	Project Initiation Document — defines the project sponsors, pre-requisites, and overall scope. Service Level Agreement (SLA) — sets the expectations between the supplier and the customer and the metrics by which the efficiency of the process is monitored and approved.
Requirement Gathering and Definition	Gap Analysis — conducted to determine the delta or missing features that must be implemented before the product meets the business requirements. Business Requirements Document — containing details of the business requirements to be met when the project is done.
Analysis	Software Orders of Magnitude — a list of high-level features and a rough order of magnitude of their efforts. High-Level Design — represents a bird’s eye view of what the solution looks like. Architectural decisions are made at this stage. High-Level Project Plan — cost and schedule are determined from the high-level estimates. Statement of Work — describes the scope, deliverables, timeframes, prerequisites and milestones. Test Strategy — a high-level plan of how the new releases will be tested. More of a guideline for testing than actual test cases at this stage. Discusses test approaches, test environments, and resourcing.
Design	Solution Design — contains all the low-level technical decisions on the product.
Development	Source Code — the new version of the source code. This includes code review, pull requests, and other artifacts produced in the development. Test Scripts — the set of test cases to test the changes. These also include any Test Automation or unit test scripts in case Test-Driven Development is used. User Guides and Reference Manuals — can also be produced at this stage.
Testing and QA	Test Cases — an actual test case with summaries, acceptance criteria, steps to produce, and test parameters. Test Report — a test run report with details on failed cases.
Deployment	Run Sheets — a step-by-step plan on how production deployment will happen. Typically with a duration for each step. This is especially important if downtime is involved. Release Notes — contains a summary of the new features of this release.
Operations	Installation, Admin, and Operations Guides — the manuals required to run the system by people other than those who built it.

4.2 Iterative Sub-loops Inside Waterfall

In the above diagram, we can see four subloops that imply an iterative process. These subloops allow the following issues to be addressed:

• Analysis -> Requirements — If any inconsistency or contradiction in the business requirements is uncovered during the analysis phase, a reassessment and modification of the requirements might be needed.

• Design -> Requirements — During the design phase, various design decisions will need to be made, and further input from the business stakeholders will be required.

• Testing -> Analysis — we assume that at this stage, any bugs uncovered in the testing are a result of some misalignment between implementation and specifications. In this case, either the specs are incorrect, or the implementation is faulty.

• Operations -> Analysis — issues uncovered in production could not have been anticipated during the software testing phase or some corner use cases.

4.3 Fundamental Principles of Waterfall

The Waterfall model rests on the following pillars:

• The core activities in a development cycle where business value has created the lie in the Analysis and Development stages. The rest is necessary because executing only those two stages could not achieve satisfactory outcomes. Aside from Analysis and Development, all project steps must be minimized in terms of cost and schedule to retain a competitive advantage.

• We do not have to go beyond the previous stage or two in the Waterfall model to fix mistakes when they happen. Unfortunately, this is not the reality; sometimes, we must return to Analysis to determine what happened.

• Massive design and documentation efforts are typically spent on Waterfall projects. A running argument that is now common sense is that the design-to-development ratio must be around 70/30. It is not evident whether tinkering your way through a challenging problem is inferior to abstract intellectual analysis. In effect, the technical profiles are not the same (and therefore do not have identical costs); a proof of concept can be made by a developer where a tricky design problem might require people with more experience. Also not evident is whether it is faster and cheaper to iterate through a design or try to get it right in the first place.

5. Final Words

We have presented in this article an overview and brief history of the Waterfall model.

The objective was not to offer it as a viable option in today’s world of large software project management tools but to:

• A) appreciate the conditions under which it was born, and 
• B) help us understand its limitations when we discuss Agile.

You can find it here if you are looking for a comparative analysis of all three models and their challenges and limitations.

Finally, if you are still using Waterfall and would like to migrate to Agile or DevOps, we have some guidelines to help you.

6. Further Readings

• Great talk by Robert (Uncle Bob) Martin.
• An extreme form of Agile in the intro to this lecture by Allen Holub.