When introducing Agile, many coaches would postulate that the methodology had existed for many long years (even before software programming was invented) disguised as an iterative, exploratory method for building complicated things.
Reference is often made to the second half of Dr Winston Royce’s famous paper on “Managing the Development of Large Software Systems” to support this hypothesis.
Agile and its various implementations have undergone much refinement and enhancement over the last two decades. It’s pretty hard to believe that it resembles anything from the past, primarily when these implementations were heavily geared towards software environments.
The bottom line is that Agile is not as intuitive and self-explanatory as we often think it is, and some mental effort is required to appreciate its advantages and challenges.
This article will decompose the complex Agile framework created in 2001 into its fundamental principles and first concepts.
The objective of this decomposition is to present the reader with a description of Agile that is easy to digest and implement should they find it suitable to do so. This is particularly important since applying Agile practices is one of the six principles we advocate for Operational Excellence.
2. Table of Contents
- 1. Overview
- 2. Table of Contents
- 3. A Brief History of Agile
- 4. The Limitations of Waterfall
- 5. Fundamental Principles of Agile
- 6. Agile Delivery Process
- 7. Final Words
- 8. Further Reading
- 9. Featured Articles
3. A Brief History of Agile
In the 1980s and 90s, a series of lightweight software development methodologies were invented to overcome the challenges posed by the rigid rules that governed Waterfall.
What prompted these changes was the radically different nature of software development that has emerged since its inception in the 1950s. Below is a list of these changes:
- Technology has significantly matured and advanced both in hardware and software. Software development became a discipline requiring specialization and formal education.
- The software produced to satisfy advanced client preferences was much more complex and diverse. From these refined preferences, complex business requirements emerged.
- Large software teams (rather than isolated software experts working solo) had become the norm. The efforts of these complex groups had to be coordinated and stakeholders managed.
Seventeen software developers met in 2001 in Snowbird, Utah, to discuss the lightweight development methods used then:
- Kent Beck, Ron Jeffries, and Ward Cunningham — Extreme Programming
- Dave Thomas, Andrew Hunt and James Grenning — PragProg, Ruby
- Jeff Sutherland, Mike Beedle, and Ken Schwaber — Scrum
- Jim Highsmith — Adaptive Software Development
- Alistair Cockburn — Crystal
- Robert C. Martin — SOLID
- Arie van Bennekum
- Martin Fowler — OOAD and UML
- Jon Kern
- Brian Marick — Ruby, TDD
- Steve Mellor — OOA.
Together they published the Manifesto for Agile Software Development. The manifesto postulated 12 principles defining the Agile framework.
4. The Limitations of Waterfall
The following conditions heavily constrain the Waterfall project delivery methodology:
- Business requirements must be well-defined at the very early stages of the project so that the sequential nature of progression in Waterfall can work. Otherwise, Waterfall will struggle and fail. Waterfall’s primary drawback is its uncompromising attitude toward changing business requirements and lack of iterative processes in any of its stages.
- Sizier projects were at a significantly higher risk of failure primarily due to the lengthy duration of the implementation and the increasing amount of expected delays in deliveries.
- An overly protracted implementation increases the pressure for unlocking requirements and scope, two sufficient conditions for Waterfall to fail.
- Also, expected project delays follow power laws rather than normal distributions; the bigger the project, the higher the ratio of anticipated delay vis-a-vis project duration.
- Other, perhaps equally influential consequences of lengthy projects and changing requirements in Waterfall-run projects were:
- Longer time-to-market
- Late detection of the divergence of customer needs from implemented features
- Irreducible and high cost-of-change
5. Fundamental Principles of Agile
5.1 Smaller, More Frequent Releases Are Better
Consider a project where you have to build ten new features. The implementation effort, from design to release, is estimated at 12 months.
Using a Waterfall approach and assuming everything works out as planned, you will have those ten new features released after 12 months.
Now consider an alternative approach where a new feature is released and deployed to production as soon as it’s completed. Assuming a feature requires, on average, one month, you will have one feature in production after one month, two features in two months, and so on.
A fair comparison between the two methods, Agile and Waterfall, in this case, would consist of counting the features in production at any one time multiplied by how long it has been released.
A simple arithmetic calculation would reveal Agile as the uncontested winner as the value delivered over time was constantly higher when compared at each point on the graph above.
Two caveats, however, exist in the Agile approach. We tried to capture the first of those in the above graph by showing the project sign-off in the Agile case lagging a bit behind the Waterfall one; chopping big releases into smaller ones will necessarily introduce some overhead.
This overhead, however, will not outweigh the benefits accumulated from the early release of new features.
The second caveat is that not all features can be isolated and independently released, and a more realistic scenario will include features of various sizes and interdependency. This complication renders the picture less contrasty in real-world situations.
Despite those two limitations, some degree of agility can (and must) be injected into any project.
5.2 Iterations over Extensive Planning and Design
What would you do if you had to work with unclear or incomplete business requirements?
Waterfall attempts (rather unsuccessfully) to overcome inherent issues in developing complex software projects by getting stakeholders to sign their names against every artifact before proceeding to the project’s next stage. The result is much fingerpointing when things go wrong (as they often do) and, more importantly, software products with poor customer value.
Agile proposes a different approach altogether. Instead of a locked and fixed set of requirements implemented and released in one batch, Agile product owners maintain a backlog of user stories prioritized (ideally) by customer value.
One of the primary duties of an Agile product owner is backlog grooming which essentially consists of reordering the backlog so that the user stories with the slightest uncertainty and highest customer value are delivered first.
As the project progresses and more information is gathered from features in production, uncertainty is driven down, and user stories previously on hold due to incomplete requirements are updated and pushed forward.
5.3 Emphasizing the Human Element in Software Projects
Four of the twelve principles published in the Agile manifesto emphasise the role of the human element in software development:
Business people and developers must work together daily throughout the project.
Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.
The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.
The best architectures, requirements, and designs emerge from self-organising teams.
The following ideas transpire:
- The customer (and all principal stakeholders) must be involved in the journey. The principle sets the interaction frequency to daily to underscore its importance. This new requirement is a significant departure from Waterfall-based approaches where the customer may not see the product before UAT (where it’s already too late).
- Employee motivation and empowerment are ever-present themes in leadership and organisational seminars. Still, the authors of Agile went the extra mile and elucidated them at the heart of the twelve principles. Employee motivation and empowerment are recognised as significant influencers in the team’s performance and should be part of any Agile implementation.
- Face-to-face discussions are prefered over documentation as they add an extra one or two dimensions to the conversation channels. Facial cues help communicate the interlocutor’s feelings about a specific idea, while kindness, empathy, and good overall communication skills increase the effectiveness and efficiency of a debate.
- The relevance of self-organisation in teams and its relationship to architecture is a more subtle concept that is not easy to come across in our day-to-day experience. Self-organisation is a property of complex systems, and human groups are profoundly complex. Self-organisation manifests, for example, through the specialisation of team members in specific areas of the product or by honing specific skills. The relationship, however, that the authors envisaged between this complexity and software architecture is not evident.
6. Agile Delivery Process
Viewed from an altitude of 30,000 feet, the Agile process is identical to Waterfall with its typical five stages of Define, Build, Test, Deploy, and Monitor, with which we are all familiar. These five stages constitute the familiar Software Development Lifecycle (SDLC).
From the ground, however, the processes cannot be more different. Four key players, engaged in a continuous dialogue, run the following cycles:
- The business analysts define the requirements, approve the changes demoed during showcases, and might provide input and feedback during the design phase.
- The product owners have a central and vital role: story writing, feature prioritization, grooming the backlog, and managing the stakeholders.
- The developers must build the features, ensure adherence to specifications, manage technical debt, showcase the changes, and maintain the infrastructure.
- Finally, testers ensure that requirements have been met and business problems solved.
The above picture can be extended to include release management, production support, and production deployment, in which case we obtain a DevOps model.
In the following sections, we will look closely at some of the novel concepts that Agile introduced and glean some insights and best practices.
6.1 User Story Writing
A User Story consists of a logical unit of business requirements viewed from the perspective of the product’s end user. There are several templates in use along the lines of:
As a <role> I can <functionality>, so that I get <business value>.
User Stories are grouped in Epics, which can be regrouped into larger units such as Initiatives.
A critical component of a User Story (and indeed any requirement) is Acceptance Criteria which is nothing but a list of statements in the form of:
The product [shall, should] support this or that <functionality>.
User Stories typically follow a standard workflow and may be divided into smaller tasks and subtasks. Their progression is controlled by specific rules called Definition-of-Ready (or DoR) and Definition-of-Done (or DoD).
The setup of a user story template, workflow, DoD, DoR, and subtask format is implementation-specific and is acceptable as long as it doesn’t contradict any of the Agile principles.
The advantage of using User Stories over something else is that:
- It provides a semi-standard process for capturing business requirements in a format with which developers and testers are now familiar.
- It emphasizes the end user’s role in designing new features that solve a specific business need.
6.2 Backlog Grooming
A backlog is an ordered list of user stories, and the top-priority items on that list are ready to be picked up for development.
Backlog grooming is a continuous process that product owners apply and consists of the following activities:
- Updating priorities of user stories
- Adding new work in the form of initiatives, epics, or user stories
- Breaking up big user stories into smaller ones
- Closing user stories that will never be implemented
The flexibility of a backlog format for storing and managing features over rigid documents is readily seen; the backlog can be:
- Shared with the stakeholders
- Used as a baseline for discussions
- Regularly updated based on new input or changing project priorities.
A release can be defined as a subset of items currently in the backlog that are either actively developed or to be completed before the next production build.
6.3 Stakeholder Management
Stakeholder management in this context refers to a product owner’s ability to answer questions about the scope, schedule, or both.
Examples of such questions can be:
- When can we expect to have feature A deployed?
- Can we have feature A deployed before DD-MM-YYYY?
- Which features can be ready by DD-MM-YYYY?
To answer these questions, a product owner will refer to the story point estimation of each user story and how many story points a team can complete within a fixed amount of time, like a week or two, a metric that is sometimes referred to as velocity).
In summary, a story point is a numeric label developers assign to user stories to indicate their implementation difficulty. story points are finer at small scales and grow coarser with the scale. The Fibonacci sequence is typically used:
Story point set: 1, 2, 3, 5, 8, 13, 21, 34...
story points of remaining work and team velocity can be placed on a chart to predict when the project will be completed.
6.4 Involving the Customer and End User
Customer and end-user involvement are accomplished by:
- Sharing access to user stories with business analysts on the customer side for feedback and validation
- Showcasing new features in production-like environments
- Early deployment of new features and monitoring usage KPIs
In general, whether it’s customers, end-users, or any other stakeholder, the best results can be achieved only through open communication channels, correct identification of key stakeholders, and prioritization of business value over technology or the number of new features released.
6.5 Managing Technical Debt
Agile is unique in raising awareness (not just at the technical level but also at the business and project management ones) on the risk of sacrificing code quality. It’s also brought the sustainability and technical excellence topics to the forefront of the discussion.
If you want to be more agile, you must maintain a constant pace indefinitely, and the primary condition for that to happen is keeping your code in good shape.
6.6 Reflection and Retrospectives
Agile requires practitioners to reflect regularly on how to become more effective. This strive for continuous improvement is implemented via retrospectives.
Retrospectives work by getting the team in a room every 4-6 weeks and retrospecting on the last period. The person running the retrospective writes down what has and hasn’t worked and, through collaboration with the team, determines what actions can be done to avoid experiencing the issues in the future.
Retrospectives and similar rituals require a high degree of discipline so that actions are followed-up and diligently executed. The team and organisational culture should also be mature enough to conduct the, sometimes painful, pinpointing of mistakes without assigning blame and finger-pointing.
6.7 Frequent Releases
Before we start this discussion, a short disclaimer is due. In effect, the notion of release frequency is not as evident as it may seem; let’s see why.
- Scenario 1 — Your company develops printer drivers. How much additional customer value would doing fifty instead of five yearly releases bring? Probably not much.
- Scenario 2 — If company A makes five yearly releases. Assuming that’s all we know about company A, can we decide if it qualifies as Agile? The answer is not apparent, and we would need some context on the product, the industry, and what releases would typically look like under a Waterfall regime.
Setting aside the question of what constitutes a good frequency level, we will now list some preconditions without which “frequent” releases are not possible in any case:
- The presence of a comprehensive battery of unit and integration (SIT) tests with decent code coverage
- Automation of build and test environments as a minimum, with build and test cycles happening on (ideally) every feature branch
- A development process where the Definition of Done of a user story includes a minimum number of new unit and SIT test cases
6.8 Team Structure and Self-Organization
Although the authors of the Agile manifesto cited self-organization in relation to software architecture, many people refer to it in the context of team structure. So let us talk about that a little bit.
self-organization can manifest along three axes:
- Specialization — A freely evolving team would self-organize so that each member would become an expert in their domain. A form of specialization would emerge built upon a combination of rare and unique skills members initially possess or gradually acquire.
- Roles — Familiarity with specific tasks, natural talent, expertise, and an amount of chance will dictate the self-organization of roles.
- Hierarchy formation — A complex hierarchy and a matrix of power relations will surface once seniority, extended networks, leadership skills, and character drive the team’s political self-organization.
However, the results of unguided self-organization are not always desirable, at least in theory.
For example, the freely evolving scenarios described above can lead to single points of failure, the concentration of too much power in the hands of too few people, and the building up of bad habits.
While imposing a rigid and inadequate structure on a team is not ideal, self-organization can only work if it’s guided by professional and expert leadership within a specific framework. The latter would include:
- Time-tested, mature, stable, and well-understood practices such as the six principles of operational excellence we advocate
- A vision for the organization that provides a sense of direction at all levels, from top management to new joiners
- A healthy organizational culture
In general, the recipe for a performing Agile development team is well-known: a broad array of cross-functional skills, members who have previously worked together, discipline, and a thorough understanding of Agile.
7. Final Words
The first time I heard of Agile was about a decade ago, and I was entirely sceptical about its performance. Being a thorough believer in the no-free-lunch paradigm, I was convinced that a trade-off was made somewhere and that Agile promoters avoided discussing it.
It took me a bit less than a decade to appreciate Agile’s advantages (and challenges), and this only happened after I had become part of a fully Agile team for the first time in my career.
Although Agile is far superior to Waterfall in every aspect, it is not a silver bullet. Cracks in the Agile edifice can become immediately evident as project size steps beyond certain thresholds or whenever Agile is applied to specific types of projects.