Agile Software Delivery — First Principles and Foundational Elements

1. Overview

When introducing Agile, many coaches would postulate that the methodology had existed for many long years (even before software programming was invented) disguised as an iterative, exploratory method for building complex things.

Reference is often made to the second half of Dr Winston Royce’s famous paper on “Managing the Development of Large Software Systems” to support this hypothesis.

Royce provided various ideas on how Waterfall could be modified to mitigate the delivery risks it presented. One of those ideas was prototyping which consists of creating a simplified model of the actual software. The prototype will be tested in a production-like environment so that any design flaws will surface immediately. For prototyping to succeed, Royce believed that engineers must possess the following skills.

Agile and its various implementations (Scrum, Kanban, Lean) have benefited from much refinement and enhancement while accumulating unnecessary rituals and “best practices” over the last two decades. It’s hard to believe that Agile today resembles anything from the past.

Perhaps the most common issue with Agile is not as intuitive and self-explanatory as we often think. Some mental effort is required to appreciate its advantages and challenges, which will be the present article’s objective.

This article will decompose the complex Agile framework created in 2001 into its fundamental principles and first concepts.

This decomposition aims to present the reader with a description of Agile that is easy to digest and implement should they find it suitable. This is particularly important since applying Agile practices is one of the six principles we advocate for Operational Excellence.

Related —

• Complexity in Natural and Human Systems — Why and When We Should Care

2. A Brief History of Agile

The Agile movement was started in 2001 when a group of software developers gathered in Utah to discuss new ways to approach software development. This meeting, known as the Agile Manifesto, created a set of principles and values to shape software development.

The historical events that led to the creation of Agile can be traced back to the 1990s when software development was characterized by the waterfall methodology. This approach involved a linear planning, design, development, testing, and deployment process. The problem with this approach was that it often resulted in lengthy development cycles, cost overruns, and failed projects. This led to a search for new and better ways to approach software development.

The main contributors to Agile are a group of 17 software developers who met in Utah in 2001. This group, which included Kent Beck, Martin Fowler, and Jeff Sutherland, came together to discuss new ways to approach software development. They created the Agile Manifesto, which outlined principles and values guiding Agile development.

When Agile was announced, it was formulated as a manifesto, which outlined four values and 12 principles that would guide Agile development. The four values were: individuals and interactions over processes and tools, working software over comprehensive documentation, customer collaboration over contract negotiation, and responding to change over following a plan. The 12 principles included delivering working software frequently, working closely with customers, and promoting sustainable development.

Initially, Agile was received with scepticism by many in the software development community. Some saw it as a fad or a buzzword, while others were concerned that it lacked the rigour and structure of traditional methodologies. However, Agile has gained widespread acceptance over time as a highly effective approach to software development.

Today, Agile is widely used by organizations of all sizes and has been credited with improving software quality, reducing costs, and speeding up time to market. While some still prefer traditional methodologies, Agile has become the de facto standard for software development in many industries.

Related —

• Waterfall, Agile, and DevOps: A Critique of Current Challenges

3. Foundational Elements of Agile Software Delivery

Several foundational elements of Agile software delivery help ensure successful implementation. These include:

3.1 Agile Teams

The software development team should consist of motivated individuals committed to the project’s success. The team should also be cross-functional, with members with different skills and areas of expertise.

3.2 Backlog

Backlog – The backlog is a prioritized list of features and requirements that must be developed. The team should work from the top of the backlog, with the most important features developed first.

3.3 Sprints

Sprints are short periods (usually two weeks) during which the team develops features. Sprints allow the team to focus on specific features and ensure the product is developed iteratively and incrementally.

3.4 Product Owner

The product owner prioritises the backlog and ensures the team develops the most important features. The product owner should also work closely with the team to ensure that the product meets the customer’s needs.

3.5 Scrum Master

The Scrum Master is responsible for facilitating the team’s development process and ensuring that the team adheres to Agile principles. The Scrum Master should also help remove any impediments the team may encounter.

Related —

• Self-Organisation in Agile Teams — How and When Does It Work

4. The Limitations of Waterfall

Waterfall project delivery methodology can be heavily constrained by several factors, as evidenced by studies in project management.

• The primary constraint is that business requirements must be well-defined at the outset of the project for the sequential nature of progression in Waterfall to work effectively (Boehm and Turner, 2003).

If the requirements are not well-defined, Waterfall will struggle and may ultimately fail. The lack of iterative processes in any of its stages is another drawback of Waterfall, which makes it less accommodating towards changing business requirements.

• Large projects are particularly vulnerable to failure when using the Waterfall methodology because of their prolonged implementation period and the associated delivery delays (Mckenna and Copeland, 2017).

As the duration of a project extends, there is increased pressure to unlock requirements and scope, which can also cause Waterfall to fail.

• Furthermore, project delays tend to follow power laws rather than normal distributions, meaning that the larger the project, the higher the expected delay ratio to project duration (Candea et al., 2017).

• The drawbacks of lengthy projects and changing requirements in Waterfall-run projects can lead to longer time-to-market, late detection of divergence in customer needs from implemented features, and an irreducible and high cost of change (Browning, 2016).

These challenges highlight the importance of carefully considering the Waterfall methodology’s suitability for a particular project and the need to weigh its advantages and disadvantages.

To mitigate the risks associated with the Waterfall methodology, project managers can consider using other approaches emphasising flexibility and iterative processes, such as Agile or Scrum.

By selecting the right methodology, project managers can minimize the likelihood of project failure and enhance their chances of delivering a successful project.

4. Basic Principles of Agile

4.1 The Agile Manifesto

Agile software delivery is a methodology that emphasizes collaboration, flexibility, and continuous improvement in software development. It is an iterative approach that values customer satisfaction, team collaboration, and rapid response to change. The Agile Manifesto outlines the core values of Agile software development, which are:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change by following a plan

The first principle of Agile software delivery is to deliver working software frequently. The software development team should focus on delivering a functional product quickly and consistently. The team should also be open to feedback and ready to adjust the product based on user feedback.

Agile software delivery’s second principle is collaborating with customers and stakeholders. The software development team should work closely with the customer and other stakeholders to ensure the product meets their needs. The team should also be open to feedback and ready to adjust the product based on user feedback.

Agile software delivery’s third principle is building projects around motivated individuals. This means the software development team should consist of motivated and skilled individuals committed to the project’s success.

The fourth principle of Agile software delivery is to respond to change. This means the software development team should be flexible and responsive to changing requirements, even if they occur late in the project’s development.

4.2 Smaller, More Frequent Releases Are Better

Imagine you’re working on a project where you need to build ten new features, which you estimate will take 12 months to complete. You have two options: the traditional Waterfall approach or the more modern Agile approach.

With Waterfall, you’ll have to wait until all ten features are completed before releasing them to the market. Assuming everything goes as planned, you’ll have all ten features released in 12 months. But what if there’s a delay in the project? Or a requirement change? Waterfall’s rigid, linear approach makes it difficult to accommodate changes mid-project.

In contrast, with Agile, you can release each new feature as soon as it’s completed. So, assuming each feature takes an average of one month to complete, you’ll have one feature in production after one month, two features in two months, and so on. You can deliver business value continuously rather than waiting for the end of the project.

A fair comparison between the two approaches would be to count the number of features in production at any given time and multiply it by how long they have been released. By doing so, we can see that Agile outperforms Waterfall regarding business value delivered over time.

Of course, there are caveats to the Agile approach. First, as seen in the above graph, the project sign-off in the Agile case lags behind the Waterfall one. This is because breaking big releases into smaller ones will introduce some overhead. However, the benefits of early feature releases outweigh this overhead.

The second caveat is that not all features can be isolated and independently released. Many features have interdependencies and require coordination to deliver. In real-world scenarios, this can make the picture less clear-cut.

Despite these limitations, it’s crucial to inject agility into any project. By embracing Agile, you can be more responsive to changes and deliver business value continuously, rather than waiting until the end of a project.

4.3 Iterations over Extensive Planning and Design

In software development, unclear or incomplete business requirements can be a major hurdle.

Traditional Waterfall methodologies attempt to overcome this issue by requiring stakeholders to sign off on every project artifact before proceeding to the next stage. However, this approach often leads to finger-pointing and a final product with poor customer value.

Agile, on the other hand, proposes a different approach. Instead of a fixed set of requirements, Agile product owners maintain a prioritized backlog of user stories. This backlog is constantly updated and re-ordered based on customer value and uncertainty around each user story.

The primary duty of an Agile product owner is backlog management, which involves constantly updating the backlog based on new information and customer feedback. User stories with high customer value and low uncertainty are delivered first, while stories with more uncertainty are updated as more information becomes available.

This Agile approach allows flexibility and adaptability when responding to changing requirements, ensuring that the final product delivers real value to customers. While Waterfall may be more effective in high-stakes situations, Agile’s focus on customer value and adaptability makes it a powerful tool for modern software development.

• Studies have shown that Agile methodologies can lead to higher customer satisfaction, improved project performance, and increased team collaboration and productivity (source: “The Impact of Agile Methodologies on Software Development Teams” by K. Petersen et al.)

Real-world examples of successful Agile implementations include Spotify, which uses Agile to continuously iterate and improve its music streaming service, and Salesforce, which uses Agile to rapidly develop and deploy new features and updates to its customer relationship management software.

Related —

• Part 3: Agile Design Techniques for Creating Strong and Scalable Solutions

4.4 Emphasizing the Human Element in Software Development

The Agile Manifesto, a document published in 2001 by a group of software developers, outlines a set of principles for agile software development.

Four of the twelve principles in the manifesto emphasise the human element in software development. These principles focus on collaboration, motivation, communication, and self-organisation.

• The first principle highlights the importance of customer and development team collaboration. The principle states that “business people and developers must work together daily throughout the project.” This principle recognises that involving the customer in the development process is crucial to the project’s success. It sets a high frequency for interactions to underscore the importance of involving the customer throughout the project. This is a significant departure from traditional Waterfall-based approaches, where the customer may not see the product until user acceptance testing (UAT), which may be too late to make any significant changes.

• The second principle emphasises the importance of motivation and empowerment in the development team. The principle states that “build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.” This principle recognises that motivated and empowered employees are significant influencers in the team’s performance and should be part of any Agile implementation. A study published in the Journal of Management found that employees’ intrinsic motivation positively relates to job satisfaction and performance.

• The third principle stresses the importance of face-to-face communication. The principle states, “face-to-face conversation is the most efficient and effective method of conveying information to and within a development team.” This principle recognises that face-to-face discussions are preferred over documentation as they add one or two dimensions to the conversation channels. Facial cues help communicate the interlocutor’s feelings about a specific idea, while kindness, empathy, and good overall communication skills increase the effectiveness and efficiency of a debate. A study published in the Journal of Business and Psychology found that employees who communicate effectively with their colleagues are more likely to experience job satisfaction.

• The fourth principle recognises the importance of self-organisation in teams. The principle states that “the best architectures, requirements, and designs emerge from self-organising teams.” This principle recognises that self-organisation is a property of complex systems, and human groups are profoundly complex. Self-organisation manifests, for example, through the specialisation of team members in specific product areas or by honing specific skills. The relationship, however, that the authors envisaged between this complexity and software architecture is not evident.

These principles recognise that involving the customer in the development process, empowering employees, using face-to-face communication, and enabling self-organisation in teams is crucial to the success of agile software development.

5. Agile Delivery Process

Viewed from an altitude of 30,000 feet, the Agile process is identical to Waterfall with its typical five stages of Define, Build, Test, Deploy, and Monitor, with which we are all familiar.

These five stages constitute the familiar Software Development Lifecycle (SDLC), and it’s hard to imagine software built in another way.

From the ground, however, the processes cannot be more different. Four key players, engaged in a continuous dialogue, run the following cycles:

• The business analysts define the requirements, approve the changes demoed during showcases, and might provide input and feedback during the design phase. They are responsible for understanding the business needs and translating them into requirements. They work closely with stakeholders to define the project’s scope, prioritize features, and create user stories.

• The product owners have a central and vital role: story writing, feature prioritization, managing the backlog, and engaging the stakeholders. Product owners are responsible for the product vision and its development. They work with business analysts to define user stories and prioritize the product backlog. They also work with the team to ensure the product is delivered on time and meets customer needs.

• Developers write code, implement features, and deliver working software. They work closely with the product owner and business analysts to understand the requirements and implement the user stories. They also collaborate with the testers to ensure that the software is tested and meets the acceptance criteria.

• Testers ensure the software is high quality and meets the acceptance criteria. They work closely with the developers to identify defects and fix them. They also collaborate with business analysts and product owners to ensure the software meets customer needs.

The below diagram shows the process flow between the four parties mentioned above.

We can immediately notice from this diagram that there are far more connections and interactions between the various players than in Waterfall, and that’s before adding project managers, scrum masters, architects, and other profiles to the mix.

What does that mean? Simply put, Agile is not intuitive to understand or explain, but more importantly, it doesn’t scale well and is bounded by a strict domain of applicability. Within that domain (see Agile Design), Agile is very powerful.

In the following sections, we will look closely at some of the novel concepts that Agile introduced and glean some insights and best practices.

Related —

• Process Engineering: Essential Concepts from Lean, Agile, and Toyota for Effective Software Development Processes

5.1 User Story Writing

The Agile User Story was first proposed by Kent Beck in 1997 and subsequently formalized and standardized in 2004 by Mike Cohn. 

A User Story consists of a logical unit of business requirements viewed from the perspective of the product’s end user. There are several templates in use along the lines of:

As a <role> I can <functionality>, so that I get <business value>.

User Stories are grouped in Epics, which can be regrouped into larger units such as Initiatives.

A critical component of a User Story (and indeed any requirement) is Acceptance Criteria which is nothing but a list of statements in the form of:

The product [shall, should] support this or that <functionality>.

User Stories typically follow a standard workflow and may be divided into smaller tasks and subtasks. Their progression is controlled by specific rules called Definition-of-Ready (or DoR) and Definition-of-Done (or DoD).

The setup of a user story template, workflow, DoD, DoR, and subtask format is implementation-specific and is acceptable as long as it doesn’t contradict any of the Agile principles.

User Stories are a popular means of capturing business requirements in Agile development because they offer several advantages over other methods:

1. User Stories are customer-focused — User Stories are written from the user’s perspective, describing what the user needs to accomplish. This helps to keep the focus on the customer, ensuring that the final product meets their needs and expectations.
2. User Stories are flexible — User Stories are brief and allow for changes and adjustments throughout the development process. This flexibility means that requirements can be refined or changed as the team better understands the customers’ needs.
3. User Stories are easy to understand — User Stories are written in simple language and are easy to understand. This makes them accessible to all team members, including business stakeholders who may not have a technical background.
4. User Stories encourage incremental delivery — User Stories are often broken down into small, manageable pieces, encouraging incremental delivery. This approach allows the team to deliver working software in short iterations, software that customers can test and validate.

Related —

• Software Effort Estimation: How to Get It Right the First Time

5.2 Backlog Management

advantages over more traditional forms of requirement gathering:

1. Prioritization — The backlog provides a clear, prioritized list of features that must be developed. This ensures that the team is always working on the most valuable features first and that progress is made towards the most important goals.
2. Flexibility — The backlog can be adjusted as new information becomes available or priorities change. This allows the team to adapt to changing market conditions or customer needs.
3. Transparency — The backlog provides transparency to the development process, allowing stakeholders to see what is being worked on and when it will likely be completed. This helps to build trust and confidence in the development process.
4. Collaboration — The team and stakeholders create the backlog collaboratively, encouraging communication and collaboration. This ensures everyone understands what needs to be done and how it will be achieved.
5. Continuous Improvement — The backlog is continuously refined and updated based on feedback from stakeholders, the team’s performance, and the changing environment. This helps the team to continuously improve its processes and practices, resulting in a better product.
6. Early Delivery of Value — With a prioritized backlog, the team can focus on delivering high-value features first, allowing stakeholders to see tangible results early in the development process.

5.3 Stakeholder Management

Stakeholder management in this context refers to a product owner’s ability to answer questions about the scope, schedule, or both.

Examples of such questions can be:

• When can we expect to have feature A deployed?
• Can we have feature A deployed before DD-MM-YYYY?
• Which features can be ready by DD-MM-YYYY?

To answer these questions, a product owner will refer to the story point estimation of each user story and how many “story points” a team can complete within a fixed amount of time, like a week or two, a metric sometimes referred to as velocity).

Effort estimation and story points are subtle notions that require lengthier discussions. They have been discussed in more depth here.

In summary, a story point is label developers assign to user stories to indicate their implementation difficulty. Story points are finer at small scales and grow coarser with the scale. The Fibonacci sequence is typically used:

Story point set: 1, 2, 3, 5, 8, 13, 21, 34...

Story points of remaining work and team velocity can be placed on a chart to predict when the project will be completed.

Related —

• Stakeholder Analysis and Management

5.4 Managing Technical Debt

Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely. […] Continuous attention to technical excellence and good design enhances agility. — The Agile Manifesto

In Agile development, technical debt refers to the accumulation of design and development issues that need to be addressed. Agile deals with technical debt by emphasizing the importance of technical excellence and continuous improvement.

Here are some ways Agile deals with technical debt:

1. Continuous Integration and Continuous Delivery — Agile development teams use automated build and deployment processes to deliver the software quickly and frequently. This helps to identify and address technical debt issues as they arise before they become more significant.
2. Refactoring — Agile development teams continuously refactor their code, restructuring it without changing its external behaviour. This practice helps improve the code’s quality and reduce technical debt.
3. Sprint Retrospectives — Agile development teams hold sprint retrospectives to reflect on the previous sprint and identify areas for improvement. This allows the team to identify technical debt issues and create action items to address them in the next sprint.
4. Technical Debt Backlog — Agile development teams maintain a technical debt backlog, a prioritized list of technical issues that must be addressed. This backlog is reviewed regularly during sprint planning, and items are prioritized and addressed as needed.

5.5 Reflection and Retrospectives

Agile requires practitioners to reflect regularly on how to become more effective. This strive for continuous improvement is implemented via retrospectives.

Retrospectives work by getting the team in a room every 4-6 weeks and retrospecting on the last period. The person running the retrospective writes down what has and hasn’t worked and, through collaboration with the team, determines what actions can be taken to avoid experiencing the issues in the future.

Retrospectives and similar rituals require a high degree of discipline so that actions are followed-up and diligently executed. The team and organisational culture should also be mature enough to conduct the sometimes painful pinpointing of mistakes without assigning blame and finger-pointing.

Related —

• Process Management, Improvement, and Redesign: The Essential Guide for Boosting Your Performance

6. Final Words

The first time I heard of Agile was about a decade ago, and I was entirely sceptical about its performance. Being a thorough believer in the no-free-lunch paradigm, I was convinced that a trade-off was made somewhere and that Agile promoters avoided discussing it.

It took me a bit less than a decade to appreciate Agile’s advantages (and challenges), and this only happened after I had become part of a fully Agile team for the first time in my career.

Although Agile is far superior to Waterfall in every aspect, it is not a silver bullet. Cracks in the Agile edifice can become immediately evident as project size steps beyond certain thresholds or when Agile is applied to specific projects.

7. Further Reading

Below is a formidable short introduction to Agile, specifically to the role of product owners, by Henrik Kniberg well worth watching.

---

8. References

• Boehm, B., & Turner, R. (2003). Using risk to balance agile and plan-driven methods.
• Browning, T. R. (2016). Design structure matrix extensions and innovations: A survey and new opportunities.
• Candea, G., Kaushik, R., & Fox, A. (2017). Power laws and the classification of extreme events in rapidly changing networks. Physical Review E, 95(2), 022310.
• Mckenna, P., & Copeland, S. (2017). Using project management software in agile projects: A review of the literature. International Journal of Managing Projects in Business, 10(4), 820-837.
• “The Impact of Agile Methodologies on Software Development Teams” by K. Petersen et al.