Operational Excellence – Principle 4: Implement Agile Processes Where Applicable

ByGeorges Lteif
Principle 3 Up Principle 5

I. What Is Agile and How Will It Help

Interestingly, two decades after the Agile Manifesto was announced, there is still no widespread agreement on what Agile looks like. While everybody agrees that Agile practices must include user stories, sprints, backlogs, and Kanban boards, the basic underlying principles of Agile (and what problems its original creators’ intended it to resolve) seem elusive.

While Agile training, certifications, and coaches abound, Agile seems to have lost its teeth. In Snowden’s words, Agile has been “domesticated.” The domestication metaphor and Snowden’s subsequent call to wilden Agile again deserve some explanation. The curious reader is invited to view any of Snowden’s lectures on Agile or head to our article on Agile’s history and first principles.

So what is Agile, and why is it vital for operational excellence in software development? Agile was born from Extreme Programming (XP), Scrum, and what was known back then as lightweight methodologies. These were presented as an alternative to Waterfall and were destined to solve some of the problems that plagued large software development projects, of which the following two are most relevant to our discussion:

  • Working with unarticulated needs:
  • How can software engineers accurately specify the design of a software application if users keep changing their minds about what they want? Because users can’t predict what technology can do for them, it is only when they start interacting with it (a prototype, for example) that they refine their requirements. User preferences and the prototype they are working with “coevolve”.
  • Because Waterfall required the design to be fixed upfront, it could not handle requirement volatility, and more “agile” methods were needed. Scrum, for example, is ideal for turning complex problems into complicated ones (in Cynefin terms). Complex problems admit various alternative solutions, whereas complicated problems have engineering solutions that already exist and can be easily implemented once identified.
  • Smaller, more frequent deliveries:
  • Waterfall stages are sequential by design, and for large projects, the feedback loop between requirement gathering stages and user acceptance testing was measured in months, way longer than it should be. Even when the project could be broken up into small features that could be delivered independently, it was more economical to lump them into one large initiative. This, however, meant that customers waited longer than they should for new features.
  • More importantly, project failure rates were extraordinarily high and, due to their size, extremely costly as well. What was missing was a delivery methodology that allowed big deliveries to be substituted for small, faster ones, and this is where Agile shines the brightest. The extra overhead of managing many smaller deliveries was compensated through sophisticated tools like source code versioning, orchestration, automation, monitoring, knowledge sharing, collaboration, and test simulation.

Does this mean that Agile is the universal solution to software development projects? Is Waterfall obsolete? Not quite. Let’s look at why this is not the case.

II. It’s Not Agile’s Fault That It Can’t Scale

Agile does a fantastic job in small teams. However, because of its high interaction levels and dynamic nature, its cost at scale becomes prohibitive. Consider the following scenarios:

  • Managing large software integration projects. You are trying to deliver a software solution consisting of many platforms with intricate interdependencies between them. The platforms are mature, and specifications could be reasonably determined upfront with enough precision. Furthermore, a new feature might require changes in multiple platforms and, therefore, has to be designed and tested end-to-end. Aside from development and unit testing, which can be done in parallel per application, all other stages of the SDLC (requirement gathering and analysis, architecture and design, system integration testing, and user acceptance testing) must be done as a linear and sequential exercise. In this case, the sequential stages will necessarily follow a Waterfall model, while parallel development on the application level could be conducted with Agile methods.
  • Agile management and organisational strategy. The job of senior management is to make educated assumptions about the company’s future potential and evolution in a fast-changing industry and competitive market. A strategy would then be built on those assumptions, and an action plan would be created from that strategy. The vision, strategy, and action plan would be disseminated gradually among the various divisions, departments, and their teams, outlining the expected contribution of each unit. Units will then set up structures, request funding, purchase tools, hire resources, and expend significant energy to make that happen. With every decision made, the set of possible alternatives diminishes. Imagine an organisation with a few thousand people strategizing and planning in two-week sprints!

It might seem absurd to discuss organisational strategy after everything we said above and, in the first part (see Operational Excellence and the Structure of Software Development and Delivery), the challenges involved in predicting an organisation’s future behaviour, let alone controlling it. A closer look, however, would dispel those doubts. Short sprints (a couple of weeks) in small teams seem appropriate. Larger teams can go for 3-6 month sprints, while it’s entirely reasonable for organisations to have 2-5 year plans.

Just like a chisel and a jackhammer are conceptually similar, their domains of applicability are entirely different. Any tool or method works great in a specific domain, but when the context shifts, its effectiveness diminishes. This applies equally well to Agile, Waterfall, and anything in between.

Agile can scale, but not necessarily through SAFe, Kanban, or Scrum. I will discuss that in the next section.

III. Agile Across the Organisation

The following are some guidelines on how to make the best of Agile in your organisation.

  • Agile is not a mindset or culture:
  • Agile is not a mindset or culture and cannot be acquired from crash courses, two-day workshops, or hiring external Agile coaches. Agile has distinctive features like fast and frequent delivery and iterative methods. If these features don’t apply, you are not agile, and that’s OK. Waterfall project management is fine in some contexts, such as major infrastructure projects.
  • Use XP, Scrum, Kanban (or any variation) if your software team is small.
  • This might sound unrealistically easy, and although it’s not, it’s worth the effort. Teams used to working in Waterfall will find the Agile learning curve quite steep. They might also lack the tools, especially in automation, to make it feasible. On the positive side, Agile methods can be broken down into their lowest coherent levels and recombined in different ways to suit new needs. More about this in a second.
  • Leverage knowledge management and collaboration tools.
  • Being agile can be summed up in two ideas: a) a high degree of interaction with different stakeholders, and b) short and frequent deliveries. To efficiently manage agile teams, a high degree of coordination and planning is required, and this is facilitated immensely by collaboration and knowledge management tools like JIRA and Confluence. Delivering complex software with spreadsheets will be daunting and take away the precious effort and time that could be invested more wisely somewhere else.
  • Leverage development, testing, automation, and deployment tools:
  • The extra overhead due to agile teams contributing daily to the master branch and the release team frequently deploying into production can be compensated by automating as much of the manual effort as possible. Development teams should have budgets allocated to continuous integration and continuous deployment tools, infrastructure, and talent. This initial investment is the biggest obstacle that teams must overcome to transition from Waterfall to Agile.
  • Scaling Agile:
  • Although the original creators of the Agile Manifesto came from different backgrounds, its core principles are surprisingly coherent. I believe that the source of this coherence comes from the high level of abstraction of the twelve principles and, therefore, no specific implementation of Agile (Scrum, XP, Kanban…) or method (Test-Driven Development, sprint, user story, backlog, retrospective, sprint planning…) embodies all twelve principles and is universal or superior to the rest. All these methods and practices have bounded domains of applicability.
  • Agile can only scale by decomposing a specific implementation into its fundamental methods and practices and recombining them in novel ways depending on the context. Think of the Hybrid model, for example, which combines elements of Waterfall and Agile. Theoretically, nothing is wrong with this model as it combines powerful Agile practices within a traditional and easy-to-explain Waterfall implementation.
Principle 3 Up Principle 5

Leave a Reply

Your email address will not be published. Required fields are marked *