DevOps: Finding Your Path to Successful and Continuous Improvement

1. Overview

Looking at the below graph from Google Search Trends, we can see that DevOps has picked up much momentum in the last five years and is steadily increasing, making it an interesting project delivery topic.

The graph compares DevOps search trends with Agile and Waterfall, both well-established software delivery methodologies.

So what exactly is DevOps? How does it compare with Agile? How does it improve software deliveries?

To answer these questions, this article will investigate the following topics:

• What constitutes DevOps and what it means for software developers, project managers, and senior management
• What are Continuous Integration, Continuous Deployment, and Continuous Delivery
• Where these new concepts stand compared to Agile

Many different perspectives of DevOps exist. We found a lecture by Martin Fowler to be the most informative. Many of the concepts, terms, and definitions adopted in this article were introduced in that lecture.

In addition, we relied on the State of DevOps report to fill in the gaps in our knowledge of the current state of DevOps.

3. Definitions

DevOps is another “tool”, or more precisely, a set of tools, practices, and production processes that you can use to enhance your deliveries.

While Operational Excellence does not dictate specific instruments for delivering large software projects, practitioners must be fluent in the popular ones before making a strategic commitment to any of those, including DevOps.

DevOps, like Agile, may or may not be suitable for your specific situation. The challenges and limitations have been discussed in this article.

Before diving into the details, let’s review a few essential terms commonly used in this context.

3.1 Continuous Delivery

If your organisation can release changes to production quickly and frequently, it is doing Continuous Delivery.

To appreciate the challenges of Continuous Delivery, we can have a quick look at the traditional software development lifecycle, which looks somewhat like this:

1. Business Requirement Definition
2. Analysis
   1. Solution Architecture
3. Design
   1. High-Level Design
   2. Low-Level Design
4. Development
   1. Code changes
   2. Code Review
5. Verification and validation
   • Unit Testing
   • Functional Testing
   • System Integration Testing (SIT)
   • Regression Testing
   • Non-functional testing, such as performance or load testing
6. Deployment
7. Operations
8. Customer support and maintenance

We have emphasized the Testing stage by listing all its types for two reasons.

First, to give the reader some perspective of the relative effort involved and second, to underscore the importance of quality testing (coverage, automation, speed of execution) for achieving Continuous Delivery.

To ship new features frequently and quickly, you must ensure they aren’t breaking anything, and when they do, you can determine the problem and fix it in hours, not days.

Automated testing is your safety insurance against breaking production code, but the pace at which Continuous Delivery operates requires automating the build, test, deployment, and monitoring activities.

Let’s now turn our eyes back to Continuous Delivery.

Continuous delivery is a pipeline of tasks that starts with a small code change and then goes through a series of test stages executed in multiple environments, each closer and closer to the production.

The series of tests is called a deployment pipeline.

3.2 Continuous Integration

Continuous Integration can be summarized as follows:

• Several developers working on new features can integrate daily changes in the same codebase.
• If a new bug is introduced, it can be identified promptly by running automated tests to determine the latest change that introduced the bug.

This can be achieved by:

1. Pushing code changes in small sizes
2. The presence of a comprehensive battery of test cases
3. Having the ability to execute entire test suites quickly and on-demand

Continuous Integration is a requirement for Continuous Delivery.

3.3 Continuous Deployment

There is some disagreement in the community on what Continuous Deployment means, but we have found the definition from Atlassian’s website to be the most useful:

Martin Fowler describes Continuous Deployment as the task of frequent deploying to production, whereas Continuous Delivery is our ability as a business to do Continuous Deployment.

Continuous Deployment requires stakeholders to have enough confidence in deploying any changes as soon as the business provides the green light.

3.4 DevOps

Although a formal definition of DevOps is hard to come across, I find the one from Atlassian a good introduction:

This definition, however, lacks a core principle of DevOps: the fusion of development and operations (deployment, monitoring, maintenance, and support).

Before DevOps, operations challenges were not things developers had to worry about.

The traditional development procedures state that a change is ready for release after being tested; in practice, the latter could occur in lab settings as close or remote from production as one could imagine.

This gap between the test and deployment environment made releasing software extremely risky.

Martin Fowler emphasizes this fact in another lucid definition of DevOps:

All these definitions imply a movement towards uniting two teams, developers and operations, that mostly lived and worked in isolated silos before DevOps. The members of these teams are not referred to as DevOps engineers.

4. State of DevOps 2021

The 2021 State of DevOps report is produced yearly and follows the adoption of DevOps practices across organizations participating in the surveys. The current account has some interesting statistics, which we look at next.

We start with the below graph showing the percentage of participants reporting the different levels of DevOps practices in their organizations.

In 2021, 18% of organizations surveyed reported having highly-evolved DevOps practices in their teams. Breaking it down, we obtain the below matrix.

A quick look shows the significant improvements highly-evolved organizations were able to achieve:

• The frequency of production deployment which went down from a month to on-demand
• Lead times for new product changes cut down from weeks or months to minutes
• And finally, the change failure rate was reduced three folds from 15% to 5%.

These numbers are pretty impressive when considering the gap between the three categories.

5. Should You Start Using DevOps Now?

5.1 Pre-Requisites

We have described the challenges of migrating from Waterfall to Agile in great detail in a separate article and will not go through them here.

However, DevOps adoption being almost identically challenging as Agile is what must be pointed out here.

Many obstacles facing a transformation in project delivery methodology are not related to the technicalities of Agile or DevOps. Still, they are primarily associated with organisational culture, lack of necessary skills, and resistance to change.

The answer, however, to the migration question is not a simple binary one. There are many variations and gradations of Agile and DevOps, and which specific implementation you pick within that spectrum depends entirely on your particular constraints.

This spectrum of possibilities is what makes the migration question worth debating.

5.2 What Is Missing in Agile

The primary difference between Agile and DevOps is that the former lacks any concerns on deployment and operations, whereas the latter has fused the two.

This indifference is exemplified by the typical quality issue of test environments and how closely they represent production.

5.1.1 Laboratory Test Settings

Let’s look at some examples of the discrepancies between lab and production settings to appreciate the extent of the problem:

• Mismatches on OS versions or patches — A common problem is updated libraries and interfaces that may become unusable. Another issue is supporting multiple OS platforms, such as Linux and Windows; you use a function that runs OK on one OS but not the other.

• Different application versions — Such as database engines, third-party software, or APIs.

• Mismatching configuration that can differ significantly between clients — unless you offer a one-size-fits-all solution, a modified feature needs to be tested under a wide array of setups, typically one per client, to be safe.

• Hardware resources — This difference can be an issue when running performance tests on lab machines and extrapolating the results into production.

5.1.2 Development and Operations Silos

The traditional separation of duties between development and operations created silos and made communications across boundaries inefficient.

Therefore, the primary objective of DevOps was to remove that friction, an idea eloquently put forward by Patrick Debois.

DevOps relies on the premise that the best people to run the software are those who built it.

6. Implementing DevOps

6.1 Overview

DevOps requires obvious ingredients like specialised skills, effort, resources, and infrastructure, but equally requires less obvious ones like a supportive organisational culture and sponsorship by the leadership.

The following sections list the most influential requirements.

6.2 Automation

We have argued in a previous article about the crucial role of automation when migrating to Agile.

The critical part was automating all testing activities but now extends to most areas where human creativity is not required, especially around building, deploying, and monitoring.

Like in the case of Agile, this requirement is mandatory as the short span of sprints coupled with a fast release pace severely constrains overhead activities.

Participants from the State of DevOps 2021 Survey reported the following:

• 90% of the respondents said their teams had automated most repetitive tasks.
• 97% of the respondents agree that automation has improved the quality of their deliveries.

The table below shows all the areas that can benefit from automation. Tools and practices already exist to achieve that level of automation.

Stage	Automated Activities
Build	Compiling and building applications on different OS platforms, with varying build and package options.
Unit Tests	Run all unit tests on the latest code and ensure they pass before applying the changes.
Infrastructure Preparations	With the availability of programmable infrastructure, containerization, and cloud technology, you can spin up a fresh environment for every build or test run.
System Integration Testing	For each new build, you spin up a new test environment, apply specific configuration settings, run all system integration tests and report the results automatically to JIRA.
Production Deployment	Avoid any user or human errors when deploying or operating new software by rehearsing all upgrade operations on test environments and executing the same scripts on production in an automated manner.
Monitor Production Environments	Special applications can look at live system parameters and report faults in real time.

6.3 Infrastructure

A DevOps infrastructure can be considered a sequential integration of tools that allow the implementation of a Continous Integration/Continuous Deployment (CI/CD) pipeline.

The table below shows the various stages of a CI/CD pipeline, the operations performed at each step, and the tools used in the process.

Stage	Operation	Tools
Plan	Project planning, resource allocation, epic and user story creation, solution design, sprint planning.	Confluence, JIRA, MS Office
Code	Feature development which includes building unit or integration tests when using Test-Driven Development	Git, JUnit, Google Test
Build	Building of applications. You might want a different build per customer, OS, or another parameter.	Maven, Ant, Gradle
Test	Testing and quality assurance are performed at this stage. These can be unit, integration, or regression tests. Containerization with Docker, for example, and the setup of fresh test systems can be highly convenient for better results.	JUnit, Google Test, Selenium, Mockito.
Release	Packaging and release of new application versions.	Codeship
Deploy	Deployment, onsite or cloud-based.	Kubernetes, Docker, AWS, Azure
Operate	Deploying and configuring applications in different setups.	Chef, Puppet, Ansible.
Monitor	Monitoring application health and smooth operations.	Nagios, Zabbix, Splunk.

A few ideas that are worth noting when thinking about a DevOps infrastructure:

1. You can use a subset of the tools in a simple pipeline to enhance your current production processes without setting up and running the whole suite in its most elaborate form. A simplified setup can be an option if there is no business need or you don’t have the budget and resources. Think of Agile, Waterfall, and the Hybrid models in between.

1. Tools are never enough, and it is easy to lose sight of the big picture once you immerse yourself in the setup of sophisticated environments. The objective of adding new tools is to upgrade your production processes. Before any major investment, it would be best to decide whether this will bring you closer to achieving Continuous Integration or Continuous Deployment, assuming that this is what you are after in the first place.

1. The number of available tools on the market is impressive, and deciding which ones are the most suitable can be challenging. A great rule of thumb is always to prefer mature and time-tested methods and means over the latest craze.

6.4 Support from Leadership

Due to the potentially disruptive nature of migrating to new and demanding software delivery processes and the cost involved in time and effort, support from leadership is imperative.

This fact has been mentioned numerous times in the State of DevOps 2021 Report with a powerful message on the impact of leadership support on the DevOps adoption process.

The quote above has two critical ideas regarding organizations with highly-evolved DevOps practices: 

1. A top-down enablement exercise emphasizing the role of leadership
2. A bottom-up transformation underlining the importance of stakeholder management in decision-making.

The results of transformations on a large scale might be evident on longer time scales only, and hurdles will need to be cleared. Without leadership support, fatigue might set in.

There are times in significant transformations when painful decisions need to be made, such as abolishing or changing the roles of QA staff. Decisions of such magnitude need to be made at senior levels.

Operational Excellence promoted the importance of process governance to ensure standardization and shared knowledge across the organization.

Process standardization is fundamental to avoid improvements remaining in isolated and inconsequential instances.

6.4 A Non-Resistant Culture

Organisational cultures play a significant role in the organization’s evolution and can inhibit or foster innovation and change.

The State of DevOps 2021 Survey reports the following statistics among mid-level respondents regarding cultural blockers:

• 21% report their culture discourages risk-taking
• 20% state responsibilities among team members are unclear
• 18% report that fast flow optimization is not a priority
• 17% insuficient feedback loops

Aspects of organisational culture that can greatly impact the adoption of different methodologies are as follows:

Let’s see if we can elaborate more on the impact of organisational culture when it comes to change. Below are leverage points where organisational culture has much influence.

1. Risk appetite for trying new things
2. Acceptance (or lack thereof) vis-à-vis “Failed Experiments.”
3. Clear roles and responsibilities, especially when it comes to process governance
4. Top-down enabled delegation of power, i.e. how much trust the leadership shows for the team and how much power it’s willing to share
5. Prioritization of process improvement over BAU
6. Unobstructed information flow in both directions: top-to-bottom and vice versa

6.5 Agile Maturity and Fluency

DevOps is closer to Agile than Waterfall, which, in my view, emphasizes the importance of having some fluency in Agile practices.

This fact, however, does not mean that you cannot move into DevOps without going through Agile first; our view is that it is somehow irrelevant to the final result.

It’s just that DevOps is an even more radical departure from the old ways of Waterfall, and some exposure to or understanding of Agile can help assimilate the ideas behind DevOps.

7. Final Words

DevOps presents significant benefits that can radically improve your software delivery capabilities; therefore, we believe it needs to be part of any discussions on process improvement.

That said, religiously following specific ideas and making a cult is never great; we believe it will have the opposite effect.

Like any other methodology, DevOps is neither good nor bad, and its utility depends heavily on the specific circumstances it applies.

8. Further Readings

Great talk by Martin Fowler.