Thoughts on Six Sigma for Developing Your Software Engineering Processes

1. Overview

I have recently taken a lot of interest in groups trying to achieve a common objective, especially in professional environments.

I wanted mainly to understand the differences between two arbitrary groups regarding performance.

Specifically, I was keen on knowing whether there was something that those groups could do to dramatically increase their efficiency and lower their risks during that journey towards their common goal.

That search led me to two different but closely related business management models: The Toyota Way and Six Sigma (6σ).

Jeffery Liker’s book (full review) is a fantastic source for understanding The Toyota Way.

The book was a great inspiration for creating a framework under which software delivery can operate efficiently. This was mainly, in my view, due to the universality of its principles.

That framework is what we refer to in this blog as Operational Excellence.

Then, I picked up The Six Sigma Way book (full review) by Peter S. Pande, Robert P. Neuman, and Roland R. Cavanagh.

I have to admit the book itself was quite disappointing. Nevertheless, it understood Six Sigma’s foundational elements and limitations well.

I also formed an opinion on whether some Six Sigma concepts are universal enough to be applicable in other fields, such as service delivery.

This topic, the applicability of Six Sigma concepts in services, will be the centrepiece of this discussion.

We will especially focus our analysis on the software development side of the services industry.

3. What Is Six Sigma?

To fully grasp what Six Sigma is, we need to discuss four concepts:

1. Variation of a probability distribution in production processes
2. Closed-loop systems
3. Continuous Improvement of production processes
4. DMAIC

3.1 Production Variation

The term sigma (referenced by the Greek letter σ) comes from statistics and denotes the variance of a certain probability distribution.

A ubiquitous probability distribution is the Normal (or Gaussian) distribution with the familiar bell-shaped curve.

The below figure shows three normal distributions, with means 0, 1.75, -1.75, and a standard deviation 1.

The variation in the quality of manufactured items is assumed to follow a normal distribution, and quality control measures aim to keep these variations under control.

In scenarios where the probability of observing a defect follows a normal distribution, an observation six standard deviations away from the mean is 1 in a billion, a vanishingly small amount.

So why six sigmas and not, say, 3 or 4?

The rationale is quite technical, but here is the gist: Six standard deviations is broad enough, even if our quality controls relax a bit. We shift the mean by 1.5 standard deviations to the left or to the right, we have enough buffer so that the chance of observing a defect from such processes is 3.4 per million opportunities (or DPMO).

A DPMO of 3.4 can be a relative measure. For example, a 3.4 DPMO in transistors on a chip (in the order of billions per chip) means that every chip is defective.

On the other hand, a DPMO of 3.4 is a fantastic number in manufacturing car parts.

3.2 Closed-Loop Systems

So, in broad terms, the pan-ultimate goal of Six Sigma is to achieve a certain level of quality such that the number of defects per million is less than 3.4. Or is it?

This number is astonishingly ambitious.

I am unsure if any organization, in the manufacturing business or otherwise, could hit that mark on any of its processes for an extended period. Certainly not all of its processes.

There are, however, a few success stories that companies like GE, Motorola, Johnson & Johnson, and Texas Instruments have reported and publicized.

Those adopters of Six Sigma claimed to have saved hundreds of millions of dollars by following this process.

So what does Six Sigma mean?

I will quote the following definition, which I found very useful, from the Six Sigma Way book mentioned earlier:

Six Sigma is based largely on creating a closed-loop-business system sensitive enough to reduce the company’s “wobbling” and keep it safely on the often-twisted path to performance and success. — The Six Sigma Way

A closed-loop system is well-known in system engineering. It is made up of four elements:

1. Production processes
2. That consume inputs
3. And produce outputs
4. As well as a feedback loop

This feedback loop carries information on the deviation of the real output from the desired target value and helps the system keep track of its objectives.

The closed-loop system, through this design, can self-correct itself.

3.3 Process Improvement

We have seen that the objective of Operational Excellence is to eliminate non-value-adding effort (or waste) and increase the efficiency of the production processes while promoting a healthy and safe work environment.

So what do companies want to achieve with Six Sigma?

Companies that apply Six Sigma want to reduce destructive variations in the quality of their products. And not just that, but they also want to do it sustainably for extended periods.

For that to happen, they rely on a systematic approach to addressing quality problems known as DMAIC, or Define, Measure, Analyze, Improve, and Control.

Besides inputs and outputs, they need to measure all metrics inside the production processes.

Duration, yield, efficiency, throughput, and other useful measures are collected and subsequently analyzed.

If the number of defects is high or efficiency is low, the leadership (or a group of Green and Black belts) would devise a plan for improving these processes.

The plan would then be applied, and new data would be ready for collection and analysis. The cycle goes on indefinitely.

3.4 Process Improvement with DMAIC

Process improvement completed in this manner is often referred to as DMAIC, which stands for:

1. Define the system’s processes, the customer and their requirements, and what you would like to achieve long-term goals.
2. Measure relevant metrics of the current processes and determine the system’s current capabilities. This will establish a baseline against which you measure future performance.
3. Analyze the data collected to find the root cause(s) of the problem. Establish correlation and causality. More importantly, understand the relationship between the inputs, outputs, and internal processes and whether any patterns in the data linking those entities can be observed.
4. Improve or update the processes if the data is sufficient for a decision. You might need to redesign some of the processes from scratch. Set up pilot phases before deploying to production.
5. Control the production process to ensure that any deviation from the norm is quickly identified and the relevant data collected.

Searching for a Guide on Process Improvement?

Process management includes process improvement and redesign. Click the button for a full guide on DMAIC as well as unique techniques from The Toyota Way.

Guide to Process Management

4. Challenges of Applying Six Sigma to Software Development

Do Six Sigma practices carry over to other industries, such as services? Are its concepts universal enough to be transferable to software, for example?

Several deficiencies make Six Sigma either inapplicable or undesirable in certain settings. Still, some useful concepts would be interesting for software businesses.

Let’s first look at some of those limitations before moving on to the benefits in the next section.

4.1 Limited in Scope and Utility

In comparison to The Toyota Way, which, in our view, is the gold standard model of business management (as the history of Toyota share prices may indicate), Six Sigma targets are less ambitious compared.

Six Sigma is often marketed as an all-encompassing framework covering the organisational culture, people, processes, and quality control.

This stretches the limits of the application of Six Sigma significantly.

A quick look at how Six Sigma works are sufficient to understand that it’s only designed and equipped to control quality under specific conditions.

Six Sigma is centred around DMAIC, a data-driven method for continuously improving and optimising business processes. That’s what it is and what it’s meant to be.

Any additional efforts to stretch its usability seems, in our view, a (poor) way of marketing the consultancy industry and the services that have flourished around it.

In addition to quality control, you must look after your people, your organization’s culture, and external challenges like competition, technological disruptions, and economic mood swings.

Whether you are in the business of software development or something else, Six Sigma may bring benefits to quality control. Still, it will need to neatly fit in a much larger framework dealing with other challenges.

4.2 Inadequate For Small-Sized Businesses

Six Sigma requires a serious investment in training, effort, and tools. This might be enough to put most small to medium-sized businesses off.

In addition, small businesses are typically more flexible and can deal with quality control problems in much simpler and more practical ways.

• The Andon system is a perfect example. It’s ridiculously efficient and extremely simple to implement. It’s also easy to understand and explain to both senior management as well as the average worker on the shop floor.

In short, Six Sigma is an effort-intensive exercise that requires a complex infrastructure to support it. This raises a bit too much for small businesses to enjoy its benefits.

4.3 Overreliance on Statistics

Methods that rely heavily on statistical analysis only work when the data is high quality.

Data is deemed high quality if it satisfies two conditions.

• First, the processes that generate the data are consistent and time-independent. This means that observing a value of 10 today means the same thing if the same value is observed tomorrow.

• Second, the collected data is large enough to eliminate sample bias. This means that the sample we are currently looking at accurately represents the population/processes it came from.

When data quality is high, the inferred results are statistically significant and can be relied upon to make decisions.

When you have quality data, you can establish a baseline against which future improvements can be measured. This is crucial if DMAIC, and therefore Six Sigma, were to work.

As anybody who has worked in software development can tell you, repetitive tasks carried over for extended periods are rare. It’s not how software development works (at least most o the time, more on later).

Each project has some unique features and a different mix of team members. Also, technology changes fast, and new ideas come and go every day.

This is all the more true in small businesses where processes are not yet mature and standardized.

These make data collection difficult and statistical analysis results very weak.

4.4 Six Sigma and Agile Practices

Agile was developed as a response to ever-changing requirements in software projects.

Agile offered a way of dealing with unexpected events, eventually becoming a hallmark of software and IT projects. This fact leaves us with two thoughts on the subject.

• Agile accepted the fluid and somehow unpredictable nature of software. Instead of eliminating those uncertainties, it tried to provide innovative ways to deal with them.

• This fluidity in a dynamic environment can be difficult to track and measure with enough consistency for Six Sigma processes to apply.

Does this mean Agile and Six Sigma cannot coexist in the same environment?

Agile has brought a lot of advantages to software delivery. This makes it very hard to justify any destructive interference that might come along from Six Sigma.

5. How Can Software Development Benefit from Six Sigma

5.1 Universal Concepts of Six Sigma

So, are there any Six Sigma practices or ideas that are generic or universal enough to be carried over to software development?

The answer turns out to be YES.

The first concept that can be easily carried over is that of a closed-loop system. The fact that you can faithfully model any complex system of processes along those lines can be very helpful in understanding its internal dynamics.

Once you understand how it works, you can then analyze its performance and improve it where required.

The second concept has to do with measurement and data collection. Even though these concepts are difficult to apply to software activities, this is less so when some of these activities have been automated.

This is, in fact, one of the objectives that DevOps aims to achieve.

Finally, repetitive tasks with a quantitative and measurable outcome, such as software estimation, are great candidates for statistical analysis, as seen in the previous paragraphs.

5.2 Six Sigma and DevOps

Automating software production tasks (building, testing, deployment) removes the human variable from the equation and makes the results more or less consistent.

Also, pre-existing tools can easily measure, track, analyse, and improve machine activities.

This means less infrastructure to invest in and a lot less impact on personal relationships with employees.

To give a concrete example, we can look at Unit Testing and its efficiency.

We have argued in a previous article that the efficiency of unit testing depends greatly on the size of the unit being tested. If the size is small, the number of test cases required to cover all the different scenarios can be tremendously high.

This immediately translates into a huge cost of ownership and an equally large running time. These effects can be easily measured if Six Sigma was applied to enhance them.

5.2 Six Sigma and Customer Service

Other areas can still benefit from Six Sigma practices even without automation.

Modern ticketing systems like JIRA (which have evolved well beyond ticket tracking tools) offer much flexibility and tools for measuring and collecting data.

Even though the usefulness of the data and its quality can always be questioned, the fact that the infrastructure is ready is a magnificent advantage.

From another perspective, customer support, for example, is not as volatile as development. With systems like JIRA, some metrics can be easily set up, and insights into customer satisfaction can be obtained.

6. Final Words

Let us, for a moment, ignore the Six Sigma hype partly due to the consultancy business built around it, which aims to portray Six Sigma as a silver bullet to all your business management problems.

Let’s see if we can judge Six Sigma solely on its merits.

Our view is that Six Sigma practices are just tools like any other. They are neither good nor bad. Their utility depends on how and when they are applied.

In this sense, we believe that turning a set of valuable practices into a cult where they are ritualized and assumed to hold in all times and circumstances can do more harm than good. This statement is quite valid for Six Sigma as well.

Finally, those concepts and tools involving a lot of statistical analysis and data gathering make them less useful for small businesses, especially in software development.

Automation, however, can drastically change the picture.

You might uncover exciting insights if you run some primary analysis on the metrics that govern those tasks.

To help you decide whether Six Sigma is right for your business, you need a principal ingredient: Fluency in the topic.

We hope this article has brought you closer to the subject at hand.

7. Further Reading

• The Toyota Way by Jeffrey K. Liker is a MUST-read.
• Understanding Organisational Culture through the works of Edgar Schein is also indispensable.
• The Six Sigma Way book by Peter S. Pande, Robert P. Neuman, Roland R. Cavanagh.