One of the earliest, if not the earliest, descriptions of continuous process improvement was laid out in Frederick W. Taylor’s book on Principles of Scientific Management (1911):
[…] whenever a workman proposes an improvement, it should be the policy of the management to make a careful analysis of the new method, and if necessary conduct a series of experiments to determine accurately the relative merit of the new suggestion and of the old standard. And whenever the new method is found to be markedly superior to the old, it should be adopted as the standard for the whole establishment.
In this short paragraph, we can observe a glimpse of several notions that will become seminal in later works on production and quality management like Six Sigma.
- Closed-loop Systems: A constant stream of feedback from the worker on the ground is collected, analyzed, and assessed for potential improvements to existing processes.
- Continuous Improvement: Taylor recommends that continuous improvement become an integral part of the management’s policies rather than ad-hoc, short-term solutions.
- Piloting: A method of testing modifications on a small scale to confirm anticipated results before deploying into production.
- Process Standardization: through the adoption of superior methods throughout the organization. Process standardization might seem too obvious, but facts suggest otherwise. In reality, “Lack of consistent processes” has been cited repeatedly as a prime obstacle for Agile adoption.
- Bottom-up Approach: Finally, Taylor acknowledges that workers on the ground are entitled to suggest improvements to how they do their work. Interestingly, such thoughts appeared in the early days of the manufacturing industry, where class separation, for example, was still the norm.
In this article, we will take the discussion beyond Taylor’s initial ideas and make a detailed, technical tour of the classical process management methods.
Process management might involve two different activities: improvement, which revolves around small incremental changes, and redesign, where processes are re-invented and re-imagined.
Specifically, the following topics will be discussed:
- Why do we need process management
- The prerequisites for success
- Process management in the software business
- The DMAIC technique
- The Toyota Way of continuous improvement
2. Table of Contents
- 1. Overview
- 2. Table of Contents
- 3. The Purpose of Process Management
- 4. Prerequisites for Success
- 5. Process Management in Software
- 6. The DMAIC Technique
- 7. The Toyota Way of Process Improvement
- 8. Conclusion
- 9. Further Reading
- 10. Featured Articles
3. The Purpose of Process Management
Unless you are in the research business, you generally do not want to tamper with running processes. The risk and cost involved might be too high.
The purpose of process management, which invariably includes a change to existing ways of doing things, is usually in response to changes in the environment.
There are two types of process improvements, each with a specific purpose:
- Incremental changes: small scale, limited scope type of changes that aim at improving the effectiveness or efficiency of the current system. Their purpose might also cover team safety and morale.
- Redesign, or Transformation: Large scale, broad scope type of changes whose purpose is to address a potential threat to the business’s survival or benefit from a significant opportunity. Transformations might require changes on the level of organizational culture.
Process management is the vehicle that supports Operational Excellence as it has been defined earlier: a continuous improvement process that allows the business to survive in an ever-changing environment.
4. Prerequisites for Success
For any process change to succeed, it needs to satisfy three conditions which we present hereafter.
Granted, minor changes might require smaller doses of these conditions. Nevertheless, experts still believe in the necessity of these conditions in some quantity or another.
4.1 Existential Threat or Major Opportunity
Existential threats disrupt the current equilibrium of the system and introduce enough anxiety or guilt so as to compel the leadership to respond.
All human systems attempt to maintain equilibrium and to maximize their autonomy vis-à-vis their environment. Coping, growth, and survival all involve maintaining the integrity of the system in the face of a changing environment that is constantly causing varying degrees of disequilibrium.
These threats can take the form of internal or external pressures. Below are some common examples:
- Rising competition: this was amplified enormously by globalization and the vast capital available to entrepreneurs and startups. Customers are not restricted anymore to local vendors where new competition cannot quickly arise. Think of Huawei and Siemens.
- Disruptive technology: the emergence of new technologies is a constant threat for all businesses. Examples abound: although people still like to read books or watch movies, online purchasing/streaming has completely changed the landscape and put many bookstores and movie rental places out of business almost overnight.
- New regulation: Industries such as banking, finance, or inusrance, are heavily regulated. These regulations are also constantly updated and for businesses to remain compliant, maintenance of their processes and products need to be continuously carried out.
Existential threats are not the only drivers for change. Sometimes, the emergence of innovative technologies can open new markets which, in turn, spike the demand for new products.
Some examples are the smartphone and the first car from Ford.
If I had asked people what they wanted, they would have said faster horses.
4.2 Psychological Safety
Change is difficult because it requires the unlearning of previous assumptions. Accepting these assumptions as valid can be the basis of inclusion in your group.
On the other hand, abandoning them for new ones might mean being expelled from that group, and perhaps compromising your identity and integrity.
If an organization is forced to make major changes, it needs to have enough psychological safety to see the opportunity of solving its problems without losing its identity or compromising its integrity.
4.3 Support from Leadership
This applies more or less to the small to medium-sized changes since the major changes are, by definition, carried out by senior leadership in the organization.
The following arguments justify this requirement.
- Changes can be disruptive: Leadership needs to be on-board in case you face challenges or resistance from your coworkers.
- Change costs time and effort: in which case a clear business case needs to be presented to the leadership showing the net benefits of implementing those changes. This will help secure the resources and funds you need.
- Changes can be risky: Difficult decisions might need to be made in face of unaniticpated challenges that can emerge while the changes are being applied.
- Benefits might need a lot of time to appear: In this case, support from leadership helps maintain the momentum and avoid the premature abandonment of the project.
5. Process Management in Software
Lots of the ideas we presented so far come from the manufacturing industry. Let’s see whether these concepts can be transferred into software and services.
5.1 Process Improvement Challenges
The below is a list of the top reasons why process management and improvement can be difficult.
Some of these have been inspired by the State of Agile Report 2021 and the State of DevOps Report 2021, both of which dedicate a portion of their analysis to understanding the reasons for the slow adoption of what appears to be superior methodologies.
- Inconsistencies in Processes and Practices: The lack of standardized processes makes problem definition, measurement, and analysis very difficult. Under this regime, solutions and changes are localized rather than spread to the whole organization. Finally, the lack of consistent processes makes it challenging to test the efficacy and useability of any improvements.
- Cultural Clashes: Both reports cite “cultural” blockers such as risk aversion, resistance to change, insufficient feedback loops, unclear responsibilities, failure to share best-practices as major obstacles towards adoption of better software delivery methodolgies.
- Dynamic Nature of Software Businesses: People tend to believe that introducing substantial changes in a dynamic and rapidly-changing environment is ill-advised. There is however one major caveat in this argument. In fact, the double assumption that things will inevitably cool down in the future, and that there is an “ideal” time for making improvements is just incorrect. If we can learn anything from experience, it’s that bad habits tend to stick more the longer they are kept.
- Lack of Skill and Experience: especially in managing people and change. Fluency in topics like organizational culture, cultural transformations, process management, and process improvement techniques is essential for changes to succeed.
- Support from Senior Management: this is one of the three conditions for success that we have listed earlier. Senior management in software, as in other places, is under constant pressure to perform: sales, profit, and revenue take precedence over long-term, risky, and costly changes where results may not be immediately apparent.
5.2 Improvement vs Redesign
As we have mentioned earlier, there are two types of process changes: the first is incremental improvements which are limited in scope while the second involves major transformations/redesign.
This is quite true in software as well. The below table shows the level of impact of some of the more common improvement techniques.
|Switching to Agile||X|
|Training and workshops||X|
6. The DMAIC Technique
Key info on DMAIC:
- DMAIC stands for Define, Measure, Analyze, Improve, and Control.
- DMAIC was invented in the first half of the 20th century and is closely related to the Plan-Do-Act (PDA) technique also known as the Deming Wheel, after its inventor, the legendary quality expert Edward Deming.
- PDCA and DMAIC refer to a systematic, problem-solving technique used for optimizing prodcution processes and the cornerstone for continuous improvement.
- DMAIC, being an essentially data-driven approach, is the core tool used to drive Six Sigma projects.
The below figure illustrates the process:
Establishing a Baseline
Before attempting to kick-off the DMAIC process, a definite baseline needs to be established against which any future gains could be measured. The baseline could be something like A) how much a project is delayed as a percentage of its estimated total duration, or B) how many features had to be reworked due to unclear requirements.
In the next five sections, we will discuss each stage of the DMAIC process. We will also compare and contrast between Six Sigma and The Toyota Way where applicable.
Given also that this blog mainly concerns itself with software problems, we will also try to provide some useful examples from the industry.
6.2 Step 1: Define
The first step in the DMAIC process is defining the problem. Here are some key ideas to consider:
- The problem needs to be reproduceable and confirmed by data. This is essential for manufacturing businesses but might be difficult to stick to with smaller-sized companies where data/observations are scarce. Nevertheless, effort should be made to establish whether a certain issue is persistent or just a one-off.
- The problem that we are trying to solve should be coming from the Voice of the Customer. It is easy to get side-tracked by issues that are inconsequential for the customer and do not add any value to the product.
- The aim of any change to existing processes should be to improve any of the following areas: A) efficiency, B) effectiveness, C) team morale or safety, or D) to address internal or external threats.
A few examples of issues from the software world illustrate the point.
|Issue||Agreement with the Rules for Define|
|Projects take significantly longer to deliver||A) Has a direct impact on the customer’s budget, timeline, time-to-market.|
B) Processes are not efficient and drive the profit down.
|Delivered value is not up to the customer’s expectations||Impacts customer user experience of the product. Customer frustrated by having to use software that doesn’t address its business challenges.|
|A significant amount of rework on product features||Processes are not efficient|
|Regression issues in new releases||A) Has impact on customer project cost as releases for testing contain a lot of broken features that need to be reported and retested.|
B) Is not efficient
|Aggressive deadlines||Have a long-term impact on team morale and mental well-being.|
6.3 Step 2: Measure
Six Sigma is a data-driven, problem-solving technique and as such, relies heavily on measurement and collected data.
The two main challenges with measurement are:
- What to measure: For a single outcome, you can collect multiple statistics: mean, standard deviation, median, etc. You can also choose to transform data from continuous numbers into categories such as Bad, Average, Great.
- How to measure: Is it enough to take a single snapshot in time or multiple snapshots? If multiple, how much is enough? How can you compare performance metrics between different products or processes? Do you adjust your results to eliminate bias? What is the best way for quantifying subjective attributes such as customer satisfaction?
- How to collect data: some measurements are not easy to perform. Can you use proxies to measure hidden variables? How would you determine if you have enough data or not?
These sorts of challenges are what make Six Sigma a highly specialized technique that may not be suitable for every process.
They are also, in our view, a severe limitation. As we will try to argue for in the next section, analysing data from your desk is much inferior to checking things for yourself.
6.4 Step 3: Analyse
The techniques explained in this section are heavily data-dependent and widely used in Six Sigma processes.
6.4.1 Statistical Analysis
In some cases, it is possible to extract meaningful insights from data just by looking at it. Tools like Microsoft Excel allow the instant generation of scatter plots, histograms, pivot tables, and many other forms of visualizations from large sets of data.
These visualizations can help you spot patterns, trends, and correlations without requiring an advanced degree in statistics.
If these observations confirm your gut feeling, you can make reasonably safe conclusions that would then allow you to adjust your processes accordingly.
For other, more complex, situations, these tools might not be enough. This is where the statistician’s toolset can offer rich and generous solutions.
Advanced statistical analysis tools like the t-test, -test, Analysis of Variance (ANOVA), regression, or correlation analysis allow you to test complex hypotheses.
The t-test, -test, and Analysis of Variance (ANOVA) tests, for example, allow you to infer whether a variation in an observed sample is either due to pure chance or something is indeed different between this sample and the population that it has supposedly come from.
Basic Example of the Application of Statistical Tests
A familiar example would be to determine whether a coin that has yielded heads (or tails) five times in a row is biased (this is referred to as the Null hypothesis) or not.
Another Example of the Application of Statistical Tests
A more sophisticated example could be something like this: we have observed during the last week a DPMO of 6.5. Given that our benchmark is 3.5, is the 6.5 an indication that our quality control has regressed or is this observation due to chance alone and next week’s observation will probably show a smaller value?
The result is usually accompanied by a confidence interval, say 95%. This would mean something like: the probability that this coin is not biased is less than 5%.
The regression analysis tests whether an output variable can be expressed as a linear or non-linear combination of input parameters:
This equation would then allow us to predict the outcome for a new set of inputs .
Finally, correlation tests measure the degree to which two variables agree. A positive correlation means they rise and fall together, while a negative correlation signifies that, as one rises, the other falls, and vice versa.
It is important that the insights derived from the data should make sense. Otherwise, it might be safe to assume that some calculation or data collection error has occurred somewhere.
6.4.2 Pareto Principle or 80/20 Rule
Vilfredo Pareto was an Italian civil engineer, sociologist, economist, political scientist, and philosopher.
In 1906, he made one of his most notable observations. Indeed, he had found that 80% of Italy’s land was owned by 20% of the people.
This ratio of wealth distribution was not restricted to Italy but was observed in different places as well.
The heavily-skewed distribution was also seen in many areas of social and natural phenomena.
In fact, it was Joseph Juran, the quality management guru, who turned it into a principle, now known as the Pareto Principle, the 80/20 rule, or sometimes the law of the vital few.
The idea behind the Pareto Principle is simple enough: 80% of all effects are caused by only 20% of total causes. So how does that actually help us in improving our processes?
The answer is not hard to understand. Actually, if we collect all the various causes and list them by ascending order, we can isolate the top players (usually on the lefthand side), and eliminate those first.
By focusing on the major sources of problems, we first
- First, avoid being side-tracked by a larger number of valid, albeit smaller, causes that, even when fully addressed, do not add up to much in terms of overall efficiency improvement.
- Second, we earn the highest gains in the earliest stages of the improvement process.
6.4.3 Traveller’s Notebook
The traveller’s notebook is a great tool for analyzing a process. It consists of registering useful information while the product being worked on moves between the consecutive stages of the production processes.
For example, while a new feature is being developed, you can keep track of the following items at each stage of the SDLC.
- How many people were involved: for example, a coder and a reviewer are invovled during development.
- How much time was spent waiting: waiting for the code to compile.
- How much time was spent working: active development or testing.
- How many defects have been observed/fixed.
- How much reword occured: because the requirement was missing or unclear.
The traveller’s notebook is especially useful when trying to shorten production cycles.
6.4.4 Process Value Analysis
This type of analysis inspects every task that employees perform and tries to place it in one of the following three buckets:
- Value-adding: to take an example from software development, this could be the actual code, test cases, and automation scripts developed for a new feature. an increase in the Intellectual Property of a software code base increases the value of the product.
- Overhead: These tasks enable the value-adding functions to be performed, but they do not add value themselves. Examples of overhead charges abound. Team management, project management, documentation, and meetings are all activities that help get the work done, but customers typically would not be happy to pay for them. Such activities are essential in large teams and complex environments. It is sometimes easy to confuse value-adding tasks with overhead to justify the existence of a specific department or job. Care should be exercised in that area.
- Non-value-adding (or waste): Such activities can take the form of delays, internal documentation, reporting, waiting for code to compile, fixing issues in the CI/CD pipelines after an upgrade, upgrading your OS, and many more.
The result of a typical task, and contrary to what one might think, would yield something like this:
The objective of Process Value Analysis is to determine which tasks can be eliminated or reduced. These are typically in the Overhead or Non-value-adding groups.
Eliminating unnecessary tasks helps increase the system’s efficiency by reducing cost and decreasing delivery timeframes.
6.4.5 Logical Analysis
Logical analysis is a process improvement technique that involves looking at the process details and figuring out the root cause of the problem by following cause/effect trails.
Kaoru Ishikawa, a well-known organizational theorist and quality management expert, invented an excellent process analysis tool known as the Fishbone diagram.
- Starting at the head of the “fish”, we write down what we believe is the effect or problem we would like to fix. From the main lines, we draw up the major potential causes involved in creating such a problem.
- Once that is completed, we brainstorm each of those causes separately by asking “why five times”.
Using this technique, we aim to get to the bottom of the problem by finding the root cause and not just the symptoms.
6.5 Step 4: Improve
Now that we have found out the root cause of the problem, let’s look at some different techniques for resolving it.
Process simplification consists of removing redundant, obsolete, or non-value-adding tasks such as overhead or waste.
In software development, for example, you can look at eliminating unnecessary internal reporting or internal documentation.
Categorizing tasks into value-adding, overhead, or waste can be challenging. This challenge is primarily due to the insulated environments that people work in, where the end-to-end process is not visible to everyone.
In such situations, people can find it easy to justify the tasks on which they spend their efforts.
Automating repetitive and labour-intensive activities is an excellent way of improving process efficiency.
Automation, however, comes at a cost. It requires the introduction of new infrastructure and the hiring of special skills. You can also expect much pushback from people whose jobs might be at risk.
6.5.3 Managing Bottlenecks
Bottlenecks are chokepoints in the process flow, whereby the load or the cycle time at the bottleneck are significantly higher than in other places. The result is a slowing down of the overall process.
Below are some options you might use to widen the bottleneck:
- Increasing the resources or capacity
- Reducing the cycle time
- Adjusting the product itself
A typical example of bottlenecks in software development is obtaining input from highly-skilled engineers such as solution architects or senior developers.
The solution, in this case, is to manage their time wisely while training other developers to assist them when required.
Parallelization can increase process efficiency by optimizing resource usage. Things that can be completed independently are carried out in parallel and integrated later on in the process.
Parallel processing is expensive, though, as it requires more overhead to manage the parallel streams and some regular handshaking to keep vital information synchronized between them.
One area in software development that has enjoyed success in optimizing development and testing effort is Test-Driven Development or TDD.
If used wisely, it allows testing and development to be completed in parallel rather than in sequence. This parallelization of activities should, theoretically at least, reduce the testing and bug fixing time significantly.
In software development, process standardization allows you, among other things, to switch developers and testers between projects relatively quickly and with little or no training.
Another significant advantage of having standardized processes is the ability to propagate and test improvements across many teams.
Without standard processes, it is impossible to make measurements and collect meaningful data that can be compared across multiple periods and between different teams or projects.
6.5.6 Improving Resiliency
Resilient processes can withstand internal or external pressures without breaking. They also cater for a wide array of possible scenarios, and in the odd case where a novel situation evolves, there are mechanisms in place to help the system go back to its nominal mode.
Small software businesses can be particularly exposed to new situations as every project or customer can be unique.
You can turn average processes into superior ones by implementing a few steps, which we have discussed in this article.
6.6 Step 5: Control
The control phase involves modifying your current processes to include the latest process updates. This modification can mean implementing new infrastructure and retraining your staff.
At the end of the control phase, you must decide, based on established criteria and new data collected after the updates, whether or not your changes are responsible for driving the observed improvement.
Applying process changes is not very challenging from a hardware or software perspective as the available tools abound in software development. You might, however, face other sorts of challenges; these can mainly be pushbacks from your team.
The cultural dimension in process improvement can be a dominant factor and has a lot to do with the organization’s culture.
7. The Toyota Way of Process Improvement
The way Toyota approaches process improvement is not entirely unique to the organization.
In fact, it is deeply rooted in the Plan-Do-Check-Act (PDCA) problem-solving routine also known as the Deming or Shewhart cycle, which Edward Deming has originally proposed and encouraged the Japanese car manufacturers to follow.
The PDCA technique was modified and enriched over the decades by Toyota’s leaders. It came to include simple yet powerful techniques that are in striking contrast with Six Sigma.
Specifically, we will discuss four concepts:
- Go and See for Yourself, or Genchi Genbutsu
- Continuous Improvment, or Kaizen
- Ask Why Five Times to find the root cause
- Self-Reflection, or Hansei
7.2 Go and See for Yourself (Genchi Genbutsu)
In order to define a problem, first-hand information is deemed essential, and reliance on data from reports should only be used to confirm what one has personally observed.
Once enough observation has been made, expert judgement and expertise are used to evaluate the situation.
Let’s see these techniques in action.
Tadashi Yamashina was the president of Toyota Technical Center. Principle 3 of the 10 management principles that he laid out read as follows:
- Principle 3: Think and speak based on verified informaiton and data:
- Go and confirm the facts for yourself
- You are responsible for the information you report to others
Taichi Ohno placed more emphasis on facts than data as the latter is just a manifestation of the underlying process. Data is one step removed from facts:
- Data is of course very important in manufacturing, but I place the greatest emphasis on facts.
The commitment to these principles (and the benefits achieved as a result) can be seen in the way Toyota resolved their Sienna car’s sales problem in North America.
The car was not very successful and Honda outsold it in 2003 by double the numbers.
Here is what happened: Yuji Yokoya, the chief engineer for the 2004 Sienna, started a road trip in 2001 where he drove the old Sienna and competitors’ vehicles 53,000 miles through every state in the U.S., Canada, and Mexico.
He wanted to get first-hand information on how the old Sienna fared on the North American roads and how it compared with its top competitors.
The new Sienna had at least 8 major improvements in aerodynamics, noise control, navigation systems, safety, and maneuverability which could probably never have been detected by analyzing charts on a desk in Japan.
7.2 Ask Why Five Times
The default method that Toyota engineers use for finding the root cause of particular problems is quite simple. It’s referred to as Ask Why Five Times.
Sophisticated statistical analysis is used here and there, but it’s pretty different from Six Sigma, with its clear structure and total commitment to its data-driven, complex methods.
Asking why five times requires a painstaking effort to get to the bottom of the problem. It would be best if you also had a solid grasp of the underlying processes so that you can appreciate the information you receive and be properly placed to make full use of it.
7.3 Self-Reflection (Hansei)
Hansei is a word in the Japanese language that we can break up into two. First, there is Han, meaning to change, turn over, or turn upside down. Then there is Sei, which means to look back, review, and examine oneself.
It embodies the act of self-reflection on a mistake you have committed and a commitment you make to remedy the situation and guarantee it doesn’t occur in the future.
Hansei is an alien and not entirely welcome concept in western culture.
- For starters, it seems to focus too much on the negative side of a given thing.
- Second, Hansei doesn’t seem to encourage celebrating the small wins, a morale-boosting exercise.
- Third, Hansei doesn’t dilute the responsibility of an individual by teamwork. On the contrary, it expects individuals to assume personal responsibility for mistakes, not to punish them, but for improving their work.
In the words of Yamashina: “Without Hansei, it’s impossible to have Kaizen”. This thought is very subtle and requires some unpacking. It seems that Kaizen is not about improving others but improving oneself.
A final thought, in the Toyota culture, even if you do a job successfully, there will still be a need for Hansei-kai, a meeting for evaluating the project, a retrospective:
An inability to identify issues is usually seen as an indication that you did not stretch to meet or exceed expectations, that you were not sufficiently critical or objective in your analysis, or that you lack modesty and humility. Within the process, no problem is itself a problem.
Process management is one of the essential pillars on which Operational Excellence rests. It can keep your business afloat during times of change, both internal and external, whether you are onboarding new clients or facing emerging technologies and new competition.
Luckily, when it comes to tools and techniques, you have several options from which to choose.
The first option is Six Sigma, a data-driven, effort-intensive framework that may or may not suit the type of business you are running.
We published some thoughts on the suitability of Six Sigma for the software industry. You can follow the link if you are interested in exploring it further.
The second option is The Toyota Way, which, as we have seen, offers a somewhat different set of tools. It’s also more encompassing as it includes other aspects of the business, such as people and leadership.
The Toyota Way emphasises three areas in particular when it comes to handling process improvement:
- First, the individual responsibility of every team member, which can be seen in the practice of Hansei.
- Second, a hands-on, close-up approach for problem-solving in stark comparison with Six Sigma’s method of analysing a challenging situation from behind your desk.
- Third, a strive for perfection that is easily noticeable when rooting out problems or making decisions.
The third and final option is DMAIC, a straightforward, no-nonsense methodology for continuous improvement. DMAIC can be observed at Toyota in one form or another.
Unlike the Toyota Way, DMAIC is limited to managing and improving processes. This limitation makes DMAIC easy to explain and apply; it also makes it a perfectly suitable solution for small businesses that can’t afford more extensive equipment.
9. Further Reading
- The Toyota Way by Jeffrey K. Liker is a must read.
- Understanding Organizational Culture through the works of Edgar Schein is also indispensable.
- Six Frames for Thinking About Information by Eduard de Bono is an enjoyable short read.