1. Process Engineering: A Brief Overview
Dr Jeffrey Liker dedicates seven chapters in his seminal book The Toyota Way to explain Toyota’s lean processes. While Toyota is primarily about automotive and manufacturing, and this website is about software development, the parallels that could be drawn between processes in the two domains are strikingly numerous. On the top, osmosis across the two industries has gone beyond drawing parallels and sharing abstract ideas, and this can be readily seen through practices like flow, kaizen, and kanban that have crossed over body and soul.
Although this article is about software processes, these cannot be wholly appreciated without looking back at some basic questions like what constitutes a process, how processes are formed, and how they can be optimized. More specifically:
The answers to these questions will come from The Toyota Way – 14 Management Principles From the World’s Greatest Manufacturer, Melvin Conway’s beautiful paper How Committees Invent, and others which you can find in the reference section. The question of process optimisation will be treated in a separate article.
The reader might wonder why we are troubling ourselves with concepts and methods so remote from software development, the central theme of our discussions. The answer is as follows. I believe ideas of great originality, like Kaizen and Kanban, have been transformed, twisted, and turned to become barely recognizable compared to their original form.
A correct diagnosis and a subsequent strategy and action plan for process improvement must return to the first principles. This deep curiosity about “Where and how processes first originated?” is the main driver for placing the Toyota Production System at centre stage.
2. What Is a Process
2.1 Defining Processes
Pictures, blueprints, most diagrams, and chemical structural formulae are state descriptions. Recipes, differential equations, equations for chemical reactions are process descriptions. The former characterize the world as sensed; they provide the criteria for identifying objects, often by modeling the objects themselves. The latter characterize the world as acted upon; they provide the means for producing or generating objects having the desired characteristics.
— Herbert A. Simon, The Architecture of Complexity
In his seminal work, The Architecture of Complexity, Nobel-laureate Dr. Herbert Simon defines a process as the means of generating an object with specified qualities and characteristics. In addition to physical, chemical, and biological objects, this definition also applies to organisational processes that aim to produce business and customer value.
A process is a structured method to achieve a specific goal, typically transforming raw inputs (material, information) into products with added value. Products are valuable to customers; they solve a well-defined business need. Therefore, processes allow the creation of value out of the following ingredients:
2.2 Why Is Structure Desirable
A structured approach to problem-solving offers the following advantages:
2.3 The Supplier, Input, Processes, Output, Consumer (SIPOC) Model
The Supplier, Input, Processes, Output, Consumer (SIPOC) model is a framework used in business process management and Six Sigma methodologies. It provides a high-level overview of a process, helping to define its key components and stakeholders:
The SIPOC model is a simplified and highly abstract model of what happens in reality. Processes can be complex, with many inputs, outputs, and steps configured in sequence or iterations, with feedback loops, branching, and other complicated structures.
Reducing complex processes to SIPOC models in an attempt to understand and optimize them destroys many aspects of their value-creation capacity. This happens when small units focus on maximising their productivity locally rather than refining inter-group collaboration. More on this later.
2.4 Efficiency and Effectiveness: Two Essential Metrics for Measuring Process Performance
2.4.1 Process Efficiency
Process efficiency refers to how well resources (such as time, money, and materials) are utilized to achieve a desired output. It focuses on:
In other words, process efficiency is about doing things right. It aims to streamline operations, eliminate unnecessary steps, and increase productivity. Process efficiency can be measured using various metrics, such as:
2.4.2 Process Effectiveness
Process effectiveness focuses on the degree to which a process achieves its intended goals or outcomes. It assesses whether the process is capable of delivering the desired results and meeting the requirements and expectations of stakeholders. In essence, process effectiveness is about doing the right things. It emphasizes the alignment between the process and its objectives.
Measurement of Process Effectiveness: To measure process effectiveness, you can consider the following indicators:
3. Cost of Inefficient or Ineffective Processes
The following symptoms are usually observed when poorly designed and implemented processes are in play:
The Role of Emotional Intelligence In Modern Organizations– An Ingredient or Byproduct of Great Leadership?
4. Process Flow and Organisational Performance
4.1 Flow and Optimal Processing Times
Flow is at the heart of the lean message that shortening the elapsed time from raw material to finished goods (or services) will lead to the best quality, lowest cost, and shortest delivery time. […] A lean expression is that lowering the “water level” of inventory exposes problems (like rocks in the water), and you have to deal with the problems or sink.
— Jeffrey Liker, The Toyota Way
Lowering the water level to expose inefficiencies is a powerful metaphor. It provides insights outside manufacturing if applied beyond issues of excessive inventory. Flow is a hypothesis that positively correlates optimal processing times with quality, low cost, and speed of delivery. Our common sense tells us otherwise, mainly that speed is synonymous to haste and that taking shortcuts today is deferring the problem to the future rather than eliminating it. Think of technical debt, a design or implementation shortcut consciously made to make the deadlines.
If we can imagine processes as entangled ropes on a floor, achieving flow means disentangling them so that each one can be used to its maximum potential. Disentangling processes can drastically increase efficiency through minimal switching of people, machines, and tools, removing bottlenecks, and serializing tasks. Process improvement is a topic; we will look at it in detail in another article.
4.2 Flow in Software Development
What if we have entangled fishing nets instead of ropes? Is flow conceivable in this context? This question is relevant when we contrast manufacturing with services, particularly software development. In fishing nets, everything is connected to everything else, and disentangling them is far more complicated.
Manufacturing is complicated, while software development can be either complicated or complex. Here, complex and complicated are not the same and refer to two domains in the Cynefin framework. The best way to understand this difference is through the following:
It is vital to understand that following established processes and applying process improvement methods cannot occur in complex domains where the correct process has yet to be established. The complexity that comes with software requires practitioners to identify the domain they are in correctly.
Scrum, Kanban, Extreme Programming, and hybrid models offer different ways to reduce this choking interconnectedness. We will look at these software development frameworks a bit later. Nigel Thurlow has some interesting ideas on flow in software development, which are worth reading. In the meantime, let’s understand where process inefficiencies originate and how methods like flow can remedy them.
5. Production Processes: A Brief History
5.1 Mass Production Thinking
Mass production is a familiar pattern of organisational setups. It’s based on the premise that grouping entities (people, machines, tools) of similar types, roles, or competencies makes production tasks more efficient. Most organisations are set up this way, with separate departments for finance, accounting, engineering, production, sales, support, and human resources. The efficiency of this setup is assumed to come from the following:
However, efficiencies afforded by mass production come at a cost, which can be summarized with the following points:
5.2 The Batch and Queue System
Imagine that you allocate a team of business analysts to a specific client project, and the work requires the analysts to be on-site for workshops. To maximize their time utilisation, you would want them to work on as many new requests as possible during their stay.
The same applies to specialised machinery, where switching from one mode of operation to another introduces costly delays while engineers reconfigure the machines. While the machines are being reconfigured, the workers are either switched to a different task or wait until the reconfiguration is done.
To minimize the overhead associated with switching projects or machinery (context switching) and the overhead of physically moving material or information between teams, we would like the payload to have the biggest size and the movers to do the least roundtrips. This operational pattern is called the batch and queue.
The batch and queue system requires that a request for new work reaches a certain threshold before a batch is moved to the next stage in the production process. The batch waits in a queue while more items are added and only moves to the next stage once it is full.
The batch and queue system is designed to maximize resource and time utilisation at the cost of higher lead times, lower quality, and lack of scheduling flexibility.
One-piece flow is how Toyota overcame the batch and queue system problems while sustaining low costs. In software, things got a bit more complicated and a combination of methods (Scrum, Kanban, eXtreme Programming, hybrids) were created to resolve those issues.
6. Sequences, Iterations, and Feedback Loops in Production Processes
6.1 Process Anatomy and Dynamics
At a sufficiently high level, any process can be described as a sequence of stages, with the output of the first feeding into the next. The Software Development Lifecycle (SDLC) is a prime example of how this reduction can be applied. The SDLC is usually depicted as a five-stage process of Analysis, Design, Development, Testing, and Deployment.
The reality, however, is much more complex, as a software developer or any other technician would gladly attest. The practitioner and the keen observer will distinguish the following structures and dynamics in a sophisticated process:
6.2 Interpersonal Processes and Their Impact on Organisational Processes
Interpersonal processes define the interactions of group members working towards a specific goal. They cover the following areas:
Interpersonal processes are crucial in shaping individual choices and judgments, both inherently uncertain facets of human conduct. Although psychologists have discerned recurrent patterns in human behaviour, these patterns are chaotic (although not absolutely random, the difference between chaotic and random is explained in the article on the Brusselator, a mathematical model of chemical reaction dynamics).
When examining production processes, it is tempting to focus exclusively on a process’ static structure (sequential, iterative, or parallel stage setup) or properties (feedback loops, backlogs, flow) and ignore influential aspects of human nature. While this approach is not incorrect, it is incomplete. This incompleteness is best described by a quote from Edgar Schein’s book Process Consultation: Its Role in Organisation Development (a highly-recommended read for any manager).
The network of positions and roles which define the formal organisational structure is occupied by people, and those people, in varying degrees, put their own personalities into getting their job done. The effect of this is not only that each role occupant has a certain style of doing his work but that he has certain patterns of relating to other people in the organisation. These patterns become structured, and out of such patterns arise traditions that govern the way members of the organisation relate to each other.
— Edgar Schein, Process Consultation: Its Role in Organisation Development
In other words, production processes can be designed to follow a blueprint, while interpersonal and group processes are emergent properties resulting from the interaction of agents in the system and, therefore, cannot be engineered.
Organisational processes exhibit complex behaviour and cannot be understood solely based on their structure while ignoring interpersonal processes. In that sense, organisations are complex adaptive systems, not machines.
The realisation that organisations are complex systems has wide-ranging implications, from management to performance, continuous improvement, knowledge management, and change management. Complexity levels will also determine whether Operational Excellence is feasible in a specific situation.
6.3 Codifying Complex Processes and the Problem of Extreme Specialisation
Have you ever been asked to document what you knew or the processes you followed for a handover or training session? How much of that knowledge would you have been able to codify in a reasonable amount of time? Probably not much; Dave Snowden, an expert consultant and the creator of the Cynefin framework, caps it at 10%.
Writing down a long list of server names, procedure calls, or product specifications is highly desirable. However, documenting processes or solutions to past problems assumes that future situations closely resemble or replicate historical ones, which is not entirely true. A problem to which we already know the answer is not a problem. Problems worth exploring are novel, complex, and highly dependent on the specific path a system has followed to arrive at its current state, making them unique. How does one cope with issues of novelty?
Dealing with uncertainty and novelty brings us to the question of specialisation. While developers who are exceptionally skilled in one programming language have great value if the organisation relies on that technology, they will struggle quite a bit if it adopts a new one. Teams should have an ideal mix of specialists and generalists, the latter combining ideas from various disciples to solve new problems in novel ways.
7. Identifying and Eliminating Process Waste
7.1 Waste: The Toyota Definition
Muda is a Japanese word meaning “senseless or meaningless”. […] This is why using Japanese words to understand the true meaning is important because muda doesn’t just mean waste. […] It’s actually about the clumsiness or lack of professionalism leading to unnecessary costs.
Waste is tightly coupled with manufacturing (at least historically through Toyota and Taiichi Ohno), and sometimes it is difficult to identify the equivalent of waste in highly complex, rapidly evolving industries like software development.
In manufacturing, waste is mainly associated with excess inventory, moving material between locations, and delays when reconfiguring machines, fixing quality problems, or switching products on assembly lines. In contrast, software development has none of these issues but a score of other problems equally damaging to an organisation’s delivery capabilities.
This section will delve deeper into the three types of waste outlined in Liker’s work, The Toyota Way. Subsequently, we will establish notable connections between these types of waste and the software development sector.
7.2 Non-Value Adding Waste (Muda)
7.2.1 Muda in Manufacturing
This type of waste refers to any activity or process that does not add value to the product or service. Muda can manifest in various forms, such as:
Toyota aims to identify and eliminate these wasteful activities to improve efficiency and reduce costs. One technique that Taiichi Ohno promoted is called the Gemba Walk or simply Gemba. Gemba is a Japanese term that translates to “the real place” or “the actual place.” It refers to the physical location where value is created, such as the factory floor, shop floor, or any other place where the work is performed.
The Gemba Walk involves going to the actual location where the work is being done to observe and understand the process. Ohno, a key figure in the development of the Toyota Production System (TPS), emphasized the importance of observing the Gemba to identify waste, inefficiencies, and opportunities for improvement.
Managers and leaders can gain valuable insights and make more informed decisions to enhance productivity and quality by directly observing the work environment and interacting with the people involved.
7.2.2 Muda in Software Development
In software development, we can think of the following activities as supporting the main effort of analysing a problem and writing the code contributing to the solution.
Which of these items has a direct influence on the final shape of the product, and which are waste? Dr Winston Royce defined waste in his famous 1970 paper Managing the Development of Large Software Sytems (the same paper that ushered the Waterfall age while at the same time heavily critiquing it).
Two essential steps are common to all computer program developments, regardless of size or complexity. There is first an analysis step, followed second by a coding step […]. This sort of very simple implementation concept is […] the kind of development effort for which most customers are happy to pay since both steps involve genuinely creative work which directly contributes to the usefulness of the final product. […] Many additional development steps are required [for larger software programs], none contribute as directly to the final product as analysis and coding, and all drive up the development costs.
— Dr Winston Royce, Managing the Development of Large Software Systems
This implementation concept that analysis and development are the only truly value-adding activities, while the rest (testing, documentation, refactoring, and everything that supports them from automation infrastructure to CI/CD pipelines) are not, is key to any process optimisation strategy. The management’s job, as per Royce, revolves around the following:
This approach to performance improvement and optimisation is centred on industrial engineering concepts. We must remember that maximizing productivity through industrial engineering is only one of three ways to manage a business. In Operational Excellence and the Structure of Software Development and Delivery, we listed two other schools of thought on business management: Organizational Theory and Behavioural Science.
Organizational Theory and Behavioural Science have alternative views on what organisations and individuals require to perform, but given we are talking about production processes, we will focus solely on Industrial Engineering.
Only quality ethnography (watercooler stories) coupled with statistics, KPIs, and raw data from the Gemba will allow managers to interpret and diagnose situations correctly.
Boosting Software Team Productivity: Innovative Management Strategies for Maximizing Your Team’s Delivery Before It’s Too Late
7.3 Unevenness (Mura)
7.3.1 Mura in Manufacturing
Mura refers to waste caused by unevenness or inconsistency in production or workload. It involves variations in demand, workflow, or resources that result in inefficiencies, such as:
Toyota uses several techniques to reduce mura. These techniques are integral to the Toyota Production System (TPS) and lean manufacturing principles. Here are some key techniques employed by Toyota to address mura:
Combined with lean manufacturing principles, these techniques help Toyota create a more stable and efficient production system, reducing mura and improving overall productivity, quality, and customer satisfaction.
7.3.2 Mura in Software Development
Have you ever experienced the stress of go-lives, end-of-release cycles, and looming deadlines? We all did, and we are prone to believing that this is the nature of the work and that the world has always been that way.
As one manager once told me, software activities are alternating cycles of “famine or feast”, and from that moment, I kept turning that statement in my head. Fast forward a decade, and the answer seems to have been partially posited and once again derived from the Toyota Production System (TPS). Let’s review some methods of levelling the workload implemented in software and see how they relate to TPS.
7.4 Overburden (Muri)
7.4.1 Muri in Manufacturing
Muri refers to waste caused by overburdening or straining people, equipment, or processes beyond their capacity. It includes activities that place excessive stress or workload on employees, leading to errors, accidents, or inefficiency.
7.4.2 Muri in Software Development
Overburdening people can happen for any of the following reasons and results in stress, anxiety, absenteeism, low productivity, and low motivation.
Overburdening can occur in well-motivated people and is different from motivation.
8. Software Delivery Processes: Anatomy, Efficiency, and Success Stories
In the previous sections, we have described various concepts related to manufacturing processes and how they relate to their software counterpart. While the link between processes in both fields is undeniably there, manufacturing and software (or services in general) have different contexts, and practices in their fields have diverged significantly. Assumptions from manufacturing are valid only when applied at a very high level.
In the following paragraphs, we will focus on the pioneering work in software development methodologies by practitioners like Winston Royce, Melvin Conway, Kent Beck, Ron Jeffries, Ward Cunningham, Jeff Sutherland, and Ken Schwaber. The objective, however, is not to fully describe each methodology or method (entire books have been dedicated to each topic) but to break down the respective methods (Waterfall, Scrum, Kanban, XP) into their essential elements. This return to first principles will allow us to understand how these methods operate on their lowest levels and what issues each resolve or introduce.
The following ideas describe the Waterfall approach, process, and implementation, with its strengths and weaknesses:
Scrum is a planning tool; it doesn’t make you Agile. But you can use it to achieve some outcomes that might lead to Agility.
Jeff Sutherland and Ken Schwaber developed Scrum in the early 1990s. Inspired by Lean Management and the Toyota Production System’s one-piece flow, Scrum presented a significant departure from the old ways of Waterfall. It all started when Sutherland inspected a large Gantt chart for a major project.
Sutherland saw on that Gantt chart a classical Waterfall implementation with many interdependent tasks, divided into batches of features to be analysed, developed, tested, and deployed as one single delivery. It didn’t take much for him to understand that a delay in any task will have a cascading effect on the rest, especially if it’s on the project’s critical path. Given the (typically large) number of tasks and people involved, delays were inevitable, and project failure was above 85%.
Scrum’s fundamental assumptions were as follows:
Scrum teams deal with waste as follows:
8.3 eXtreme Programming (XP)
Kent Beck, Ron Jeffries, Ward Cunningham, and others were the key contributors to the development and evolution of eXtreme Programming (XP) in the late 1990s.
Extreme Programming (XP) and Scrum are both agile software development methodologies, but they have some differences in their approaches and focus. Here are a few critical differences between XP and Scrum:
Kent Beck relates in his 2015 talk at the Lean IT Summit the impression that The Toyota Way – 14 Management Principles From the World’s Greatest Manufacturer made on him. eXtreme Programming (XP) was intended to be the Lean equivalent of software development, heavily inspired by The Toyota Way, and once again, traces of one-piece flow can be distinguished upon close inspection.
8.4 DevOps and Continuous Delivery
Agile and DevOps are two related but distinct approaches in the software development and IT operations domains. Let’s start by looking at some key differences between them:
What conclusions can therefore be made regarding DevOps processes? Here are some suggestions:
In manufacturing, Kanban is a scheduling system that helps manage and control the production flow. The term “Kanban” is Japanese and translates to “signal” or “card.” It was initially developed by Taiichi Ohno and his team at Toyota as part of the Toyota Production System (TPS), also known as Just-In-Time (JIT) manufacturing.
In software development, the Kanban methodology was adapted from manufacturing and applied as a framework for managing and improving the software development process. It emphasizes
It is helpful to compare Kanban and Scrum to understand how each is different. Here are the main differences between them:
Kanban is probably best described as a technique for managing work-in-progress and visualizing work rather than a software development methodology. Its essential contribution is getting Agile teams to switch from a push to a pull system without necessarily implementing Scrum or other Agile methods.
9. Over-optimisation, Resilience, and Fragility
Management overly focusing on industrial engineering techniques of process optimisation often lead to problems. Severe optimization of production processes leads to fragility due to several factors:
Software is a fast-paced, highly-evolving field, and adaptability is key. While structured approaches are desirable, excessive constraints on a complex system that is naturally uncontainable make it fragile, brittle, and unable to respond to change.
10. Process Performance: When Metrics Become Objectives
It is not uncommon for managers and teams to confuse the metrics they are using to measure performance with performance itself. Under these conditions, the metric and the objective become disassociated, and an improvement in the former might even be at the expense of the latter. Let’s see how.
Metrics can be compared to body temperature; lowering body temperature with paracetamol during a fever is good, but it doesn’t mean the infection is gone.
We started this article with a generic definition of processes and why it’s vital to have a structured approach to doing things, especially in large groups and complex situations. We then presented some ideas from Toyota, which, although very specific to manufacturing, have heavily influenced software methods like Scrum and eXtreme Programming (XP).
The conclusions that one can derive after such an examination are as follows:
As Taiichi Ohno emphasized, success cannot come from analyzing reports and second-hand information but by observing a process in the Gemba through an experienced eye. We hope this (long) guide on software development processes has brought both managers and developers intimate knowledge of what’s required to set up the right processes for development teams.
- The Toyota Way – 14 Management Principles From the World’s Greatest Manufacturer — by Dr Jeffrey Liker
- How Committees Invent — by Dr Melvin Conway
- Managing the Development of Large Software Systems — by Dr Winston Royce
- The Six Sigma Way — How GE, Motorola, and Other Top Companies are Honing Their Performance — by Peter Pande, Robert Neumann, Roland Cavanagh
- Nigel Thurlow – Have we forgotten our key denominator: FLOW? – Lean and Agile Summit 2022
- Scrum talk at Google by Jeff Sutherland