One of the subtle and fundamental principles of software development is that no single person can deliver software by themselves; even in the most trivial cases, at least one employee from the client and vendor side must be involved.
Straightforward as it seems, the constant collaboration of all parties will be required, as is the case in any complex social group.
This collaboration must be effective and efficient, two concepts we will explore later in this article. Of the necessary prerequisite for achieving these objectives, properly engineered production processes must be a top priority, and these will be the topic of our present discussion.
In the software business, these processes constitute the Software Development Lifecycle or SDLC.
Having written down the rules governing the SDLC is a significant step but is not sufficient by itself to guarantee success.
You need a few metrics to gauge the process performance and a mechanism to fix or update it if it fails or becomes obsolete. Process improvement would then need to kick off and remain steady for the entire lifetime of the business.
Designing superior production processes and applying a strategy for continuous improvement can be complicated.
We will try to provide insights that will hopefully help you design efficient and effective processes throughout the following paragraphs.
It is important to note that for these suggestions to be useable, there are two main criteria that it needs to follow.
- Firstly, the concepts must be methodology-agnostic, i.e. they must be universal enough to apply to any project management and delivery methodology we choose (Agile, Waterfall, or DevOps).
- Secondly, any process design that satisfies those rules must be in total harmony with the principles we laid out for Operational Excellence. For example, there is no point in optimizing our activities for maximum performance at the cost of the staff’s wellbeing.
To complete the discussion, we will examine some of the symptoms of poor processes. We will also have a few words to say on Six Sigma and perhaps borrow some powerful concepts.
Next, we describe the rules that should be applied when designing superior processes.
Finally, we briefly discuss process improvement and redesign (the topic is discussed in fuller detail in another article).
2. Table of Contents
- 1. Overview
- 2. Table of Contents
- 3. Performance and Poor Processes
- 4. Modelling Production Processes
- 5. Cost of Poor Quality (COPQ)
- 6. Superior Processes
- 7. Guide to Implementing Superior Processes
- 8. Further Reading
- 9. Featured Articles
3. Performance and Poor Processes
3.1 Symptoms of Poor Processes
The following symptoms are usually observed when poorly designed and implemented processes are in play:
- Symptom 1: Unable to meet basic customer requirements, let alone offer any innovative solutions.
In the customer satisfaction model proposed by Noriaki Kano, there are three levels of customer requirements.
- Basic Requirements (or Dissatisfiers). These are the must-haves. You get absolutely no credits for meeting those requirements but a whole lot of dissatisfaction if you don’t.
- Variable Requirements (or Satisfiers). These have a direct impact on your organization’s ratings. They typically include timely delivery and good customer service.
- Latent Requirements (or Delighters). These set you apart from competition and are usually the product of innovation.
- Symptom 2: Inconsistent results that vary wildly between teams, projects, and people.
- Symptom 3: Process Improvements are typically local, sporadic, and short-lived; they are not properly assessed, standardized, or propagated to other teams.
The improvement of production processes must itself be a robust, well laid-out process.
It must also be sustainable so that continuous improvement, propagated across the organization, can result in a net positive and measurable difference.
- Symptom 4: Scaling problems immediately arise when the team takes on bigger projects or grows in size.
Fragile systems can fracture and break under pressure. One of the challenges observed in startups for example is their inability to scale once they start achieving exponential growth.
Among IT projects, failure rate corresponds heavily to project size. An IT project with a budget over $1M is 50% more likely to fail than one with a budget below $350,000.
The problem is not better for larger companies as they take on bigger projects. Poor processes lead to less resilience in the face of challenges.
- Symptom 5: Frequent management intervention is required to put out fires.
Constant fire-firefighting and crisis management are sure signs of processes breaking down.
3.2 Poor Processes and Organizational Cultural
Organizational culture plays a pivotal role in the performance of a group. This can make things a bit confusing as the symptoms of a toxic culture can overlap with those of poor processes. A thorough understanding of both is required to identify the subtle nuances.
A great culture coupled with refined knowledge and expertise on the leadership level creates superior processes.
Sadly, we cannot say that this would be true the other way around. In fact, good processes and great knowledge can diminish the effects of toxic cultures but will not eliminate them.
4. Modelling Production Processes
4.1 Proposed Model
The model we use is derived from the Supplier, Input, Processes, Output, Consumer model also known as SIPOC.
In its basic form, we can define production processes as a set of activities that takes raw input and transforms it into consumable outputs.
We can then break up those activities into different, specialized stages.
Each stage takes a product that’s still a Work-In-Progress (WIP), adds value to it, and pushes the modified WIP to the next stage.
Once the process is completed, the product would have increased in value by an amount that we will call:
Where s is the total number of stages and is the business value generated in stage I.
The Six Sigma model introduces the notion of opportunity. A complex product is one where more than one thing could go wrong. Each “thing” that can go wrong is one opportunity.
This view makes the comparison of the production processes of different or complex products possible.
This makes the value introduced at each stage the sum of the opportunities added:
4.2 Defining Value-Adding Effort
4.2.1 Tangible Value
Tangible business value can be observed, measured, and compared. The most common form this value can take is the dollar figure that we believe customers will pay to acquire and use the product. In this case, the term “business value” is a synonym for “company asset”.
A product, software, idea, or service is valuable if it satisfies the below conditions:
- Addresses the customer’s business needs
- Has a low cost of ownership and maintenance
- Boasts excellent customer support
Tangible Business Value can be present in any of the following offerings:
- Software code and applications
- Data and insights derived from it
- Customer services or consultancy
- Intellectual property and innovations
- Hardware, specialized tools, or high-tech equipment
- A differentiating and efficient business model
4.2.2 Intangible Value
There are also other forms of value that we cannot directly observe or measure such as benefit to society.
A familiar example would be the number of families whose members have a job in the company. Another example is how certain technologies have considerably improved the quality of life of some segments of a population.
For the majority of small businesses, perhaps it is OK not to worry too much about anything other than the monetary value of your products.
This might be more relevant to mega-corporations whose products can have a significant impact on society.
4.2.3 Resellable Value
It is typical when producing IT products such as software that you would want to sell that product as many times as you could.
- In fact, one of the reasons why stocks in software companies are valued based on revenue, rather than profits, is exactly due to their resellable value.
We can now rewrite the business value of a certain product as a combination of two elements follows:
- is the monetary value that can be tapped only once
- is the monetary value that can be received for each new sale
- being the size of the Serviceable Addressable Market (SAM)
A more conservative approach would be to take the Serviceable Obtainable Market instead, which is usually a subset of the SAM.
4.3 Defining Non-Value Adding Efforts or Waste
We can define waste as any activity, expense, or overhead that requires effort but does not add any value to the overall product.
To better understand waste, we break it down into four separate categories.
We call the total amount of waste generated throughout the whole process and:
- is the amount of overhead required for the process to completed.
- is the amount of hidden waste. More on that in the next paragraph.
- refers to waste resulting from low skill levels, waiting times, inefficient processes.
- this is the amount of work that needs to be redone because something was missed in the first run.
This type of waste refers to any effort expended to get the work done but which does not contribute directly to the product’s value.
A perfect example of overhead is management. Management intervention can vary depending on how smooth and resilient internal processes are.
Other forms of overhead can include internal documentation, compliance, and security measures.
When calculating the price of a certain consumable product, most companies do not factor in hidden costs such as pollution, or health and safety risks. This helps keep the prices competitive at a cost that will only be evident in the future.
A similar concept can be applied to software development where waste generated today can only be visible in the future. The most notorious example of such waste is technical debt.
Unfortunately, we are not able to detect or measure hidden waste as easily as we want to.
This leaves us with an approximate value of waste which we can think of as a fraction of the total waste :
Waste from Inefficiency
This type of waste is the result of the inefficient execution of production processes.
Some examples of such types of waste can be the result of:
- Poorly trained resources
- Poor cross-functional team coordination
- Idling and waiting times
- Ambiguous roles/responsibilities
- Low-quality execution standards.
Any time a development task is not completed in the first pass, some of its subtasks will need to be reworked.
The top reason for rework in software development is oftentimes the results of unclear requirements.
There are other reasons as well like:
- Not following coding and security standards
- Compliance problems
- Poor user experience.
5. Cost of Poor Quality (COPQ)
According to this report, the Cost of Poor Quality (COPQ) in the US software industry is a whopping $2.08 trillion.
The COPQ is dividing, according to the same report, into three major pieces:
- Software failures – 75% or $1.56 trillion
- Legacy code – around $520 billion
- Unsuccessful development projects estimated at $260 billion
COPQ is another metric that can be used to measure the efficiency of your production processes.
Its advantages over Waste are:
- It provides a fair tool for comparing the performance of two processes or products
- It speaks a language that the leadership can understand (money)
- It allows the leadership to perform a cost/benefit anlaysis for addressing those issues by using the same unit of scale (the dollar value)
6. Superior Processes
What does it mean to have Superior Processes?
Having Superior Processes in place means that your system is
- Processes are effective, customers are happy
- Product quality is consistent, variations are low
- Efficiency is high, the product is profitable
- Operations are smooth, crises are rare
- Processes that control the improvement of production processes are smooth and efficient, in which case the system is capable of responding to change
- Performance can be tracked, Key Performance Indicators can be measured
In fact, we can define superior processes as those that exhibit the following properties.
Effectiveness describes your ability to deliver on time, and within budget. Equally important is meeting the customer’s expectations. All three pillars, time, budget, quality form the basis of this metric’s assessment.
You can assess the effectiveness by looking at the two metrics.
The first metric is how far off we are from the original estimations of the project in terms of timeline, budget, and scope.
The second metric measures the variation of the first one between projects. Here we are looking for consistency in our performance.
Efficiency describes your ability to deliver quality products within the budgeted/optimal amount of resources.
With efficient processes, the cost is tamed, profitability is high, and the price is competitive.
Superior processes are efficient if they produce little waste. To model efficiency, we propose the following formula:
Where is the amount of waste generated at stage I.
This metric can be used to monitor existing processes. You would typically want to kick off a process improvement exercise if efficiency goes below a certain threshold, or if it exhibits a constant downward trend.
The problem with this formulation is that it uses , a quantity that cannot be easily measured, instead of .
Substituting for won’t solve the problem either; if we do that, we would be overestimating the efficiency of our processes.
6.3 Smooth Operations
When running smooth operations, at no point in time does anyone have to stop and think what to do next, nor do they have to improvise to get things moving again.
In fact, superior processes:
- Prevent deadlocks, this is where two resources can end up in a situation where they wait on each other indefinitely
- Eliminate single points of failure
- Are rigorous enough to cover all types of scenarios, eliminating any need for improvisation
You can identify smooth operations very easily just by asking how many times your manager needs to intervene in your day-to-day routine to get things moving again.
6.4 Process Improvement Should Be Easy
Process improvement is essential if your organization wants to survive amidst constant internal and external pressures.
This means that executing smooth, process-improvement exercises should be easy and non-traumatic.
The top condition for successful process improvement is to have a solid process for just that, with clear roles and responsibilities.
Companies using Six Sigma have dedicated staff (Green Belts and Black Belts) for just that.
Process improvements should result in a measurable net benefit in efficiency over time.
To model this property, we assume that process improvement efforts are factored in .
For process improvement to take place, the efficiency difference of the system over an arbitrary period of time should be positive:
Where is the time after which the process changes have been deployed into production.
While the overhead effort may increase temporarily due to the additional effort investment in process improvement, the net overall benefit inefficiency should be noticeable in the long term.
7. Guide to Implementing Superior Processes
Now that we know what superior processes look like, let’s see if we can come up with some rules on how to properly build and govern them.
7.1 Accountability and Responsibility
The first rule to look at when designing superior processes is perhaps the proper allocation of ownership and accountability.
A designated accountable person(s) for a process is someone who makes sure:
- All the elements of the process are well-defined and documented. If you are using the SIPOC model, these would be the: Supplier, Inputs, Outputs, Customer, internal stages, operational guidelines.
- The process governing the process improvement excercises is established
- Regularly follows-up with accountable staff and initiates discussions on how things can be improved
In terms of responsibility, the person responsible for the process (or a stage in the process) has the following duties:
- Ensure that the process is efficient and that it delivers on its promises.
- Ensures process improvements are executed whenever required.
A few rules of thumb:
7.2 Inputs, Outputs, and Transformations
Every stage of the process should have a well-defined set of inputs and outputs in line with the SIPOC model. These are generally referred to as artifacts.
Artifacts can be:
- Project documents such as project plans
- Technical documentation such as design documents or user guides
- Source code
- Test and automation scripts
Activities that consume inputs and produce outputs are referred to in this context as transformations or operations.
A few rules-of-thumb to follow:
7.3 Utility and Applicability
Superior processes should be useful and applicable in a very wide range of scenarios.
Opportunities, where staff need to improvise or get assistance from management to unblock a situation, should be extremely rare.
Having this in place means less overhead, reduced waiting times, and higher efficiency.
This means that for every different scenario you will have a specific process or sub-process that governs it.
Example of Different Scenario: Project Size
One example where you might want to consider having different sub-processes is project size. The number of documents that you prepare and the templates you use may vary between projects of different sizes. You might also want to apply additional stages to the bigger ones.
A few rules-of-thumb to follow:
7.4 Quality Assurance Checks and Balances
Law number four decrees that there should be no instances where quality problems are allowed to be hidden or passed on to the next stage.
You might want to stay away from sophisticated metrics that measure minute details of every process.
These tools are either reminiscent of bygone ages (Taylorism) or only applicable for large manufacturing businesses.
For small software development teams, they are almost always are counterproductive.
Instead, try to form a culture that rewards swift and immediate resolution of quality problems.
Another potent method to apply is the standardization of processes across your organization.
This will help you ensure that improvements are propagated between different teams and that the results of those improvements are statistically significant.
7.5 Involving Stakeholders
Processes that are meant to be followed by smart individuals need to make sense to them. This is rule number 5.
Therefore, relevant stakeholders must be involved in the decision-making process which led to those processes in the first place.
Beware of Vetocracies
Do not confuse stakeholder involvement with an (ugly) form of democracy, sometimes referred to as vetocracy, where decisions can only move forward if unanimously approved.
7.6 Clarity and Accessibility
Finally, always make sure to publish an Operations Guide with all the details of the processes mentioned above.
This will help everybody get on the same page, provide constructive feedback when required, and make continuous improvements.
8. Further Reading
- The Toyota Way by Jeffrey K. Liker is a must read.
- Understanding Organizational Culture through the works of Edgar Schein is also indispensable.
- The Six Sigma Way — How GE, Motorola, and Other Top Companies are Honing Their Performance is a good read as well.