Operational Excellence in Software Development: A Brief Introduction
I. Introduction
Experience tells us that delivering large software systems is fraught with complexity. The traditional method of dealing with such was a mixture of best practices without theoretical backing and a variety of scientific (sometimes pseudo-scientific) frameworks poorly applied to problems in the field.
In this article, we describe Operational Excellence and how it can help overcome these challenges. But first, let us review the specifications of a successful software company, examine its main challenges, and imagine new solutions.
II. Software Development and the Challenge of Complexity
A. Successful Software Business
The efficient organisation and its integration in a competitive environment.
Customer needs drive product design and production, while environmental changes push production, cost, quality, and sustainability to their limits.
Successful software businesses have the following properties:
If we examine these properties closely, we will notice a striking resemblance to manufacturing and almost any other product-developing organisation. The requirements for a successful service-oriented organisation are also not so dissimilar; aside from the first requirement on quality, which might be rephrased to suit the service context, the remaining three remain the same.
Companies also have their challenges; let’s examine them next.
B. Common Challenges of Delivering Software
Uncertainty has many sources, creating a complex setting in which software development and delivery can easily struggle.
Here are some examples of why being a successful software company is more challenging than it looks.
In Operational Excellence and the Structure of Software Development and Delivery, we examined the complexity of diagnosing and troubleshooting organisational issues. We concluded that superficial solutions wouldn’t do, and successful ones would have to rise to the required level of sophistication and potency.
Luckily, many original thinkers have articulated the assumptions, frameworks, and paradigms upon which those sophisticated solutions should be built. These assumptions, frameworks, and paradigms will form the basis for our approach, which we now describe.
III. A New Framework for Thinking About Software Development
A. People, Processes, and Technology
The solution we propose, which will form the philosophical foundations of Operational Excellence in Software Development as we describe it, has three pillars: A) people, B) Processes, and C) Technology.
Operational Excellence in Software Development revolves around gearing processes, technology, and people towards a common vision and strategy.
The interactions between these three entities in both directions allow complex behaviour to emerge.
More important are the interactions between these three pillars, which we summarize as follows:
B. Theoretical Foundations of the Operational Excellence Framework
These ideas are rooted in the Toyota Production System (TPS), Organizational Theory and Culture, and Complexity and the Social and Natural Sciences. Let’s review some of those principles.
Natural Sciences
Organisations as Complex Systems
In an effort to better understand organisations and their behaviour, organisational theorists created several models, inspired by the leading scientific paradigms of the day.
First, there was the “organisation like a machine” or mechanistic model, followed by Systems Thinkings, Cybernetics, and finally Complex Systems. In between we had Open Systems Theory, the Learning Organisation and a few more.
An organisation comes into being, evolves, develops an identity, a structure, and a purpose, and subsequently disappears. It displays emergent behaviour, is hard to predict and control and is composed of human beings. Thus, it has all the properties of a complex adaptive system.
The Software Value Chain–Infinite Variations on a Single Theme
Optimizing the software delivery process includes identifying where and how business value is generated (the value chain) and understanding the process that allows this to happen (the Software Development Lifecycle or SDLC).
These three pillars—organisations as complex systems, the importance of the human element, and the commonality of the underlying drivers in the value chain—will form the basis of the seven Principles of Operational Excellence in Software Development.
IV. The Route to Operational Excellence
Only one thing seems to be clear just now. It is that program construction is not always a simple progression in which each act of assembly represents a distinct forward step and that the final product can be described simply as the sum of many sub-assemblies.
Understanding what a successful software business looks like and how it should operate requires effort, patience, and, most of all, curiosity. Even when you think you got the facts right, synthesizing those facts into a coherent and logical narrative is fraught with complexity, and deriving the correct insights is rare. Only time can tell how well a job such exercises do.
My personal story started with some questions in two specific areas: software development processes and the structure of software businesses.
Software Development Processes
The first area had to do with software development processes, which differed wildly between teams and organisations. This puzzled me quite a bit as I thought surely it could not all be arbitrary, and there must be some fundamental underlying principles that any software practitioner would accept and apply in the field.
Structure of the Software Business
The second area covered the many relationships between the various units (business, management, operations and support, technolgy) of a software organisation. I was constantly searching for a hierarchy that never existed. Instead, it turned out ot be a dense web of relationships.
Areas 1 and 2 above produced two pressing questions, which I could state as follows:
Or, combined…
Arriving at the destination…
The search for answers to these two questions led me explore the evolution of software development as a practice and of software as an industry. The framework that emerged was Operational Excellence in Software Development.
V. Developing a Framework for Operational Excellence
A. On the Shoulders of Giants
The answers to the two questions above came gradually and from varying and unrelated sources. Some ideas came from engineering courses, while others were distilled from manufacturing, software engineering, business management, and natural and social sciences.
Many of the concepts came from the seminal works of Robert Martin, Martin Fowler, Kent Beck, and a few other experts in the tech industry. However, these pioneers focused mainly on the software development process; the human side of the story, the organizational aspects, and group dynamics were conspicuously missing. This is where equally insightful experts like Peter Drucker, Henry Mintzberg, Ralph Stacey, Edgar Schein, and Dave Snowden rushed to fill the gaps. But that’s not all. The first place is reserved for Jeffrey Liker’s book “The Toyota Way”.
The ideas behind Principles of Operational Excellence in Software Development are a large assemblage of smaller, more varying concepts. Some parts fit together effortlessly, while others had to be refined and reshaped. Below is a summary of each of those sources that contributed the most to the Operational Excellence narrative that we are telling.
B. The Story of Toyota
Operational Excellence has its roots in the automotive industry. It started in the 1950s when post-war Japanese car manufacturer Toyota struggled to catch up with the potency of mass production – a method in which their giant American competitors had a significant edge. Henry Ford deployed the assembly line for the first time in 1913, capable of mass-producing an entire vehicle.
With massive resources and a vast market at Ford’s disposal, there was very little the Japanese automakers could do to compete. The situation, however, changed drastically in the late 1980s, and the Japanese carmakers, especially Toyota, were doing significantly better than their American counterparts. For the next few decades, the Japanese produced exceptionally better cars at highly competitive costs. Their market share kept increasing year on year.
To make that happen, Toyota and other Japanese manufacturers deployed a strategic weapon that would later be known as the Toyota Production System (or TPS). This system was a set of 14 business management principles (articulated fluently in a now classic work by Dr Jefferey Liker) that Toyota lived by. The principles were also supplemented by a philosophy of business management that gave meaning to the organisation’s existence and the work Toyota’s employees were doing.
The Toyota Way is a brilliant book by Dr Jeffrey K. Liker which is a must-read for anyone interested in the greatest ideas of manufacturing and how concepts like Kaizen and Kanban were invented.
As Toyota grew from a Japanese car manufacturer to a global player, it had to promote and apply the Toyota Production System in its overseas factories. To explain TPS in a language that transcended culture, The Toyota Way was introduced. Essentially, The Toyota Way is centred around respecting people and applying continuous improvement.
The Toyota Way and the Toyota Production System were so efficient that professionals from different industries started looking at incorporating its ideals into their businesses. Think of Kaizen, Kanban, and 5S. All three originated at Toyota and carried its trademark. Although, on the surface, car manufacturing bears minimal resemblance to software product development (or services in general), there is enough overlap to warrant a serious discussion, as we have thoroughly discussed in Operational Excellence and the Structure of Software Development and Delivery.
We believe software development can benefit from various concepts successfully applied in the car business. This article will look at how software development and delivery can profit from Toyota’s philosophy and principles, not by repeating the tenets most elegantly narrated in Liker’s book but through practical guidelines that developers can immediately relate to.
Today, Toyota is a Japanese multinational automotive manufacturer with headquarters in Aichi, Japan. It is the largest automobile manufacturer in the world, producing about 10 million vehicles per year.
Two books, “The Toyota Way” and “The Machine That Changed the World,” popularized the story of Toyota and, more precisely, the Toyota Production System (TPS). Both books inspired many tech experts.
C. The Development of Large Software Systems
In 1981, Winston Royce published a seminal paper on the delivery of large software systems. In this paper, he describes the basic steps and challenges of what later came to be known as the Waterfall project management model. If you read this paper, you will immediately notice that some of the fundamental problems associated with software development have already been uncovered, and experts have already started looking for answers.
In 2001, a group of software professionals who had independently invented various ways of dealing with software delivery issues came together and launched the Agile movement. The Agile Manifesto enunciated 12 principles that are now the basis of every software development methodology in this school.
A few years later, when the build and test automation tools improved, we got DevOps. Today, top tech companies use Continuous Integration and Continuous Deployment (CI/CD) practices to deploy production changes daily and hourly. The next revolution will probably be around AI and Large Language Models (LLM).
It is hard to think about anything in software without having all these models, their evolution, and first principles in mind.
D. Individual and Group Behaviour and Psychology
Until a decade ago, my understanding of software development and the management of a software business consisted of parallel narratives occasionally presenting irreconcilable paradoxes. The technical aspect of software development was straightforward. But whenever an individual or a group becomes part of the equation, we immediately move into a more complex domain.
As agents in a complex system, we are influenced by multiple factors, identities, and drivers. Our preferences and attitudes vary with context and constantly evolve through interactions with other individuals. Our behaviour as a group or organisation is unpredictable and almost uncontrollable.
For instance, the following paradoxes were most intriguing:
The key to reconciling these paradoxes lies in the fact that each person has multiple identities (Dave Snowden’s concept of anthro-complexity). These identities flip and switch depending on the current situation. For example, a software developer is also a team member, family member, community member, citizen of a country, an alumni of a particular school, a human with a unique experience and possibly a lot more. The (mostly unconscious) forces emanating from these attractors influence our behaviour in an infinite variety of ways.
VI. Organisation’s Lifecycle and Operational Excellence
Operational Excellence requires a minimum level of organisational stability, where leaders and employees have the time to analyse and rethink their actions. They must also be able to run experiments in continuous improvement and transformations without jeopardizing production or sustaining damaging failures.
A. The Organisation’s Lifecycle
Various management scholars and practitioners have widely discussed and promoted the concept of an organization’s lifecycle and its different stages. Notable thinkers and management consultants were behind these ideas:
B. Critiques of the Organisation’s Lifecycle Model
While the model of organizational lifecycles has been widely discussed and utilized, it is not without its critiques. Here are some common criticisms of this model:
C. Organisation’s Lifecycle and Operational Excellence
This stability is, by definition, not present in startups or organisations in crisis. In startups, power hierarchies, typically dominated by the founders, dictate what can and can’t be done and how the organisation should operate. The main objective of startups is to establish themselves as viable players in the market. Sustainability, through Operational Excellence, is for a later stage.
Companies in crisis have high anxiety levels and have not had the luxury of slow, rational, and deliberate pondering on improvement through Operational Excellence. Their main objective is surviving the crisis.
Discussions on Operational Excellence can be futile at these two extremes: startups and in times of crises.
Therefore, the following prerequisites must be available before Operational Excellence can be conceived as a long-term goal.
VIII. Summary
Let’s recap some of the key ideas of this article.
Now that everything is in place let’s examine the seven principles of Operational Excellence in Software Development.
IX. References
- The Toyota Way – 14 Management Principles From the World’s Greatest Manufacturer — Jeffrey K. Liker
- Strategic Management and Organisational Dynamics — The Challenge of Complexity to Ways of Thinking About Organisations — Ralph Stacey
- HBR at 100 — The Most Influential and Innovative Articles from Hard Business Review’s First Century
- Cultures and Organisations — Software for the Mind — Geert Hofstede, Gert Jan Hofstede
- Organisational Culture and Leadership — Edgar Schein
- The Practice of Management — Peter Drucker
- The Evolution of Civilizations: An Introduction to Historical Analysis — Caroll Quigley