Principles of Operational Excellence in Software Development

I. Introduction

A few years back, and after a decade and a half of curiosity, research, and field experience in software development, I was able to write down a few simple principles that I thought were a great start to understanding Operational Excellence in Software Development. My main aim was to equip myself with a set of principles that could be applied to any software development business or team, regardless of its specificities. Armed with that knowledge, I would overcome any challenges presented in my role as a software engineer.

At that time, I knew very little about the term “Operational Excellence,” which I had heard a few times on separate occasions. Eventually, I stumbled upon the book The Toyota Way, which opened my eyes to numerous previously unknown topics in manufacturing and industrial engineering. A few other encounters, some premeditated, others serendipitous, led me to business management, organisational theory, and complexity theory. These topics, which I previously dismissed as too “imprecise” (I was brought up in the traditional Newtonian mechanistic school), reshaped my worldview.

I tried to distil many of those ideas into “Principles of Operational Excellence in Software Development,” which eventually became a lengthy article. I also thought that one would need to understand how software development processes are organized before one could start analyzing or improving them. This led me to yet another lengthy discussion, which the interested reader can find below under “Further Reading.”

I hope these two articles will be enjoyable for the enthusiastic reader. However, for the busy reader, a more concise and summarized one would be more practical. This article will do exactly that.

II. What Is Operational Excellence?

There will be various definitions of Operational Excellence, depending on the context and the author. The one that resonated well with my thoughts on software development was the following:

Internal pressures arise from the differences in how organisational subgroups (up to the individual level) perceive their interests, roles, and objectives and how aligned these are with the overall organisational vision and goals. They are mostly integration challenges. External pressures come from the market, competition, economic situation, and disruptive technologies.

An organisation that has achieved operational excellence is able to survive in the long term, not only despite those challenges but also by leveraging the opportunities they present.

III. What Is Operational Excellence in Software Development?

While the definition of Operational Excellence can be retained for software development, software development organisations face challenges similar to, albeit more acute, those faced by other organisations in manufacturing or services. Below are some of the reasons why the challenges of software development are unique.

First, technological advancements in software and IT technology are moving at an incredible pace compared with other industries, such as automotive. In the last 70 years, we have had a digital revolution, a dotcom revolution, and now an artificial intelligence (AI) one. Software development organisations unable to leverage those advancements can be swiftly outcompeted and replaced.

Second, there are too many different variations in software development processes (Agile, Waterfall, and everything in between) coupled with the significant number of best practices, tools, and programming languages (and the lack of a unified, solid set of widely accepted principles). Software teams spend an enormous amount of effort in semi-religious wars on which tools and methods are the best. This specific problem also emphasizes the role of the human element in shaping the performance of the organisation and its efficiency and effectiveness.

Finally, there is the problem of scale. Despite what many would say, it is not yet clear how software organisations can grow in size while efficiently coordinating major project deliveries, all running in parallel, competing for resources and talent.

These three issues and others conspire to make software development particularly challenging, and the need for guidelines to achieve operational excellence is all the more vital.

IV. The Path to Operational Excellence

It took me a while to appreciate the human element’s influence on any business endeavour. Once I did, I found myself hard-pressed to catch up on topics of organizational theory and psychology. Groups, of which all organisations are composed, display varying dynamics and behaviour from individuals (as if individual behaviour was not complex enough!).

From there, the paths explored diverged into complexity theory on one end and business management on the other. I must say that succeeding as a software developer in a large organisation will require much more than programming and IT skills.

Once all that research was done, a set of principles crystallized. I found remarkable comfort in the fact that they made sense and did not contradict my intuition. Best of all, they were grounded in natural science and, therefore, had a resilient foundation.

However, I am not at all sure that these principles are universal and will successfully apply in all contexts. On the happy side, the approach itself (deriving guiding principles from natural sciences) will easily cross over boundaries.

Let’s have a look now at what those six principles tell us.

V. Principle 1: Eliminate Cultural Blockers

Organisational culture is the sum of all experiences (good and bad, with traumatic events carrying significantly more weight) experienced by a group sharing a common objective. It is heavily influenced by the group’s founding moments and personalities and is relatively hard to change once it sets in.

Eliminating culture blockers cannot be achieved by reciting platitudes over a deck of slides. Cultural transformations are generally tricky, risky, and anxiety-ridden, during which old, taken-for-granted assumptions are replaced with new ones. The unlearning and relearning process is never easy.

VI. Principle 2: Implement Sound Process Engineering, Management, and Governance

Most of the confusion that we see in managing software development is a result of one or more of the following factors:

All software is built as a unique product. No software company specializes in the production of “standardized reusable components” similar to those observed in the manufacturing of mechanical or electrical parts. Additionally, the technology, development processes, and skills employed in producing the product greatly influence the final design, so the product is the outcome of applying the software design and development process to business requirements. Finally, software development is the only industry where customer feedback is incorporated into the design and development in real-time. In cars, kitchenware, and fashion, the feedback loops are delayed. In software, the feedback loops operate in near-real time.

As such, software development is unique; it combines elements of both unique product production and process production, and this synthesis has significant consequences for the management style applied and the skills required of both managers and employees.

In this context, management and end users have to be close to the developers and must be technical enough to understand and shape the design of the final product. On the other hand, developers must understand the whole ecosystem (products, solutions, end users, operating environment) and cannot focus solely on the application or module they specialize in.

Software development processes must incorporate all these constraints to produce working software that solves business needs.

VII. Principle 3: Draw a Line Below Which Quality Does Not Go

Quality is an umbrella term that encompasses all the desired attributes of a good product. It covers design (user-friendliness, aesthetics, feature richness), build (technology, fault tolerance, service life), performance, and service levels (sustainability, low maintenance cost, good customer support).

Most people appreciate that quality is a weighted sum of all these attributes, and the weights vary between users. For example, some users care more about user-friendliness, whereas others are more interested in performance. So, one must appreciate that when people talk about quality, they don’t necessarily mean the same thing, and this can bring about misunderstandings.

Despite those differences in viewpoints, most people in software development will agree on a common denominator of what software quality means or where to draw the line between what is acceptable and what isn’t. The important thing is that this line is known, published and socialized among all stakeholders. Otherwise, quality can deteriorate slowly but surely until the point of no return, where addressing the technical debt becomes prohibitively costly. From that point on, the product becomes a liability instead of an asset.

VIII. Principle 4: Implement Agile Processes Where Applicable

While Agile may not be suitable for all software projects, many powerful tools were invented, adapted, and refined through its twenty-year journey and are now an integral part of the Agile toolset. These are user stories, backlogs, story points, sprints, and daily planning meetings. In fact, most modern knowledge management systems like Confluence and Jira have all these built-in and available for use at the click of a button.

But that’s only half the story. The major revolution that came with Agile was the semi-giving-up on the massive planning and design phases and the homogenous stages that characterised Waterfall (Analysis, Design, Development, Testing, Deployment…) in favour of an incremental development process that incorporates user feedback by allowing iterations to occur anytime during the process. The human element is built into Agile.

For the two reasons above and a few others, Agile methods and practices are recommended where applicable.

VIII. Principle 5: Focus on Business Value

The objective of a business is to create customers and stay in the game as long as possible. People often talk about generating business or customer value (the distinction is made in a separate article; see the link below). This brings us to the question of what creates business value and ensures the company’s survival. Is it profit, competitive edge, cutting-edge technology, or luck? Is any of these really relevant to the end user? The answer is “Yes, indirectly”.

The net result is better products at lower costs with great benefits to the end users.

The key message here is none of the above (profit maximization, coming first in the competition race, using the latest and most remarkable technologies available) is an end in itself. The goal is always to produce better and cheaper products and make them available to a broader market.

VIII. Principle 6: Use Mature and Proven Technology to Serve Your Customers

In many industries, customer needs and preferences are generally ahead of technology. In most cases, either the technology does not yet exist, is too expensive, or its integration costs are prohibitive. In software, it is often the case (as with GPTs at the time of writing) that the technology exists, but end users may not have fully understood its potential. This phenomenon also extends to tools, frameworks, and programming languages, where developers are inundated with new tech every day.

What does one do under such circumstances? Ignoring new technology might be fatal if the latter is powerful and disruptive enough and could be equally leveraged by competitors. On the other hand, most of these technologies will soon be superseded by even more modern contenders, while only a fraction will survive to dominate the future. Predicting which of these technologies it’s going to be is very tricky.

Furthermore, in some of these cases, the end product might not significantly change if the underlying programming language is replaced. The replacement might satisfy developers for one technical reason or another, but no additional customer value is generated.

When selecting technology to use, it is often wiser to choose the ones with the longest history.

IX. Have We Missed Anything?

Yes, the list can stretch quite a bit more, including communication, leadership, group psychology, norms, culture, organisational behaviour and theory, interpersonal relationships, conflict resolution, resource allocation, and management. So where does Operational Excellence stand on these points, and how do these issues, in turn, affect an organisation’s ability to achieve high execution standards? We will try to answer these questions through a chess metaphor and some basic game theory concepts, specifically on infinite games.

A. Operational Excellence and Middle Game Mastery

In chess, the game can be divided into three main phases: the opening, the middle game, and the end game. Each phase has its own goals and strategies. Organisational development and evolution can be thought of along similar lines.

The opening game is similar to the founding stages of the organisation or group where culture, norms, and processes are created (initial conditions in complex systems parlance). These initial conditions have a long-standing impact on the group’s future development. The goal of the “opening game” in organisational development is to create a great culture, identify your value proposition, hire the right talent, and work on a strategy for growth and development.

Operational Excellence is particularly out of context in the following situations:

What about endgames? The objectives and strategies can vary in chess endgames depending on the position and material balance. They typically include checkmating the opponent’s king, promoting pawns, gaining a material advantage, and keeping one’s king safe. Organisations have no endgames; therefore, the rules of play differ.

B. Short-Term Wins, Operational Excellence, and Infinite Games

In game theory, infinite games extend indefinitely over time, whereas closed games are finite with a specific endpoint. These two types of games differ in duration, strategies, and potential outcomes.

Running a business, especially if you aspire for Operational Excellence, must consider the following topics:

X. Conclusion

Writing down six principles and innumerable ideas, concepts, and theories on how something ought to be done is very different from putting those thoughts into practice. One cannot but feel powerless when contemplating the job’s immensity or the intimidating height of its entry barrier.

This feeling of overwhelmedness puts into question the fruitfulness of such an endeavour and whether people are better off focusing on low-hanging, isolated, and localised objectives in the hope that a global, cumulative, and incremental improvement will become noticeable shortly. As we have seen earlier, this reasoning is behind the concept of “managing in the present,” a fundamental approach to working with complex adaptive systems.

Maybe the Toyota Production System and its version of operational excellence would not have been possible outside of Japanese culture. Maybe Schein’s organisational culture and processes model does not apply effectively outside Western cultures. This line of inquiry again pushes the issues of context and cross-industry learning to the forefront.

While the answers remain open, we have little choice but to hang on to what we believe is a consistent framework (and structure) on how software development should be carried out. Just like democracy became the norm only a few centuries ago, despite its fundamental concepts elucidated more than two thousand years back, we hope the young software industry will adopt a similar framework. Whether it settles down on the six concepts above or new ones altogether does not matter.

As far as this author goes, having a working compass and knowledge firmly rooted in theory and practice is crucial. This knowledge provides meaning to our actions and allows us to make sense of our experiences while the compass guides our future steps. We hope the reader has found similar insights into those ideas.

XI. References

• The Toyota Way – 14 Management Principles From the World’s Greatest Manufacturer, Jeffrey K. Liker
• Strategic Management and Organisational Dynamics (4th edition), Ralph Stacey
• Organisational Culture and Leadership, Edgar Schein
• Process Consultation: Its Role in Organisational Development, Edgar Schein
• Thinking Fast and Slow, Daniel Kahneman
• The Six Sigma Way, Peter S. Pande, Robert P. Neuman, Roland R. Cavanagh.
• HBR at 100 — The Most Influential and Innovative Articles from Hard Business Review’s First Century, Harvard Business Review
• The Practice of Management, Peter Drucker
• Mintzberg on Management: Inside Our Strange World of Organizations, Henry Mintzberg