Operational Excellence and the Structure of Software Development and Delivery

1. Overview

I wrote the first article on Operational Excellence in Software Development in 2020. Since then, the original (and crude) ideas have become more lucid and crisp, and it’s time for a rewrite. In this two-part series, we’ll discuss the theory and principles of Operational Excellence in Software Development, a modern guide to understanding software development and the people involved in it.

This article (Part 1) will delve into the theories behind the principal ideas of Operational Excellence, exploring some critical junctions in the history of software development, why IT projects often fail, and the software delivery value chain. We’ll also examine the influential drivers in software delivery businesses and discuss the cultural transformations required for success. Then, we provide a comparative analysis of the manufacturing and service models and why the latter is more suited to software businesses, highlighting the benefits of a theory-informed practice. With these principles in mind, software technicians and professionals can appreciate the forces driving software delivery and software teams.

Part 2 will then elaborate on the six principles of Operational Excellence, focusing mainly on Software Development and Delivery.

This article series provides practical guidance for businesses seeking to optimize their software development and delivery processes. Organizations can drive innovation, improve quality, and deliver value to their customers by understanding the theory and principles of Operational Excellence in Software Development.

The main themes of this discussion are as follows:

  • Fundamental Principles of Organisational Theory
  • The Software Delivery Value Chain
  • The Different Schools of Business Management
  • Leadership and Transformation
  • The Manufacturing vs Services Models
  • The Evolving History of Software Development

2. Software Development: A Brief History

Understanding the current state of software development requires a (brief) inspection of the historical trajectory it has followed since its inception in the early 50s. Significant technological breakthroughs dot this trajectory. Closely following these advancements are noticeable changes in customer (or end-user) behaviour and preferences, fuelling more changes and bringing about further innovations.

These intertwining historical threads (technological advancements and changing customer preferences) will allow us to understand the current trends, practices, and mainstream thoughts in software development. More importantly, they will provide some context to the industry’s present challenges, although these should be examined under anthropological and psychological lenses to fully appreciate their significance.

2.1 A Timeline of Major Milestones in Software Development

The following is a brief history of software development with some of the most significant milestones, technological advancements, and people involved:

Timeline of the major milestones in software technology
Timeline of the major milestones in software technology
  • 1950s: The first programming languages, such as Fortran and COBOL, were developed. John Backus led the team that created Fortran, the first high-level programming language. Meanwhile, Grace Hopper was working on developing COBOL, which became the most widely used business language for decades.
  • 1960s: The concept of software engineering emerged with the publication of the book “Software Engineering” by F.L. Bauer, C.W. Bachman, and others. In addition, the first operating system, Unix, was developed by Ken Thompson and Dennis Ritchie.
  • 1970s: The personal computer revolution began with the introduction of the Altair 8800, one of the first personal computers. Bill Gates and Paul Allen founded Microsoft, one of history’s most influential software companies.
  • 1980s: The graphical user interface (GUI) became popular with the introduction of the Apple Macintosh and the Microsoft Windows operating system. The object-oriented programming (OOP) paradigm was also developed with C++ and Smalltalk.
  • 1990s: The internet became widely available to the public, leading to the development of the World Wide Web and web development technologies such as HTML and JavaScript. The open-source software movement gained momentum with the creation of the Linux operating system by Linus Torvalds.
  • 2000s: The rise of mobile computing led to the development of new platforms such as iOS and Android. Agile software development methodologies gained popularity with the publication of the Agile Manifesto in 2001. Cloud computing became more widespread with the introduction of Amazon Web Services and Microsoft Azure.
  • 2010s: The advent of big data and artificial intelligence (AI) led to the development of new technologies such as Hadoop and TensorFlow. DevOps, a methodology emphasising collaboration between developers and operations teams, became popular. The blockchain technology behind Bitcoin also gained attention, leading to the development of new cryptocurrencies and blockchain-based applications.

The above timeline is a high-level summary of the major events that formed and shaped the software industry. Countless individuals and organizations contributed to its ascent. It continues to evolve rapidly as new technologies and approaches are developed. Despite being a relatively young field, software development has brought products and applications to every aspect of our lives.

2.2 Dealing With a Changing Customer Base

The customer base for software products has undergone a significant transformation over the years, which has had a major impact on the industry. The following are some of the major historical bifurcations:

  • Large and tailored software applications: In the early days of software development, customers were typically large corporations or government agencies with the financial resources to invest in custom software applications. These applications were often highly specialized and tailored to the needs of a specific organization.
  • Rise of the personal computer: However, with the rise of personal computers in the 1980s and the internet in the 1990s, the customer base for software products began to expand dramatically. This was partly driven by the increasing affordability of hardware and the democratization of computing power, which enabled more people to access and use software applications.
  • Internet and the globalisation of the software market: The widespread Internet adoption in the 1990s and 2000s brought about further changes as companies began to develop software products for a global market. This shift was fueled by the growth of e-commerce and the increasing importance of online services, which led to the development of new software categories, such as web browsers, email clients, and search engines.
  • Smartphones and mobile apps: The rise of mobile devices in the 2010s has also had a significant impact on the software industry, as companies have had to adapt their products and services to meet the needs of mobile users. This has led to the development of mobile apps, which have become a significant revenue source for many companies.

Overall, expanding the customer base for software products has profoundly impacted the industry, driving innovation and competition and creating new opportunities for companies of all sizes. However, it has also brought new challenges, such as the need to develop software that is accessible, user-friendly, and compatible with a wide range of devices and platforms. As the customer base for software products continues to evolve, the industry will need to remain agile and adaptable to stay competitive.

2.3 An Ever-Changing Landscape of Challenges

As the software industry has evolved, so too have the skills, education, expertise, organization, project management and delivery methodologies required to succeed. In the early days of software development, many programmers were self-taught or learned on the job, and there were few established best practices or standards for development.

  • As the industry matured, however, a growing need for specialized expertise in software architecture, design, and testing emerged. This led to the development of formal training programs and certifications in these areas, adopting industry standards and best practices and introducing the Software Development Lifecycle or SDLC.
  • Organizational structures also evolved, with companies developing more specialized teams focused on specific areas of software development, such as front-end development, back-end development, or quality assurance. The rise of Agile methodologies in the 2000s brought about further changes as companies sought to break down silos and promote cross-functional collaboration and communication.
  • Interoperability and platform proliferation have also become increasingly important issues in software development. With the rise of mobile devices and the Internet of Things (IoT), developers must be able to create software that can operate on various platforms and devices while also integrating with other software and systems.
  • Finally, project management and delivery methodologies have evolved to meet the changing needs of the industry. Agile methodologies such as Scrum and Kanban have become increasingly popular, as they offer a more flexible and iterative approach to development that can help teams to adapt to changing requirements and priorities.

The changing nature of the software industry has required developers to become more specialized and knowledgeable while promoting cross-functional collaboration and communication. The industry has also placed a greater emphasis on interoperability and platform proliferation, as well as adopting agile project management and delivery methodologies.

3. Troubleshooting Organisational Behaviour

3.1 Organisational Strategies and Behaviour

To respond effectively to organisational challenges, we must first comprehend organisational behaviour. Organisational psychologists have developed frameworks to aid this understanding over the past few decades. These frameworks include:

While software engineers may not see the immediate relevance of these theories when dealing with the challenges of upcoming releases, it is crucial to recognize their importance. This is especially true when considering the topic of IT project failure, which will be explored in the following sections. It will become clear why simplistic approaches to this issue will not suffice and why more sophisticated interventions are required.

3.2 Why IT Projects Fail

IT projects have become vital to modern business, enabling organizations to streamline processes, improve productivity, and enhance customer experiences. However, despite the widespread use of technology, IT project failures are still prevalent. The consequences of these failures can be disastrous due to poor planning, unrealistic expectations, or lack of communication, resulting in financial losses, damaged reputations, and missed opportunities.

What puzzles me is not the event (IT project failures) itself, by why it has become so unsurprising in the industry and why professionals have become so accustomed to it almost to the point of resignation. The answer to this question lies heavily in the complexity of business management issues and the immense challenges of turning the ship around.

To illustrate this complexity, we will do the following exercise. In the next two sections, we will try to uncover the underlying issues causing IT projects to fail widely. The aim is not to pinpoint the causes per se but to expose their intricacy, depth, and interconnectedness.

3.3 A Brief (and Potentially Shallow) Analysis

List of major causes of IT project failure you can expect to find online
List of major causes of IT project failure you can expect to find online

If you ask Google or ChatGPT about the “Top reasons for IT failure”, you will get an answer resembling the following:

  1. Poor Planning, which translates into:
    • Inadequate project management
    • Lack of clear goals and objectives
    • Inadequate resource allocation
  2. Poor Communication, often manifesting as:
    • Inadequate communication between stakeholders
    • Inadequate communication within project teams
    • Communication barriers due to language and cultural differences
  3. Technical Issues, summarised by:
    • Inadequate technical expertise
    • Technology incompatibility issues
    • Technical infrastructure issues

But if this diagnosis is already made and widely accepted, why can’t we proceed to the resolution? The answer lies in the multiplicity of interpretations and the depth at which the analysis might be made. Causes interpreted at different levels or meanings will lead us to different solutions. This proliferous analysis will induce conflict as stakeholders support whichever version of the solution they favour (or that which causes them the least damage).

To investigate these ideas further, let’s continue with this exercise. We will use the Five Whys method to investigate one problem, particularly the “Inadequate technical expertise”.

3.4 More Subtle Reasons

Using Inadequate technical expertise as an example, here are some hypotheses that can explain the lack of talent in an organisation:

  • Not hiring the right skills for the job. For example, focusing too narrowly on technical skills and ignoring cultural fit, communication skills, critical thinking, and team play.
  • Insufficient training and development opportunities for recruits.
  • Poor leadership and management, thus leading to high turnovers among the staff.
  • Limited career advancement opportunities where ambitious and talented people cannot be retained.
A Root-Cause-Analysis of the lack of technical skills in an organisation can be examined at many levels with potentially many causes on each one. The level and depth of the analysis will dictate the solution.
A Root-Cause-Analysis of the lack of technical skills in an organisation can be examined at many levels with potentially many causes on each one. The level and depth of the analysis will dictate the solution.

Again, the idea here is not to develop a complete list of reasons why a particular project is not adequately resourced but to make two observations:

  • Observation 1: A multiplicity of factors must be considered in any “Inadequate technical expertise” discussion.
  • Observation 2: There is no reason to stop at this (second) level; going one or two more levels deeper may uncover deeper underlying causes, whereas the ones listed above might only be symptoms.

To illustrate Observation 2, let’s take Insufficient training and development opportunities and see if we can break it down further. Here is a list of candidates:

  • Lack of prioritization for training and skills development. This can manifest in poor training material and the allocation of experienced resources solely to ongoing projects.
  • Misalignment with business goals. For example, and especially for contractors hired for short durations, it’s more important to get the job done rather than train the contractor.

The tree we have now built has many branches, each of which can be a potential cause or symptom of a, yet again, deeper underlying issue. At this stage, we can see the following questions arise:

  • Question 1: Is the current depth of our investigation enough, or should we dig deeper?
  • Question 2: If we accept that some of the issues listed are indeed causes, what would their relative weights and influence be?
  • Question 3: What is the optimal strategy for resolving these issues regarding resource allocation and choice between alternative paths?
  • Question 4: What can be learned from these failures, and how can we adapt our processes to anticipate future issues before they occur?

The (practical) answers to these questions are probably for a different article (in Part 2, we will examine the six Principles of Operational Excellence in Software Development, which will shed some light on one or two aspects). Here, we will focus on the question itself: how to understand organisational problems and what solutions can be applied.

3.5 Organisations as Complex Adaptive Systems

According to this Newtonian view of the world, humans will ultimately be able to dominate nature. […] That thinking is based on the belief that managers can, in principle, control the long-term future of a human system. […] These theories say that when a deterministic nonlinear system moves away from stable equilibrium towards a state of explosive instability, it passes through a stage of bounded instability in which it displays highly complex behaviour.

— Ralph Stacey, Strategic Management and Organisational Dynamics — The Challenge of Complexity to Ways of Thinking About Organisations

The following is a list of fundamental principles and observations that will allow us to answer “how to understand and act in an organisation” (or how to achieve Operational Excellence in software development teams) by modelling organisations as Complex Adaptive Systems. This list should hopefully dispel some old myths and replace them with new insights.

  • Complex domains: You know you are operating in a complex system when multiple competing hypotheses can all be supported by evidence. In our example above on IT project failure, subsets of the listed issues can be formed to (partially or fully) explain the failure. This proliferation of symptoms and causes, all readily supported by organisational stakeholders, presents a compelling argument for adopting this view.
  • The appearance of positive feedback loops. Some issues appear as symptoms and later on as causes. For example, a lack of resources leads to deprioritization of training and development, leading to recruits not performing quickly and efficiently and aggravating the resourcing problem. In Complex Adaptive Systems, interactions between agents and their outcomes do not cancel out on average but accumulate and reinforce each other.
  • Material causality: The multitudes of influential factors involved blurs temporal and material causality. Solid lines connecting causes and effects in complex systems do not exist. Instead, we have a summation over many drivers and a dependency on previous system states. This presents two problems for organisational strategists: First, the outcome of actions cannot be readily isolated and measured, and second, all actions will have unintended consequences.
  • Leadership is part of the problem: Organisational strategists are rarely the all-powerful objective observers who can dictate the long-term evolution of organisations. They are often deeply embedded in the organisation, and innumerable past experiences colour their views. We now have the following paradox: how can a problematic system fix itself? Managers often hire external consultants to provide an outsider’s (hopefully more objective) views. However, external consultancy often has its own problems. For example, external consultants have little or no “skin in the game”. They also may not fully appreciate the intricacies of the cultural issues within an organisation to the scale of the technical issues involved.
  • Constraints and interactions: Organisations are not machines in the Newtonian sense. This has profound implications on how they can be managed. Machines can be driven by manipulating their input parameters. From there on, the (deterministic) laws of dynamics will dictate their future evolution. On the other hand, complex systems are driven by constraints (through which order and self-organisation emerge from chaos) and interactions among their typically large number of agents. During evolution, complex adaptive systems change not only their states but also their dynamics. Tiny fluctuations have radical outcomes, making evolutionary paths impossible to predict (and consequently manage long-term).

Eminent thinkers like Dave Snowden and Ralph Stacey have provided alternative tools (such as the Cynefin framework or process thinking) for managing complexity. Dave Snowden has fantastic lectures online, while Stacey’s seminal textbook Strategic Management and Organisational Dynamics — The Challenge of Complexity to Ways of Thinking About Organisations is a must-read.

4. The Software Delivery Value Chain

Value generation occurs when end users realize what technology (automation, fast computation, mechanization) can do for them. Once a solution is prototyped or deployed in the field, end users start interacting with it, spotting new opportunities they never thought of before, subsequently altering their preferences and creating new business needs. This cycle drives value generation.

The value generation cycle with technology and end users locked in a closed loop.
The value generation cycle with technology and end users locked in a closed loop.

4.1 Software Delivery and Value Creation

It’s impossible to understand software delivery without a solid grasp of customer and business value creation processes. The mainstream view of value creation rest on the following assumptions:

  • Business needs: Technology can assist businesses in improving productivity through the mechanisation and automation of production processes, enterprise data storage and management, and fast computation. Business needs dictate what software applications will be developed and what they will look like.
  • The Software Development Lifecycle: Businesses use the Software Development Lifecycle (SDLC) to manage the production processes that create working software. The SDLC consists of sequential stages (allowing for iterative cycles where necessary) where raw input (business needs) is fed into the system to produce software that can be deployed and operated in the field. Each SDLC stage adds value to the work-in-progress product, although the size and relevance of some of the SDLC stage’s contributions can be debated. For example, Dr Winston Royce (famous for his paper on Waterfall project management) maintains that only analysis and development (the primary activities) actively contribute to the creation of business value. In contrast, design, testing, and operations, for example, are secondary and only exist to support the primary processes.

4.2 Optimising the Value Chain

Optimising the value chain requires an in-depth understanding of the value-generation process and the key stakeholders who can support or obstruct your plans.
Optimising the value chain requires an in-depth understanding of the value-generation process and the key stakeholders who can support or obstruct your plans.

The debate on the primacy of certain stages regarding value creation is fundamental to process optimisation. Its outcome will determine how you optimise your value chain for efficiency and maximum productivity by selecting the stages that need to be expanded, reduced, automated, or optimized.

The debates also cover the optimisation of primary stages or what the analysis, design, and development processes should look like. The following questions illustrate the point:

  • How effort estimation is conducted and improved
  • How much should developers are involved in other SDLC stages
  • Which technology stack to use
  • How much unit testing is sufficient
  • Unit vs System Integration Tests

Requirement gathering and definition is also a topic for optimisation. These SDLC stages are trickier and more complex as they involve end users instead of solely technicians. For example:

  • The definition of requirement readiness
  • Developers interacting with end users
  • Should developers and/or end users drive the design process
  • Dealing with requirement volatility
  • Creating reusable software

The answers to these questions are not universal but context-specific. However, fundamental principles and best practices can be identified to aid in selecting the best project delivery methodology (Agile, Waterfall, DevOps), for example.

4.3 Politics, Psychology, and Optimisation

Office politics play a crucial role in driving organisational behaviour. Aside from the common motives for increasing one’s power and prestige vis-a-vis one’s peers, the incentives and rewards system is another influential moderator of behaviour. Consider the following quote from Conway’s article:

A manager must subcontract a crucial and difficult design task. He has a choice of two contractors, a small new organization that proposes an intuitively appealing approach for much less money than is budgeted and an established but conventional outfit asking a more “realistic” fee. He knows that if the bright young organization fails to produce adequate results, he will be accused of mismanagement, whereas if the established outfit fails, it will be evidence that the problem is indeed a difficult one.

— Melvin Conway, How Do Committees Invent

What it’s saying is that, more often than not, people will choose a conventional, safe, but suboptimal solution over a radically different, potentially risky, but promising solution. This can be readily observed today if a minority of developers might, for example, argue against an Agile methodology in a context where Agile can be shown to be suboptimal. Instead, job security considerations dictate that Agile is the best there is.

Power relations will also influence which teams or departments have adequate budgets, resources, and attention to effectively conduct optimisation exercises. Organisational politics do not always reflect the true requirements of the situation. Teams with no champion for their cause might not get the right headcounts, budget, or tools they need to perform their job adequately.

Most design activity requires continually making choices. Many of these choices may be more than design decisions; they may also be personal decisions the designer makes about his own future. As we shall see later, the incentives which exist in a conventional management environment can motivate choices which subvert the intent of the sponsor.

— Melvin Conway, How Do Committees Invent

Here also, the problem diagnosis (i.e. finding the source of inefficiencies) will determine which solutions will be applied. In most cases, technicians will endeavour to downplay their inefficiencies while emphasizing those of other departments.

Optimising the value chain is strategically important and must be data-driven and applied regularly and consistently (much like hygiene).

5. The Impact of Tools and Technology

5.1 Business and People First

It might seem absurd to the reader who has followed us thus far that a discussion about software development has not yet discussed programming languages, frameworks, automation, or any tech-related topic. The reasons are quite straightforward:

  • Principles and guidelines on Operational Excellence in Software Development must be technology-agnostic and have an element of universality. Otherwise, their validity will be severely restricted, and their application will not scale
  • Although technology severely impacts many aspects of a business and the efficiency and productivity of its processes, it is of secondary importance when it comes to business needs or people. Therefore, any discussion on technology will have to be under those two lights.

In the following two sections, we will discuss technology and tools, not from a technical perspective, but from a delivery one. Instead of listing and comparing two programming languages or algorithms, for example, we will focus on what aspects of technology drive business decisions and what the implications of this are.

5.2 Technology Serving the Business

The following few paragraphs are centred on the following idea: Any technology discussion without referencing time and organisation scales is futile. For example, technology is rarely influential in small initiatives with a limited scope since the tools and software products involved are not being questioned. Most of the time, it’s either an integration project, a change request, or a new feature with a predefined tech stack.

Let’s distinguish three contexts and see how technology drives decisions in each.

  • Short time scales: As mentioned, technology in the short term should not be very relevant. For small projects and short terms, people generally work with what they have, and the focus is on delivering projects rather than strategic business objectives.
  • Mid-range time scales: Tech decisions like migrating to a new framework, upgrading your infrastructure, or redesigning core product components have a mid-term business impact as they distract resources from delivering capitalizable business features. The core idea here is that technology should serve the business (and its clients) and not vice versa. Technicians generally forget this basic concept and spend plenty of time chasing the latest trends without building a business case for their proposed upgrade project or considering the long-term viability of the new framework.
  • Long-range time scales: Technology has a strategic impact at longer time scales and should be considered carefully. The Kodak story is known all too well. Disruptive Technologies — Catching the Wave is an influential article published in the Harvard Business Review in 1995, where its authors discussed disruptive technologies and proposed methods for dealing with them. One of the insights presented in this article was how organisations optimise their structures and processes for a given customer base, product, and technology, making it near impossible for them to accommodate alternative products. This view aligns with Dave Snowden’s apex predator theory on the product lifecycle.

5.3 Tools Working For (and Not Against) You

The DevOps age ushered in an unprecedented deployment of sophisticated tools aiming to automate many aspects of repetitive tasks where manual interventions never add any value. This change has brought in immense benefits such as:

  1. Build automation of complex projects
  2. Automated unit and system integration testing
  3. Automated deployment and monitoring

Points 1 and 2 allow what is now called Continuous Integration (CI), while point 3 allows for Continuous Deployment (CD). Continuous Integration and Continuous Deployment combined allow businesses to achieve Continuous Delivery, a software delivery methodology whereby new features are deployed on production daily.

Types of cost incurred in maintaining sophisticated tools and infrastructure to support software development and delivery
Types of cost incurred in maintaining sophisticated tools and infrastructure to support software development and delivery

What about the costs associated with this software delivery method? Here is a list of the top contributors (in no specific order):

  • Licensing costs: Many tools (IDEs, databases, sourcecode versioning tools, code repositories, monitoring tools, ticketing systems, and project management tools) that have become indispensable to implementing a capable CI/CD pipeline have high licensing costs and form a significant part of the IT budget. The next item in an IT budget is probably headcount, although this varies heavily based on the geographical location of your development teams. In countries where hiring great talent is expensive relative to the tools being used, investing in tools that would enhance a developer’s productivity, even by a small margin, makes a lot of sense.
  • Running costs: Sophisticated as they are, IT tools and the infrastructure they are deployed on require constant maintenance and investments to keep them running. There are two costs associated with running a large and complex IT infrastructure: first, the direct cost from licenses, maintenance, and support, and second, the hidden cost associated with delivery delays when infrastructure issues occur. Development teams have come to depend heavily on their tools, and the developers are usually blocked when these are unavailable. The second part of the cost is particularly challenging when the tools slow down gradually (as they certainly will without proper maintenance). In such cases, revamping their capabilities requires significant effort and knowledge, much like refactoring legacy code.
  • Skill acquisition and retention: Running a DevOps (CI/CD) infrastructure requires specialized knowledge and skills, such as database, network, and system admins, in addition to expertise in the tools used. As IT infrastructures grow, these skills must be organised and structured (yet another silo) for efficiency, and constant collaboration with developers must be maintained. While many tools are now offered as services in the cloud, expertise remains an ongoing requirement.

The complexity involved in maintaining an up-to-date continuous delivery pipeline is quite high and must be managed accordingly. There is no point in owning a car that breaks down every 100m; in this case, walking or riding a bicycle would be much more efficient.

6. Business and People Management: Three Schools of Thought

6.1 Overview

One More Time: How Do You Motivate Employees? is an influential article by Frederick Herzberg published in the Harvard Business Review in 1968. The title is a bit misleading as the article offers insights well beyond employee motivation and is, therefore, a highly recommended read. Surprisingly, while quite dated, its ideas remain as relevant today as they were in the 60s, and for a simple reason: the nature of the relationship between the organisation and its managers on the one hand and employees on the other changes slowly.

The following sections discuss three schools of thought on managing people: Organisational Theory, Industrial Engineering, and Behavioural Science Theory.

6.2 Organizational Theory

6.2.1 Overview

The organizational theory school of thought focuses on an organisation’s formal structures and processes. It emphasizes the importance of clear lines of authority, defined roles and responsibilities, and standardization of processes to achieve efficiency and effectiveness. In terms of people management, this school of thought strongly emphasises hierarchy and the control of individual behaviour through formal rules and procedures.

6.2.2 Merits

  • Clarity and structure: Organizational theory provides a clear and structured approach to managing people, which can be effective in certain situations.
  • Consistency and fairness: This school of thought emphasizes the importance of formal rules and procedures, which can help ensure consistency and fairness.
  • Scalability: Organizational theory can be helpful in large organizations with many employees, as it provides a clear hierarchy and lines of authority.

6.2.3 Critiques

  • Stifles innovation: It can create a rigid and inflexible environment that is not conducive to creativity or innovation.
  • Rules over outcomes: This may lead to focusing on following rules and procedures rather than outcomes.
  • Autonomy and job satisfaction: This can lead to a lack of autonomy for employees, which can negatively impact motivation and job satisfaction.

6.3 Industrial Engineering

6.3.1 Overview

Industrial engineering focuses on optimising organisational processes and systems. It seeks to identify ways to improve efficiency and reduce waste to increase productivity. In terms of people management, this school of thought views employees as resources to be utilized to achieve maximum productivity. The focus is finding the most efficient way to get work done, often through technology and automation.

6.3.2 Merits

  • Efficiency and productivity: Industrial engineering emphasizes the importance of efficiency and productivity, which can lead to cost savings and increased profitability.
  • Lean principles: Encourages technology and automation to improve processes and reduce waste.
  • Ideal for manufacturing: It can be helpful in manufacturing or production environments where efficiency is a key priority.

6.3.3 Critiques

  • Results over people: This can lead to a focus on outputs rather than the well-being of employees. It can create a culture of overwork and burnout if not managed properly.
  • A mechanistic view: May undervalue the importance of human factors such as emotions, motivations, and relationships in the workplace.

6.4 Behavioural Science

6.4.1 Overview

The behavioural scientist’s school of thought takes a more humanistic approach to people management. It emphasizes the importance of understanding individual behaviour and motivation and the social and cultural context in which work is performed. This school of thought emphasizes empowering employees, creating a supportive work environment, and fostering positive relationships between managers and employees. The goal is to create a workplace where employees are motivated to perform at their best and can achieve their full potential.

6.4.2 Merits

  • People over outcomes: Emphasizes the importance of individual motivation and well-being, which can lead to increased job satisfaction and engagement.
  • Innovation and creativity: Encouraging a collaborative and supportive work environment can increase creativity and innovation.
  • Humanistic approach: Recognizes the importance of social and cultural factors in the workplace.

6.4.3 Critiques

  • Not ideal for all cultures: It can be challenging to implement in organizations emphasising hierarchy and control.
  • People over results: It may be less effective in environments where efficiency and productivity are key priorities.
  • Result quantification: It can be challenging to measure the effectiveness of these approaches.

6.5 Choosing a Management Style

Several factors can determine which of the three schools of thought (Organizational Theory, Industrial Engineering, or Behavioral Science) is applied in an organization and at what ratio. Here are some of the critical factors:

  • Organizational goals and priorities: The goals and priorities of an organization can play a significant role in determining which school of thought is applied. While most, if not all, organizations seek to maximize their shareholders’ profit (and therefore maximize efficiency and productivity), the path to those objectives lies in hiring, training, and retaining skilled personnel. This is especially true in industries where possessing industry and domain knowledge is key, and software is one of those industries. In high-tech organisations, employee satisfaction is an influential factor and a mix of Organisational Theory and Behavioral Science can be employed, sometimes at the expense of Industrial Engineering.
  • Organisational culture: An organisation’s culture can also influence which school of thought is applied. For example, a strong culture centred around technology might focus more on the engineering side and pay less attention to people’s attitudes or business growth. Cultures that prioritize hierarchies and processes over people might be more inclined to use Organisational Theory and Industrial Engineering concepts over anything else.
  • Industry and sector: The industry and sector in which an organization operates can also influence which school of thought is applied. For example, manufacturing organizations may be more likely to adopt an industrial engineering approach, while service organizations may be more likely to adopt a behavioural science approach.
  • Leadership style: The leadership style can also play a role in determining which school of thought is applied. Leadership styles vary widely between managers, departments, regions, and levels. Preferences, background, education, value system, and culture strongly influence whether a leader is rational, altruistic, or assertive. These attributes and a particular incentive system determine the leader’s management style.
  • Employee demographics and preferences: The demographics and preferences of employees can also play a role in determining which school of thought is applied. For example, younger employees may prefer a more collaborative and flexible work environment, which may align more with a behavioural science approach.
  • Organization size: Large organizations with complex structures and many employees may be likelier to adopt an organizational theory approach, emphasising formal rules and procedures, a clear hierarchy, and centralized decision-making. These elements can help ensure consistency and coordination across the organization and manage complexity. On the other hand, smaller organizations with fewer employees may be more likely to adopt a behavioural science approach, which emphasizes individual motivation and well-being, collaboration, and adaptability. This is because smaller organizations often have a more flexible structure and fewer formal rules and procedures, allowing for a more personalized and collaborative approach to management.

Organisations and leadership must balance two conflicting forces: maximising profit while keeping employees engaged and motivated. Although these requirements are not always mutually exclusive and do not necessarily come at the expense of one another, poor management can undoubtedly make it so.

6.6 The Management Style Triad

management style triad
The management style triad

Imagine a triangle with three corners representing the Organizational Theory, Industrial Engineering, and Behavioral Science management schools of thought. We can locate a specific leadership style by placing a point inside this triangle. Dave Snowden advocates using this triangle to assess and enhance leadership rather than relying on the conventional, hypothesis-based questionnaire, an exercise that people can easily game.

We label the vertices of this triangle as follows:

  • A: Organisational Theory. Style is assertive, pragmatic, and results-oriented.
  • B: Behavioral Science. Style is empathetic and altruistic, placing people at the top.
  • C: Industrial Engineering. Style focused on rational decision-making, prioritizing processes and results over people or innovation.

Ideally, we want the leadership and management style to be balanced, i.e. equidistant from the three vertices. That, however, might not be the optimal state of affairs. Consider the following situations where a tilt towards one vertex or another can provide considerable advantages.

ContextLeadership and management stylesWhy will that work
Crisis ManagementAssertive and pragmatic leadership, able to go against basic assumptions. Takes immediate action to restore order.People are happy to conform to restore order and predictability and reduce anxiety.
Strategic, Long-Term PlanningSlow and rational decision-making processHere assertiveness will backfire, and people will rarely cooperate.
Business-as-Usual (BAU)Rational, fair, and process-orientedThis would be the best time to apply incremental improvements to leverage lessons learned.
Disruptions and TransformationsAssertiveness, clarity, visionary, empathyHigh anxiety levels typically accompany major transformations.
Four different contexts where varying leadership and management styles can apply.

Let’s review these in more detail:

  • Crisis Management: In times of crisis and turmoil, we would expect leadership to be more assertive and pragmatic, able to reconsider rules and basic assumptions when they stop working, and take immediate and effective action to restore order. This can be justified by assuming that people are happy to suspend their freedom for a short duration to restore order and predictability and reduce anxiety.
  • Strategic, Long-Term Planning: When strategic issues are being considered with long-term impacts on the business and the employees, people generally prefer a slow and rational decision-making process where their concerns can be heard and considered. In this context, assertiveness will backfire, and people will rarely cooperate.
  • Business-as-Usual (BAU): During regular periods of medium workloads and little anxiety, people might prefer rational, fair, and process-oriented leadership and management styles. They expect operations to be smooth and productivity high. This would be the best time to apply incremental improvements to leverage lessons learned. Process engineering can be safely applied since anxiety is typically low in such situations.
  • Disruptions and Transformations: Although there might be no crisis in full development, changing times (mergers, acquisitions, new management, transformation efforts, major technological disruptions) are typically accompanied by high anxiety levels. Assertiveness in such situations might be welcome if people are informed and involved. Emotions are usually high during changing times, and an altruistic stance will be reassuring.

6.7 Leadership and Technical Proficiency

One of the drawbacks of using the triad from the previous section is that a point in the middle of the triangle can mean several things. It tells us that, on average, all three management styles are applied, but it does not tell us whether they were applied at the right time or whether actions and decisions were random and neurotic.

To avoid this problem, technical proficiency might be a prerequisite for the triad to work. This means that, at a bare minimum, management must fully understand the technical and commercial aspects of the business. It must also seek to understand the people, the processes, and the work involved in producing the organization’s products or services.

Erratic, ad-hoc decisions or frequent backpedalling are usually signs of technical deficiency, panic, or both.

While it is not always necessary for management to be technically proficient in every aspect of their business, having a good understanding of the technical aspects of their industry can be beneficial.

Managers who possess technical skills can better understand the intricacies of their business operations and be more effective in making technology, innovation, and product development decisions. They can communicate more effectively with their technical staff and make informed decisions about investment in new technology or IT infrastructure.

7. Managing Organisational Change

7.1 We Understand the Problem, Why Is It So Difficult to Change?

The answer to this fundamental question on transformation and change can be articulated by examining a textbook theory of change, such as John P. Kotter’s model. In summary, the success of Kotter’s model relies on the following assumptions being held for all transformations:

  • A sense of urgency can be established among the powerbrokers to induce them to act.
  • Enough influential people can unite in a coalition to champion and lead the change.
  • A vision of the (somehow distant) future can be articulated.
  • A majority of key stakeholders will commit to the vision.
  • Empowered people involved in the transformation will have the same agenda and expectations as the authors of this vision and will work towards it.
  • An organisation fully geared and optimised to work in a specific context can be reengineered to work differently.

As ancient wisdom goes, there are many ways to go down but only one way to go up, and in this case, it’s enough to invalidate one of these six assumptions before the entire edifice comes tumbling down.

In the famous article Disruptive Technologies: Catching the Wave, authors Joseph Bower and Clayton Christensen recommend that organisations investing in potentially disruptive technologies create a separate organisation to deal with new products, markets, and customers and keep the old and new organisations independent. In other words, they seem to be saying that transforming an old organisation so that it simultaneously sells two potentially competing products, an old and new, is extremely challenging. Here, the best approach is to let the sister organisations compete and thrive independently, and if the old one loses its customer base, it would be allowed to retire gracefully. This sounds more reasonable and straightforward than a cumbersome and highly risky transformation.

Kotter’s model, like many others, is based on a few fundamental premises regarding the nature of reality and how organisations behave:

  • Reality is objective and can be faithfully constructed by meticulously describing the environment with factual data and rigorously defining its dynamics laws.
  • Organisations are machines (cybernetics, systems thinking), and their evolution can be influenced by locating their leverage points and applying pressure at the right lever and moment.
  • The primacy of the individual is above that of the group. The individual acts as an agent in the group, participating in forming the group and not vice versa. The social constructivist approach says that an individual’s identity is formed by belonging to one or more groups. Therefore the group is primary, and the individual is secondary.

This view was refuted earlier in favour of an approach based on Complex Adaptive Systems. In the next section, we will use this paradigm to present a radically different theory of change by management expert Dave Snowden.

7.2 An Alternative Theory of Change

Dave Snowden’s theory of change relies on the following premises based on our current state of knowledge in philosophy and natural sciences and his rich experience in the field.

  • Organisations are not machines but complex systems; in that sense, they are dispositioned to evolve in certain directions (although the ultimate future remains unpredictable). Snowden talks about the “adjacent possible” rather than a future target. Finding what’s possible (but which may not be ideal) is one way of managing in the present. This approach is more pragmatic, and its chances of success are significantly higher.
  • Snowden prefers to use a “vector theory of change“, which indicates a direction rather than a destination. He believes setting objectives induces attention blindness, and we miss serendipitous opportunities along the journey. Putting this into practice requires understanding the landscape and people’s stories around the water cooler. Once we have those narratives in a database, we can ask ourselves: “How do we get more good stories and fewer bad ones?”
  • Promoting change occurs via applying “governing and enabling constraints“, not through motivational speeches. Snowden maintains that complex systems can be managed by inhibiting undesirable effects through governing constraints (setting boundaries, implementing a suitable rewards and punishment system) and promoting desirable ones through enabling constraints (allocating resources and funds, leadership support).

The ideas we presented above are but a few of which Snowden champions in his business management theories. We highly recommend you watch his online videos and read his blog posts.

8. The Manufacturing vs Services Model

8.1 What Is Similar

Some concerns have recently been raised among experts that we are too reliant on the manufacturing model as our source of inspiration. This source, however, has its merits; there are too many similarities between producing cars and software to be ignored. Let’s list a few of those similarities:

  • Similar market dynamics. This includes fierce competition, scarce talent, and managing large projects and organisations.
  • The need for innovation and occasional transformation: This can be seen in the shift to electric cars or the creation of smartwatches, smart TVs, and digital cameras.
  • The complexity involved in dealing with end users and delivering quality products is almost identical: This can be readily observed in the interconnectedness of novel technology and changing user preferences.
  • The immense influence of human nature in terms of employee talent, organisation, motivation, commitment, and participation in the decision-making processes.

8.2 What Is Different

It’s also true that most of what goes on in Toyota’s vehicle centres is routine product development making incremental change from one model to the next. But the beauty of the Toyota Way is that it allows Toyota to periodically break from this “conservative” mold and innovatively develop a new vehicle with a new developmental approach.

— Jeffrey Liker, The Toyota Way – 14 Management Principles From the World’s Greatest Manufacturer

Remarkable differences also exist, meaning we must be critical of what applies and what doesn’t. Let’s look and some areas where the manufacturing model and the service model diverge:

  • The dependency on supply chains. This element is fundamental in manufacturing, especially in a global economy, and almost non-existent in software. Most big companies prefer writing their frameworks and satellite applications rather than relying on third-party software. When the latter is not an option (such as using Microsoft services), it would not be an issue impacted by global incidents as it all moves in cyberspace.
  • The assembly line. While Waterfall looks like an assembly line, the dissimilarities are immense and increase significantly with Agile. Software creation includes an element of creativity in at least the design and development stages, while in manufacturing, creativity occurs on the planning table (or CAD applications). Once the part, component, or model is proven to work efficiently and reliably, it is mass-produced, typically for a long time, with only minor incremental improvements with new iterations. Furthermore, employees on the assembly line are not involved in the design (in most cases) and typically don’t talk to end users. New models are designed, prototyped and tested extensively in laboratory conditions before being released to customers. There is no concept of fast iterations or continuous delivery.
  • Very low levels of uncertainty. While uncertainty will always reign when new technology is introduced or a new model is launched, very little uncertainty is involved in mass-producing a spare part. This is not true with software, as every new feature is unique. A module can be used and reused, and a bug fixed in one copy of the application can be included in all others after an upgrade.

In manufacturing, many problems are engineering, where the solution is already known to experts, and different solutions can be tested under laboratory conditions to determine the best. The problems are more complex in software, as we have endeavoured to show in the previous sections, fundamentally because the human element is strongly present and software applications tend to be unique. The potential of software applications (AI, ChatGPT, social media) radically modifying customer behaviour is non-trivial. At the same time, new digital camera models have not changed how we take pictures.

8.3 Cross-Industry Learning

We conclude this section with the following thought. Best practices can be effectively shared between industries, and corporations like Toyota have led the way in innovative techniques for dealing with organisational challenges.

While lessons learned from Toyota have been and continue to be applied across many industries, the same cannot be said for Six Sigma, so we must be selective in our choices, with decisions made based on theory combined with practice regarding what will work vs what won’t.

9. Evolution of the Software Development Industry

9.1 Reconstructing History

It seems natural to think about aspects of our modern lives, such as technology, as static and that “things were always that way”, but that’s very far from the truth. The smartphone and all the sophisticated apps that live in it have become such an integral part of our lives that we can barely remember what those lives looked like before them.

This stretching of the past in a manner that makes old tech look ancient and not worth understanding is OK when making minor decisions. However, when it becomes crucial for mid-term or strategic planning. More important is retrospection, or going back to first principles and examining the history and evolution of certain ideas (Agile is a great example of a concept where context is vital). This approach to certain problems involves critical thinking to determine how these ideas fared and whether they are still relevant today.

History, however, is a tricky business. For ordinary people, the past is a series of inevitable events, one leading the next. Historians would profoundly disagree and prefer to view historical events as a series of dots that could be connected in many different ways. Each path of connected dots is one narrative seen from a narrow perspective.

Moreover, these narratives can intermingle and dramatically influence each other’s evolution. These junctures (sometimes called bifurcation points or milestones) must be pinpointed to truly understand what happened.

9.2 A Regression of Many Intertwining Narratives

At the beginning of this article, we described two such narratives, technological advancement and changing customer preferences. They were and continue to be heavily intertwined, reinforcing each other (think of positive feedback loops) within the complex system comprising the tech organisations and their client bases.

Let us add to these two narratives a few more, which we list below. The order of the list is important. As you move down the list, the frequency of change in how things are done decreases, i.e., the top elements change faster than the ones below. Let’s see how this works.

9.2.1 Advancements in Software and IT Technology

This represents changes in programming languages, frameworks, deployment methods, business models, project management and delivery methodologies. The average lifetime of these changes is about 5-10 years, with a tiny minority becoming mainstream (C++, Java, Agile, Waterfall, client/server architecture, etc.).

9.2.2 Major technological advancements

This category includes groundbreaking leaps in technological know-how that can transform markets and client preferences and topple corporate giants. Examples are printers, personal computers, Compact Disks (CDs), DVDs, walkmans, the internet, the iPod, tablets, smartphones, social media, artificial intelligence, and quantum computing. The average lifetime of these technologies is 10-20 years, with naturally some surviving a lot longer (the lifetime probably follows a power law distribution rather than a Gaussian).

Technology NameDate it Became MainstreamDate it Died (if applicable)
Microwave Oven1950sN/A
Portable Radios1950s-1960s1990s-2000s
Personal Computer1980sN/A
Video Gaming Consoles1970s-1980sN/A
Social Media2000sN/A
Streaming Video2010sN/A
Virtual Reality2010sN/A
Artificial Intelligence2010sN/A
This table shows technological advancements that have become part of our lives at some point during the last 80 years. Some of it has survived, significantly altering our lifestyles, while others have died out.

9.2.3 Business Management and Organisational Theory

Now, the pace has slowed down considerably and rather than a few decades; it can be significantly more. The below table presents some of these ideas, their key thinkers, their inception dates, and how long they survived. Notice how many of these ideas survive today, some holding wildly contradicting views of others.

Theory/ConceptDate of EmergenceKey ThinkersDate it Stopped Becoming Mainstream
Scientific ManagementLate 19th/early 20th centuryFrederick Taylor1930s-1940s
Administrative TheoryEarly 20th centuryHenri FayolN/A
Bureaucratic TheoryEarly 20th centuryMax WeberN/A
Human Relations Theory1920s-1930sElton Mayo, Mary Parker Follett1950s-1960s
Systems Theory1940s-1950sLudwig von BertalanffyN/A
System Dynamics1950s-1960sJay ForresterN/A
Complexity Theory1980s-1990sStuart Kauffman, John Holland, Dave Snowden, Ralph StaceyN/A
Systems Thinking1990s-2000sPeter Senge, Donella MeadowsN/A
Contingency Theory1960s-1970sJoan Woodward, Paul Lawrence, Jay LorschN/A
Total Quality Management1980s-1990sW. Edwards Deming, Joseph Juran, Philip Crosby1990s-2000s
Lean Management1990s-2000sTaiichi OhnoN/A
Agile Management2000s-2010sJeff Sutherland, Ken Schwaber, Martin Fowler, Robert MartinN/A
Design Thinking2000s-2010sTim Brown, David KelleyN/A
A close look at Business Management and Organisational Theory main schools of Thought.

9.2.4 Organisational Culture

Our final narrative (as far as this discussion goes) is Organisational Culture. The main schools of thought have been listed below, with the majority still surviving.

ConceptDate of EmergenceKey ThinkersDate it Stopped Becoming Mainstream
Organizational Culture1950s-1960sEdgar ScheinN/A
Organizational Climate1960s-1970sRobert QuinnN/A
Organizational Identity1980s-1990sAlbert and WhettenN/A
Organizational Learning1980s-1990sChris Argyris, Peter SengeN/A
Organizational Citizenship Behavior1980s-1990sDennis OrganN/A
The table above shows the main theories of Organizational Culture. Seeing that all of them are currently mainstream and still competing, we can safely deduce that this will continue to be the case in the short to mid-term.

9.3. Key Takeaway

The key takeaway from this discussion is the following heuristic:

The amount of time invested in a specific aspect of the business (technology, processes, management philosophy, and organisational culture) must be inversely proportional to its change frequency.

This implies the following:

  • Organisations are better off investing in proven technologies, methodologies, processes, and management styles.
  • Non-tech-related topics are equally, if not more, important than tech itself, as the latter is constantly changing relative to the former.

On a slightly tangential, but not completely off-topic, subject, a heuristic for determining the lifetime of ideas might be the following: if a product, idea, or concept’s age is X years, it’s likely to live for another X years. (see Fooled by Randomness)

10. Summary

Let us now summarize the fundamental ideas of this article by order of importance from least to most important.

  • Technology is subordinate to the business. Most of the time, end users want something that works (solves their business needs) and, in case it’s a consumable product like a smartphone, to be aesthetically pleasing. Naturally, the technology used will ultimately impact the software’s performance, cost, and ability to follow consumer trends and, ultimately, its survival in the market.
  • First principles and context: When a new idea is put forward, it presents a radically different perspective on a specific issue at a particular time. In this sense, it is best characterised as a context-specific solution. If the idea becomes mainstream, it will likely evolve, specialize, and diversify enormously. A great example of this is Agile. After a few decades since its inception, it has become very challenging to untangle truth from myth in Agile-related matters. We can fully appreciate its relevance today by understanding the context in which it emerged and the first principles that made it an original and radical innovation.
  • Kadence of change in influential factors: Having a sense of the rate of change of schools of thought or technology is crucial when deciding which of these to adopt. Investing in novel technology or implementing new ideas can be very risky. Investing too heavily in technology at the expense of other areas (culture, people, organisation) is also risky.
  • Cross-learning from other industries: While manufacturing and services are not exactly similar, learned lessons in one can be transferred to the other. Concepts like Kanban, Kaizen, flow, and others originally conceived in car manufacturing companies like Toyota have been successfully merged into mainstream software development ideas.
  • While not straightforward, leadership is not a rare skill only a few can possess. Leadership skills can be acquired by becoming technically proficient, understanding human nature through philosophy and natural sciences, and having thorough exposure to organisational theory and business management concepts. True leadership, however, is all the previous skills combined with maturity (or what is commonly referred to as Emotional Intelligence).
  • Organisations and humans are not machines in the Newtonian sense and are best understood as Complex Adaptive Systems exhibiting complex behavioural patterns that cannot be controlled in the long term through management and planning. This has significant implications for change management and corporate strategy.

11. Conclusion

We hope the previous paragraphs have taken the reader on a journey of self-discovery, whose primary aim is to help them make sense of their past experiences by placing at their disposal new tools and frameworks distilled from natural sciences, organisational and human psychology, complexity theory, and business management.

While the infinite alternative future paths potentially available to us collapse and narrow with every decision we make or new person we meet, our past is also under perpetual reconstruction, redesigned and coloured by novel ideas and insights we acquire. Reinterpreting the past on a sound and solid bases prepares us for future challenges, of which we hope to have enough to make our lives interesting but not too stressful.

For software engineers whose education is primarily in mathematics, science, technology, and engineering, humans and organisations are cybernetic machines led by objective and rational leaders whose main job is to formulate long-term organisational strategies. Once those strategies are translated into action plans which they and their peers can dutifully apply, success should follow. We hope that the ideas presented in this text have convinced the reader of the mythical nature of this view and the complexity and messy nature of the real world.

If the world is wild, messy, unpredictable, and unwieldy, organisations must modify how they sense, interpret, and act in this world to survive. While the ideas we explained barely scratched the surface of their respective disciples, we hope they have intrigued the reader’s mind and spurred them into looking further. We also hope they have provided enough theoretical background to allow you to efficiently use (and potentially adapt and modify) the six Principles of Operational Excellence in Software Development articulated in Part 2 of this series.

12. References

Leave a Reply

Your email address will not be published. Required fields are marked *