Principles of Operational Excellence in Software Development

Georges Lteif

Georges Lteif

Software Engineer

Last Updated on May 23, 2022.
Subscribe now to stay posted!
About Us
20 min read

1. Overview

Operational Excellence had its roots originally in the automotive industry.

It started in the 1950s, where post-war Japanese car manufacturers were struggling to catch up with the potency of mass production – a method in which their giant American competitors had a significant edge.

The assembly line was deployed for the first time in 1913 by Henry Ford. It was capable of mass-producing an entire vehicle.

With massive resources and a vast market at Ford’s disposal, there was very little the Japanese automakers could do to compete.

The situation, however, changed drastically in the late 1980s, and the Japanese carmakers, especially Toyota, were doing significantly better than their American counterparts.

For the next few decades, the Japanese produced significantly better cars at highly-competitive costs. Their market share kept increasing year on year.

To make that happen, Toyota and other Japanese manufacturers had deployed a strategic weapon: something that will later be known as Operational Excellence.

Toyota then moved on to perfect its production methods and management model.

While the former was subsequently known as the Toyota Production System (or TPS), the latter became The Toyota Way. Both heavily inspired many industries.

The Toyota Way and the Toyota Production System were so efficient that professionals from different industries started looking at incorporating its concepts and methods in their businesses.

Think of KaizenKanban, and 5S. All three originated at Toyota and carried its trademark.

Although at the surface, car manufacturing bears minimal resemblance to the development of software products, there is a significant overlap in some other areas.

We believe that software development is well-posed to benefit from the concepts successfully applied in the car business.

In this article, we will look at how software development and delivery can profit from the philosophy and methods of Toyota.

2. Table Of Contents

3. Challenges of Delivering Software

Let’s look at why new solutions are required for issues in an industry that’s more than 70 years old.

3.1 Types of Challenges

The challenges organisations face can be broadly divided into shared and industry-specific.

Challenges of Software Delivery
Challenges of Software Delivery

Shared challenges are heavily centred around the human element and can include any of the below:

  1. Organisational culture
  2. Group behaviour
  3. Leadership
  4. Stakeholder management
  5. Collaboration
  6. Team motivation and morale
  7. Business, team, and individual performance
  8. Dealing with internal and external threats and opportunities
  9. Managing Risk
  10. Recognizing and dealing with complexity

Industry-specific challenges are mainly technical. An example can be:

  1. Tools and infrastructure
  2. Processes, frameworks, methodologies, and techniques
  3. Technology and innovation
  4. Competition and the fast-changing environment

Every industry in the technology business can have its own set of tools with which it carries about its business. Choosing a suitable technology stack, a set of tools, delivery methodologies, and production processes must follow specific rules to be efficient, effective, and enabling as well as increase your value proposition.

3.2 Why Software Projects Fail

To get a feel of how these challenges translate into real-world problems, let’s look at some examples.

A significant portion (around 80%) of IT projects either finish late, over budget, or with less business value than initially promised. Some projects never even make it into production.

The top reasons for IT project failures (sources: Forbes magazine, a paper published in Association for Information Systems, and this article that appeared in PMI) were given as follows:

#Reason Why Projects Fail
1Unclear business requirements
2The solution design and the final application did not match user requirements
3Substandard planning and execution
4Lack of standardised processes
5Working in silos
6Lack of agile practices
7Friction caused by undefined roles
8Lack of discipline
9Inadequate testing and quality assurance
10Focusing on technical detail instead of business value
Top 10 Reasons Why IT Projects Fail

Let’s have a look at a different but not entirely unrelated topic: the adoption of Agile as seen in 2021. Agile is a project delivery methodology that has been around for more than two decades.

Participants in the 2021 State of Agile Report survey provided the following answers regarding the slow adoption of Agile in their organisations:

#ReasonPercent of Participants
1Inconsistencies in processes and practices46%
2Cultural clashes43%
3General organisational resistance to change42%
4Lack of skills and experience42%
5Absence of leadership participation41%
6Inadequate management support and sponsorship40%
Reasons behind Slow Adoption of Agile

The following sections will provide guiding principles that will help tackle most of these challenges.

3.3 Addressing the Challenges of Software Delivery

To address the challenges of software delivery, we propose the following three-step plan:

  1. Find the root cause of those challenges
  2. Leverage Other People’s Experiences (OPE) to find mature and proven solutions
  3. Create mechanisms that can deal with novel or atypical future challenges
  • Step 1: Root Cause Analysis. Rooting out the cause of an issue requires fluency in the field and persistent perseverance to get to the bottom of the problem.

To get up to speed, we have done quite a lot of research and dedicated full articles to many of those challenges that are shared between organizations:

On each occasion, we went back to the source in an attempt to understand the circumstances that led to the creation of a specific idea or concept. We also looked at some retrospectives from influential opinion leaders who got their hands dirty trying out their ideas.

The research effort allows for making an informed, evidence-based opinion on what works and what does not. In most cases, firsthand experience was added to the mix.

  • Step 2: Leveraging Other People’s Experience. This is not an easy job; it requires a lot of poking around, an open mind, and no small amount of luck. The latter helps you stumble upon those precious sources of knowledge where valuable techniques can be transferred from a different field or industry to your own.

Earlier, I have divided the challenges of software delivery into two categories: shared problems mainly centred around human groups and industry-specific problems of a more technical nature.

We will lean on the Toyota management principles for inspiration on issues of a slightly technical nature such as production processes and quality management.

Moving solutions across borders is called creative problem-solving, and we have used this technique extensively when looking at Toyota and Six Sigma practices.

As we shall see later, Toyota has developed and deployed articulate and highly-effective strategies for addressing quality and performance issues with excellent results. Six Sigma also has taken proven techniques like DMAIC to a whole new level.

We believe those strategies are industry-agnostic and can be put to great use in software.

  • Step 3: The third and final step in the plan is process management. The latter can be defined as an overarching governance process that monitors production processes and intervenes when they degrade.

The technology world is vast and dynamic. Staying ahead of the curve is essential for the business’s survival.

Whether internal or external, every new challenge will require a modification of the way things are done today. 

Sometimes the changes can be small and localized. Other times, they are significant and disruptive. In the former case, they are called incremental changes, and in the latter, they are called redesign. Together, they fall under the umbrella term process management.

What to Expect in this Article?

In the following sections, we focus our attention on Steps 1: Finding the Root Cause(s) and 2: Leveraging Other People’s Experience.

We will look at some of the principles applied at Toyota as well as Six Sigma and remould them in such a way that they become useable in software.

We will then offer some thoughts on how best these principles can be implemented, mostly using mature methods that have existed for some time now.

Step 3 has been generously covered in two separate articles: Cultural Transformation and Process Management; we will not reproduce these here.

4. Operational Excellence in Software Development

4.1 Approach

The Toyota Way involves 14 management principles implemented in the Toyota Production System (TPS).

Some of these principles, like the Just-in-Time (JIT) system, are very specific to car manufacturing and, although they would make for a great read, their utility does not readily come across in the software space. Other principles like Genshi Genbutsu will be incredibly beneficial.

Six Sigma is also another place to look for solutions. Despite the limited utility of strategies that heavily depend on data and statistics in small to medium-sized businesses, the backbone of Six Sigma (the DMAIC problem-solving technique) is a valuable addition to our toolbox.

4.2 Objective

So why Operational Excellence? What exactly is it about, and what does it aim to achieve?

In summary, Operational Excellence describes the ability of an organisation to conduct its operations flawlessly. In the context of this website, the “operations” we are primarily interested in are the delivery of software solutions.

Origins of Operational Excellence

We have collected seven insightful quotes from leading figures in the business management and quality fields that we hope will give you a strong sense of what drives Operational Excellence.

4.3 Implementation Plan

Achieving Operational Excellence in Software Development requires us to follow the same three-step plan we outlined earlier (root cause analysis, finding creative solutions, and setting up governing processes).

Operational Excellence
Major Actors in a Software Business

As part of the root cause analysis, we need to figure out where the challenges originate, and not just the symptoms that we have listed earlier.

Is it the people, the processes, the technology, or some combination of those three elements that make up the origin of delivery challenges?

The answer is a combination of all three. This statement is relatively accurate in most cases. Let’s have a closer look at each of these categories in turn.

4.3.1 People and the Organization’s Culture

At the core of every organization is a team of professionals, and this team is typically divided into management, technical, and administrative.

The team can be newly formed or might have a long-standing history. In the former’s case, the first formative months are decisive as this is when the group’s culture is created. In the latter’s case, the culture is already established, and its transformation in response to new threats can be particularly interesting to study.

We accept challenges with a creative spirit, and the courage to realize our own dreams without losing drive or energy. We approach our work vigorously, with optimism and a sincere belief in the value of our contributions.

In both cases, Operational Excellence cannot be attained without a relentless drive for continuous improvement by everybody, and maintaining constant momentum requires keeping the employees engaged and motivated.

There needs to be a strong driving force behind it that transcends the monthly paycheque to keep the motivation flowing.

Employees are usually motivated when:

  1. Their ultimate goal as a group gives meaning to their efforts (such as benefiting the society and not just improving shareholder profits).
  2. They feel emotionally secure (as when their opinions are valued and their contributions appreciated).
  3. They have the opportunity to learn and grow.

Perhaps one of the most insightful studies published on satisfaction was by the US psychologist Mihaly Csikszentmihalyi in 1961, in which he coined the term flow when discussing motivation.

The flow is experienced during:

  1. Intense focus
  2. On an activity that we choose
  3. That is neither too far above nor below our skill levels
  4. With clear objectives
  5. And immediate rewards

When approaching the human element in organizations, it is always helpful to remember two things.

  • First, human hierarchical groups are complex systems and cannot be viewed as a simple mechanistic model. This characteristic of human groups means that making decisions, planning policies, and implementing strategies is not as easy as we would like it to be. Stong internal forces in complex systems render the predictions of the group’s behaviour, among other things, very challenging.

Frameworks like Six Sigma systematically ignore the human element when addressing organizational performance issues and focus more on the technical aspects.

The Toyota Production System is quite the opposite, with 14 principles that span the whole spectrum of issues to be tackled.

4.3.2 Process Management

A production process is a set of instructions applied to raw input to produce an output with increased business value over the inputs. These processes must be managed for efficiency (maximized value over cost) and effectiveness (maximal impact).

Production processes are usually set up in a specific context, and they, therefore, can become obsolete when the circumstances change. A process owner must monitor these processes and look for degeneration and lowered performance signs. She will then implement methods to improve or replace them.

Process Management Improvement, and Redesign
Process Management, Improvement, and Redesign

There are three stages to managing production processes:

  1. Design — During the process design phase, the process owner looks at the specific tools, infrastructure, and skills present in the team and designs processes fitted for this context.
  1. Incremental Improvement — This is an ongoing activity that provides remedies to issues arising from a process failure. Issues of such nature occur when the process is not correctly implemented, followed, or is inherently flawed. For example, a bug in a system interface that makes it into production may result from process failure. In this situation, improved measures for quality assurance can be implemented.
  1. Redesign — Process redesign is what you do when the processes become obsolete or require a significant modification to continue doing their job. For example, a systematic failure to deliver quality products signifies outdated frameworks or methodologies rather than simple processes. Moving to Agile or DevOps is one example.

The people at Toyota believe that following the right processes will always lead to the correct results. For them, it is preferable that you get incorrect results with the proper procedures over getting good results through flawed processes; you can only get lucky so many times.

4.3.3 Choosing the Right Technology

Going back to the Venn diagram at the top of this section, we see a tight connection between technology on the one hand and people and processes on the other.

Let’s explore these relationships through the following ideas:

  • Technology as a Strategic Choice — The selection of the technology stack, tools, and infrastructure is a strategic decision for two reasons. First, it is extremely hard to change once you acquire a few clients. Second, it directly influences many other strategic drivers, such as the ability to integrate your product with other technologies and keep up with innovation. Mainframe legacy applications are still running core banking applications at a considerable cost and limitation to the client.
  • Technology and People — The technology you use, its maturity, ubiquity, and popularity will dictate the size of the available pool of resources from which you can choose. Needless to say that a larger pool is always preferable as it drives down the costs of resources and allows for hiring skilled and experienced talent ready to start delivering. Using legacy technology, despite its ubiquity, may not attract talent as people look for new challenges to broaden their horizons.

5. Core Principles of Operational Excellence in Software Development

We chose six principles to represent the core ideas of Operational Excellence in Software Development.

Core Principles of Operational Excellence in Software Development
Core Principles of Operational Excellence in Software Development

5.1 Principle 1: Eliminating Cultural Blockers

The 15th State of Agile Report and the 2021 State of DevOps Report cite cultural blockers as primary challenges in adopting Agile or DevOps.

In this sense, perhaps it makes sense that the first principle of Operational Excellence would focus on managing cultural blockers that impede growth and success.

We say managing and not eliminating as the latter would be too difficult if not impossible, as we shall see in a moment.

So what are those cultural challenges? Let’s have a look at some examples.

5.1.1 Resistance to Change

Cultures are cognitive constructs deployed as defensive mechanisms that help groups cope with internal integration challenges as well as challenges posed by their environment.

Examples of internal challenges are the group’s cohesion, integration of new members or ideas, and replacing old ones.

External pressures can come from novel technologies, new customer requirements, or new competitors. Any change in the environment can be a threat (or opportunity) to the business. If the group were to survive, it would need to adapt and respond to those threats in new ways.

Its leaders and members will need to unlearn some old methods and learn new ones. This process is traumatic and does not necessarily succeed all the time.

A culture that recognises and acknowledges change as a constant of nature and has the psychological safety to carry out cultural transformations is much better adapted for survival.

5.1.2 Stakeholder Management

Stakeholders are influential people in the organisation who can influence its progress or regression. Any business strategy is bound to fail if it does not adequately analyse and manage its stakeholders.

It is also essential to recognise that organisational hierarchies are complex systems, making them difficult to control and predict.

Power distribution among the stakeholders and their desire and ability to support/undermine your plans will need to be determined before the project kicks off and closely monitored during execution.

Implementing long-term strategic improvement plans (such as Agile or DevOps) will require patience as the results will not be immediately evident. Therefore, continuous support from the leadership over the plan’s lifecycle is vital.

5.1.3 Organizational Culture

Organisational culture determines how groups perceive time and space, dictates the reward and punishment system, and imposes group inclusion or exclusion rules.

For example, a culture that encourages hierarchical boundaries, punishes failures, and constantly seeks stability and equilibrium does necessarily not facilitate innovation or the adoption of new ideas.

The perception of time is another example of how organisational culture governs the evolution of a group. A culture that focuses on the present may discourage the expenditure of time on experimentation and improvement if the results are not immediate.

Time management (beyond the to-do list) is a manifestation of the dominant culture.

5.1.4 Continuous Learning and Self-Improvement

Engineers, in general, have a particular way of looking at things, which comes from their formal training. The world, in their views, can be modelled as a machine with inputs, outputs, and rules that govern its dynamics.

The complexity of the natural world, especially in organisational hierarchies, fades away due to the reductionist methods applied in the mechanistic model.

One way of combatting such tendencies is through the study of philosophy, cultural anthropology, and organisational theory. This knowledge will inevitably lead to a radically different perception of the world, for example, using an informational paradigm instead of a mechanistic one.

A never-ending learning journey increases one’s awareness of their cognitive limitations, skewed perception of the world, and biased judgements.

This awareness is humbling on the one hand and empowering on the other, allowing them to make better decisions.

5.2 Principle 2: Process Management and Governance

Managing intelligent people can be difficult. They almost always have strong opinions on everything and can be challenging to convince to follow a process they can’t make sense of.

To make things even more complicated, we have various valid options to choose from in almost every aspect of software delivery, each with its cult following. Think of Agile/Waterfall or any of the many programming languages from which you can choose.

And if that is not enough, many organisations provide employees with dual or more roles and dotted and solid reporting lines which blur the lines between different roles and responsibilities.

Unless the governing processes are mature and the team is quite disciplined, this setting can only lead to chaos and suboptimal results at best.

What can the leadership do in similar situations? Here are some suggestions:

  1. Follow industry best practices for setting up your processes. Despite every project and organisation being unique, you can save yourself much trouble by leaning on other people’s experiences.
  1. Standardise processes across all your teams, customers, and project. When all your processes are similar, it is easier to gauge the impact of any improvements you make and validate the results. It’s also easier to switch your staff between teams and projects and to onboard new members.
  1. Appoint a process owner who has oversight on process performance and improvement. Their job would be to investigate any failures and decide whether it is systemic (because the process is inadequate or flawed) or a user error (because someone failed to follow the guidelines).
  1. Get up to speed with process design, improvement, redesign, and the tools and techniques available today (DMAIC, Six Sigma, Kaizen, Hansei). Attempt to solve issues with technical tools and evidence-based arguments instead of opinions and thoughts. This approach helps keep the discussion productive and non-personal.

5.3 Principle 3: Draw a Line Below Which Quality Does Not Go

When you have aggressive deadlines or your project is running out of time, there are only four things you can do:

  1. Reduce the scope
  2. Extend the deadline
  3. Commit additional resources
  4. Sacrifice quality
Software Project Constraints

While the first two options are your best bets, and the third option is not always feasible, the fourth option (sacrificing software and testing quality) is what most people would consider an easy way out.

There is a good explanation for why this can be appealing, and that’s because the results of sacrificing quality today will only be evident in the future. They will end up being someone else’s responsibility.

Sacrificing quality increases technical debt, which in turn creates a drag on your delivery capabilities. If left uncontrolled, it will transform your product from a lucrative asset to a liability.

Organizational Behavioural Pattern of Eroding Goals
Organizational Behavioral Pattern of Eroding Goals

Constantly sacrificing quality, in the long term, will have detrimental and irreversible effects on your business. This organizational behavioural pattern of Eroding Goals was documented in Peter M. Senge’s book (1990) The Fifth Discipline.

But what if postponing the delivery, reducing the scope, or adding resources is just not possible, and the only open option is to lower your quality standards?

The answer is: draw a line below which you don’t go. This threshold is where the risks will start to outweigh the benefits.

Here are some examples that help illustrate the point.

  1. Postpone or decline non-strategic and non-profitable projects. Focusing on lucrative or strategic customers is vital, especially during a fast-growing period.
  1. Reduce or eliminate non-essential activitiessuch as internal documentation, refactoring of old code, or daily meetings for smaller projects. Create different project delivery guidelines depending on project size.
  1. Carryover non-critical bugs into future releases. Manage your technical debt efficiently.
  1. Make software drops frequently so that the customer can start testing and uncovering issues as early as possible.

Ideally, it is best to avoid these situations altogether. You can do that with better planning, efficient processes and resource allocation, and being open and honest with your clients.

5.4 Principle 4: Implement Agile Processes Where Applicable

Once you separate the hype accumulated around Agile (and its close cousin DevOps), you will find that it offers immediate and substantial gains.

Granted, Agile is not for every project or product. Still, we can safely assume that of the 12 principles it advocates and the various practices that have flourished around it, there is a universal and generic set that can be applied everywhere.

Let’s have a look at the most powerful of these principles and practices.

  1. User stories: this is a topic that we will return to in its own section. For the moment being, suffice to say that user stories have brought back the customer to the forefront of the stage. Technical staff can get distracted quite easily from what is essential and valuable for the customer to what is purely a technical matter the customer is oblivious to.
  1. Face-to-face interactions over documentation: in this day and age, where teams are operating across different continents and timezones, it is easy to get sucked into excessive dependency on online collaboration tools such as emails and instant messaging. In these virtual worlds, misunderstandings proliferate, and progress is unnecessarily slow.
  1. Frequent iterations: In the pre-Agile world, customers would have to wait for months before getting their hands dirty with the new software. It was often the case that what they got was a long way from what they had in mind. Agile tries to resolve this problem with frequent software drops. These frequent drops allow valuable information to find its way back to the developers to adjust their requirements and design. The gains achieved are not small when we consider that unclear requirements are the number one reason for IT failures.
  1. Automation: although not strictly the child of Agile for historical reasons, automation is a necessary tool for Agile to succeed. It is impossible to release frequently and reliably without automating repetitive work, especially around building and testing.
  1. Agile Architecture: Architecture is by definition “what is important and hard to change” in your software design. It might be the tech stack you are using or how you organise your software’s modules and components. So how does that work with Agile where “change is always welcome”? Creating Agile architecture changes the question from “What is the best design I can come up with?” to “How can I design my architecture so that future changes are feasible and easy to introduce?”.

In most cases, you do not have to make a hard choice between Agile and Waterfall. You always have the option of creating a Hybrid model that is customized specifically for your delivery preferences.

5.5 Principle 5: Focus on Business Value

It can be deceptively easy to lose sight of what the customer genuinely needs and act upon what your product managers or technical staff think they need.

This diversion can happen when, for example, you measure your success based on how many new features you release instead of how many new customers you acquire.

It can also happen when you hand over the reins to your technical staff. They might, for example, decide to switch to new technology (at the immense cost of rewriting the product) without giving two hoots of whether the customer will notice any benefits.

Numerous other examples follow the same trend. Overengineering a feature in the hope that future customers would be easier to integrate is another common mistake. It’s just impossible to predict what prospective customers want, and you invest time extensively parametrizing a feature that will not work any differently than today.

Here are some things you can do to stay focused on what customers need:

  1. Create prototypes based on Minimum Viable Product (or MVPs) models when you want to test new ideas
  2. Take the customer along during the whole journey from project inception to end
  3. Create a minimalistic feature for your first customer. Parametrize it for the next one.
  4. Always subordinate the technology to the business
  5. Separate and limit the amount of Research and Development (R&D) that you would like to do.

5.6 Principle 6: Use Mature and Proven Technology to Serve Your Customers

Unless you work in high-tech mega-corporations like Google or Tesla, you are much safer using mature and proven technology. The benefits of such an approach are easy to spot.

First, you can easily find resources with knowledge and expertise when the skillset you want is not residing in a small niche.

Second, online help, support and documentation will be abundant for ubiquitous and mature technologies.

Third, you can integrate a lot easier with other systems and protocols.

Fourth, your products will be more stable and reliable; both properties are highly desirable, if not mandatory, in any product.

Finally, communication with customers and peers is smoother and less prone to misunderstandings when using standard methods and techniques.

While the above might seem boring and perhaps not so cool, it is safe, dependable, and profitable. These properties are fundamental when your products are running critical business operations.

Difficult and Strategic Technical Choices

IT professionals are required to make technical decisions every now and then, some of which will have paramount importance on the future of the product or the business. Looking for some guidelines on how to make better choices?

6. Practical Considerations for Operational Excellence

Now that we have looked in sufficient depth at the core principles of Operational Excellence for software development, we will discuss some practical methods to get there.

But before we proceed, a word of caution is due to any potential overdependence on tools alone as a means of achieving successful delivery.

Tools Alone Are Never Enough

While using the correct tools in the best possible way is essential to any activity; unfortunately, tools alone can never be enough. This statement is especially true in the technology field.

Because of the many different tools and techniques available, and more so because there is no single perfect way of developing, implementing, or delivering IT projects, it is necessary to get everybody engaged and onboard.

They need to understand why these specific tools have been deployed, and they need to know how to use them to achieve a common objective.

6.1 Leveraging Industry Best-Practices

The Software Development Lifecycle is the cornerstone of any software delivery methodology. In its most elaborate form, it consists of the seven following stages:

  1. Planning. This activity is, historically, not part of the standard representation of the SDLC. However, we have decided to include it because, in our view, activities such as resource planning, which naturally belong to this stage, directly impact the success or failure of your project. In addition to resource planning, work prioritization, management of multiple workstreams (such as ongoing projects and BAU), and resource allocation constitute the core activities of this stage.
  1. Analysis. In this stage, architects and project managers collaborate to produce a high-level design, high-level effort estimates, and a high-level project plan. A Statement of Work (SoW) is then created and signed-off before any technical work commences.
  1. Design. During the design stage, a detailed solution design is generated and, in some cases, signed off by the customer. We have explained in great detail the importance of having a solution design in a previous article.
  1. Development. Developers, testers, and DevOps engineers add business value to your product in the development stage.
  1. Testing. Different forms of software testing are applied to ensure the product stands up to the customer’s expectations. Functional requirements and non-functional design considerations have to be satisfied before any deployment to production can be made.
  1. Deployment. Deployment used to be a short and rare event, but with Continuous Delivery and DevOps practices, this has significantly changed. Megacorporations can deploy thousands of small changes to production every year, whereas the traditional way was just a few big ones.
  1. Support. The quality of customer service is an essential metric in vendor assessment exercises. Support is also a source of recurring revenue and needs to be looked after.
Software Development Lifecycle - SDLC
Software Development Lifecycle – SDLC

The table below summarises the techniques available today as part of the standard toolkit for businesses in the software development space.

IDTaskTechniques to Achieve the Best Results
1PlanningWhile planning and allocating resources, delivery managers should aim for two targets. The first target is to create self-sufficient teams. These teams can perform better as they do not need to go outside their group for approval or advice. The second target is to keep the group together during future projects (as much as possible) to leverage their shared experiences and mature relationships.

Also, balance the load on your resources so that projects can be delivered without having to sacrifice quality when trying to meet deadlines and avoid idling when work is scarce.
2AnalysisYou need to satisfy two criteria for this stage to be successful.

First, clarify and document all business requirements. Remember, unclear requirements is one of the top reasons for IT project failures.

Second, get the estimations right so that the project remains profitable and customer expectations remain well managed.
3DesignA low-level detailed solution design or LLD is a great place to document all the implementation details.
4DevelopmentSuccessful developers follow a unique coding style guide, look after the system’s architecture, and use Test-Driven Development (or TDD) as the backbone of their development processes.

Development should also investigate the way it uses Unit Tests to understand the benefits and pitfalls.
5TestingSoftware Testing is always a great place to start when looking for ways to optimize your delivery. Two significant factors immediately come to mind. The first is automation, and the second is Test-Driven Development.

Test automation significantly improves your delivery capabilities and boots the quality of your testing. Without automation, Agile and DevOps are very difficult, if not impossible, to implement. Automating repetitive testing activities also gives your QA staff more opportunities for growth.

Test-Driven Development or TDD is an excellent tool for streamlining your activities and allowing developers and testers to work in parallel when implemented with automation and a modified version of unit testing.
6DeploymentDeployment to production has moved to a whole new level with the concepts of continuous delivery, automated deployment, and frequent releases.

The infrastructure that supports automated and advanced deployment methods (both tools and processes) falls under DevOps. DevOps practices allow the fusion of development and operations in a single, streamlined process.
7SupportTriage, time management, SLA requirements, and customer satisfaction are the hallmark of customer support.
Leveraging Industry Best-Practices

6.2 Standardizing Processes

Employees sometimes view standardization as a means for curbing creativity, stifling innovation, and limiting personal initiatives.

While that statement may not be entirely untrue, it is still essential to standardize your production processes across the organization for various reasons.

There is a prevailing assumption that Agile is superior to Waterfall in project management and delivery should be the default approach for any team or project.

That may indeed be true, but it needs to be exercised with caution. Agile is just like any other tool; it may or may not be as suitable as you would like it to be.

However, what is essential in the context of standardization is that you pick one method and use it consistently. You can subject it later on to continuous improvement for optimal results.

If, however, you choose to create methods that are unique to your organization, there will probably be significant implications. While this may end up working better for you in the long run, its initial cost can be pretty high.

Generally speaking, and unless you have the size and influence of Google or Facebook, it is recommended that you communicate via a language that everybody understands, and start with industry best practices and customize from there.

6.3 Reducing Rework and Non-Value Adding Tasks

Taiichi Ohno implemented the following technique for eliminating waste on the shop floor of the Toyota production plants to an astounding degree of perfection.

Waste Identification and Elimination

The process is as follows:

  1. Observation
    • Spend as much time as possible observing the production process, preferably by experienced staff.
  2. Make a List
    • List all the repetitive steps required to complete a particular task.
  3. Mapping
    • Find out which ones produce no value.
  4. Identify Non-Value-Adding Activities
    • Work with the team members to identify non-value-adding steps such as overhead, rework, or enabling activities.
  5. Eliminate Non-Value-Adding Activities

This five-step process should not be confused with scientific management (a.k.a. Taylorism). Scientific management, proposed by Frederick Taylor in the 1880s, was archaic and counter-productive.

On the contrary, if you expect your technical staff to work like surgeons, their environment must be like an operation room in a modern hospital.

6.4 Structuring Your Team for Performance

Although it would not make much sense to have a prescription for team size and fixed profiles for its members, some rules-of-thumb might indeed work in most situations.

Martin Fowler, one of the 17 founders of Agile, describes the most performant team as follows:

Team Size

Keep the team size small for maximum communication and efficiency. Ideally, the golden number is less than 10 people.

Eric Schmidt and Jonathan Rosenberg published a book in 2014 called How Google Works. In this book, they state that the maximum number of people that a single person can efficiently manage is around 7.

Team Composition

Use the same team across multiple projects so that they could benefit from their shared knowledge of the application.

In the same book How Google Works, Google focuses on getting teams to work in smaller spaces, without physical barriers, and encourages relationship building. They also have a specific recruitment process that makes sure only the right people get on board.

New Members

At Google, they strive to hire interesting people. The criterium is as follows: if you get stuck in an airport with a colleague, would you be able to find interesting topics for discussion?

Let’s now turn to the Agile manifesto for more inspiration.

Team Structure

The best architectures, requirements, and designs emerge from self-organizing teams.

Team Collaboration

The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

Team Building Exercises

It is undeniable that kayaking with the team always has a positive impact on the mood. However, it is my conviction that team-building exercises alone are never sufficient (or even remotely relevant) when it comes to improving the team’s performance.

Emotionally charged, traumatic experiences dwarf the positive ones. It simply has to do with the way our brains function.

Vital as it may be, such team-building exercises should only be an addendum to the real ones:

  1. Valuing employees’ contributions
  2. Maintaining a safe work environment
  3. Keeping morale high
  4. Offering them the opportunity to grow

6.6 Tracking Progress and Performance

The concepts and ideas that will be the subject of this section are probably not the most popular as they go against some of the more modern management styles that have been trending for a while.

Tracking Progress and Performance
Tracking Progress and Performance

Have a look at the below table to compare and contrast.

Traditional MethodProposed Method
Heavily dependent on lengthy reports, and second-hand information to understand the situation on the ground.Go and see for yourself. Use your long experience and trained eyes to understand the processes and spot anomalies.
Uses complex statistics and KPIs such as Six Sigma, Taylor’s scientific management methods to monitor and improve performance.Robust, measurable, relevant, and simple KPIs that focus more on outcomes rather than results.
Unfounded optimism and positive attitude. Ignore negative indicators.Use visual aids to signal quality problems. Stop and fix the problem before it snowballs.
Treat the symptoms as opposed to the root cause. Has a “not my problem” attitude.Ask “Why” five times until you get to the root cause of the problem. Openly discuss any issue and fix it.
Traditional vs Proposed Progress and Performance Tracking Methods

6.7 Continuous Improvement

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.
Agile Manifesto

As the organization grows and pressure for faster and better deliveries increases, people will find less time for improving what they do. In his timeless classic The 7 Habits of Highly Effective People, Stephen Covey describes this phenomenon as “sharpening the saw”.

Even if individual initiatives were made here and there, these would continue to be local, spurious improvements and cease to suffice.

To maintain software delivery in top shape, continuous improvement methods need to be ingrained in the DNA of the system. In addition to that, they also need to be amalgamated into a single, system-wide activity.

Scrum uses retrospectives after each sprint. The double activity of Kaizen and Hansei is slightly different in its way, but the objective is the same.

I also find such exercises important for morale. They carry a positive message to the staff that the company is committed to continuous improvement of its people and processes, and it’s not just about the present but also the future.

7. Responding to Change

The pace at which software, and indeed any technology-based industry, is changing is breathtaking. Sometimes, that leaves us with a feeling of helplessness that’s difficult to ignore.

The question then becomes, how do we, as professionals in the software industry, keep producing premium-quality products despite the ever-changing nature of technology, tools, techniques, and methods that come with it?

The Lifetime of Technology Sub-fields

It might help to look at the software industry’s evolution, not as a homogenous, monolithic entity, but as four major sub-fields, each equipped with a specific lifetime. 

LevelTech SubfieldLifetime
1Programming languages, database engines, web frameworks< 10 years
2Project management techniques, project delivery methodologies (Agile, Waterfall, DevOps), business needs (eCommerce, payment methods, IoT, smartwatches, cloud technology…)10-20 years
3Computer science, global tech revolutions (quantum computing, digitization, internet, smartphones, AI, …)20-30 years
4Business Management (Taylorism, The Toyota Way, Six Sigma, Lean Manufacturing, Operational Excellence)30+ years
The Lifetime of Technology Sub-fields

Like any other field in science or technology, progress seems to happen in jumps, followed by plateaus, rather than monotonic, incremental improvement.

This segregated view of the world will help us prepare by properly managing our expectations. Let’s have a look at those four levels in some detail.

Level 1: Fluctuations

Level 1 entries appear and disappear every day. Use reliable technology, always! The gravest mistake that you can commit would be to try and catch up with their destructive pace.

Level 2: Big Changes

Level 2 entries are technological trends that change dramatically every decade or so. People still use cash nowadays but they would also like to take advantage of the convenience of mobile phones or watches.

While embracing those new methods is not strictly vital, not doing so places your business in a less than ideal situation vis-a-vis competition.

Level 3: Major Disruptions

Looking at entries from Level 3, we immediately recognize those game-changing technological revolutions that put so many giants out of business overnight.

Those past giants have failed to reinvent their value propositions and were replaced swiftly with newcomers who leveraged new technologies to offer innovative solutions.

Level 4: Paradigm Shifts

It is somehow fortunate for us that the more potent concepts of Level 4, those that make a business successful or otherwise, change extremely slowly. In fact, there is more than enough time for everyone to master them.

These occur as part of larger sociological, economic, or technological revolutions.

Because Level 4 fields, such as Operational Excellence, sit at the top of the hierarchy as one of the most stable concepts, ideas, and technological constructs of the lot, it makes sense to thoroughly consider it and invest time in applying and mastering it.

8. Final Words

Software development is one of those fields where technologies, disciplines, practices, and methodologies abound, and discussions on which one to adopt can quickly turn into a war of cults.

This article proposes a few principles that could help you make the right choices while avoiding any such conflicts. We hope that, in that sense, it has achieved its objective.

I believe that doing something great is always gratifying and provides a deep sense of self-satisfaction. That’s why sculptures, composers, artisans, and maybe software developers do what they do.

However, success in software development also demands technical expertise and extensive know-how. Both of these come within reach if the principles around which this article is centred are closely followed.

Passion, mastery of skills, and in-depth knowledge of human nature are all you need to succeed.

9. References

Technical Risk Management and Decision Analysis — Introduction and Fundamental Principles

1. Overview I could not find a better way to start an article on Risk and Risk Management than by quoting the opening lines of Donald Lessard and Roger Miller’s 2001 paper that, briefly but lucidly, summarizes the nature of large engineering endeavours. It goes like this: This article leans heavily on three handbooks thatContinue reading “Technical Risk Management and Decision Analysis — Introduction and Fundamental Principles”

Complexity and Complex Systems From Life on Earth to the Universe: A Brief Introduction

1. Overview Dealing with complexity is an integral part of our lives, even if we do not realise it.  An organisation can be modelled as a complex system from the scale of megacorporations right down to the smallest teams. The architecture of software solutions can be equally complicated, and megaprojects and implementations are certainly involved.Continue reading “Complexity and Complex Systems From Life on Earth to the Universe: A Brief Introduction”

Book Review: Programming the Universe — A Quantum Computer Scientist Takes on the Cosmos

Synopsis Most physical theories adopt a mechanistic view when examining natural phenomena where any system can be modelled as a machine whose initial conditions and dynamics govern its future behaviour. In this book, Programming the Universe — A Computer Scientist Takes on the Cosmos, Professor Seth Lloyd proposes a radically different approach centred around aContinue reading “Book Review: Programming the Universe — A Quantum Computer Scientist Takes on the Cosmos”

From Abstract Concepts to Tangible Value: Software Architecture in Modern IT Systems

1. Overview Software design and architecture are two very elusive concepts; even Wikipedia’s entries (ref. architecture, design) are somewhat fuzzy and do not clearly distinguish between the two. The Agile manifesto’s statement on architecture and design is especially brief and raises more questions than answers. The most common definition of software architecture is as follows:Continue reading “From Abstract Concepts to Tangible Value: Software Architecture in Modern IT Systems”

Business Requirements and Stakeholder Management: An Essential Guide to Definition and Application in IT Projects

1. Overview The complexity of business requirements in IT projects has experienced exponential growth due to pressures by increasingly sophisticated client preferences, novel technologies, and fierce competition. Consider, for example, the case of financial payments. In the mid-80s, most payment transactions occurred inside bank branches, and only the biggest banks offered services on ATM orContinue reading “Business Requirements and Stakeholder Management: An Essential Guide to Definition and Application in IT Projects”


Something went wrong. Please refresh the page and/or try again.

Leave a Reply