Understanding and Managing Technical Debt
1. Overview
Technical debt is a concept first coined by Ward Cunningham, one of the co-authors of the Agile manifesto.
When recounting the story, he explains how he used the metaphor of financial debt to explain why they needed to refactor part of an existing codebase for a product they were developing.
The business stakeholders needed to justify spending money on something already working. From there on, the term technical debt took off.
The story goes like this:
With borrowed money, you can do something sooner than you might otherwise, but then until you pay back that money you’ll be paying interest. I thought borrowing money was a good idea, I thought that rushing software out the door to get some experience with it was a good idea, but that of course, you would eventually go back and as you learned things about that software you would repay that loan by refactoring the program to reflect your experience as you acquired it.
— Ward Cunningham
3. What Is Technical Debt
3.1 Definition
You accumulate technical debt when you sacrifice a portion of software design or software quality to make an early release.
These sacrifices include relaxing test coverage or quality or design constraints.
Accepting technical debt today is made assuming it will be paid later through refactoring when better knowledge or more time is available.
Complex functionality requires complex code. However, technical debt will considerably slow your software delivery pace in the long run.
3.2 Technical Debt: Intentional or Unintentional?
Bad code and technical debt are not the same. The former results from poor coding practices or lack of skills, while the latter is the price we pay to release code before it’s mature.
In the former case, we are unaware that the code is bad, while in the latter case, a risk-informed decision is made to release software prematurely.
Two divergent views exist on the source of technical debt:
- The first is by Ward Cunnigham and by Bob Martin, who emphasized that “A Mess is not a Technical Debt“.
- The second is by Steve McConnell (cited in this research paper), who stated that technical debt could be unintentional, for example, due to poor programming skills or the usage of an inconsistent coding style.
We believe the first definition (technical debt is different from bad code) is closest to what the original author intended.
3.3 Justifying Technical Debt
There are several scenarios where the risk of taking on technical debt is deemed acceptable:
- Your current understanding of the business requirements is incomplete, and you have decided to create and release an MVP to gather early information from the users.
- You are pressured by project schedule or budget constraints to use software architecture or implementations with known limitations.
- Old solutions may not age well and become obsolete. You can decide to fix them now at the risk of derailing the next release or accept them as technical debt and postpone their refactoring to a more convenient time.
- Using code hacks or duct tape fixes to solve bugs. This is OK in times of crisis and when test suites with decent coverage do not exist to cover changes from a better solution. Such short-term remedies increase technical debt.
- Developers simultaneously working on many feature branches create the additional risk of accrued technical debt. Collaboration is required to synchronize views, share knowledge, and align plans. The more features are developed in isolation and teams working in silos, the higher the technical debt.
3.4 Technical Debt Is Hidden
Technical debt is primarily visible only to the developer. When a software product’s shortcomings are obvious to the business stakeholder or end-user, they are not considered technical debt; the value of addressing them can be directly observed and measured.
On the other hand, missing documentation, poor software architecture or design, or low test coverage are not evident to the non-technical stakeholder. They present a risk because their future impact cannot be precisely determined and therefore is difficult to manage.
4. Types of Technical Debt
A research team investigated the literature on technical debt and published a paper listing its different types. Below are a few examples Of the thirteen classifications identified:
- Architecture Debt — Architecture debt consists of non-modular designs, ad-hoc interfaces, structural coupling, and dependencies.
- Design Debt — This category includes issues like strong coupling, large classes, duplicated code, or any violation of the software design principles like S.O.L.I.D‘s Single Responsibility Principle or Interface Segregation.
- Coding Debt — Coding debt can be slow algorithms, obsolete solutions, lack of design pattern usage, and duplicated code.
- Documentation Debt — Poor, outdated, or missing documentation is the central theme of this class.
- Test Debt — Poor coverage, manual testing, and unit tests with a high cost of ownership.
5. Managing Technical Debt
5.1 A Business Incentive
The fundamental challenge in managing technical debt lies not in quantifying it but in identifying the top areas in your codebase where deploying your (limited) refactoring efforts will maximize the business returns of this exercise.
Adam Tornhill revolutionized technical debt management by repurposing data obtained from code versioning tools.
The ideas driving his thought processes were as follows:
- Static code analysis tools can locate and quantify technical debt in a codebase, but that will not be sufficient, especially if the amount of debt is enormous.
- For massive technical debts in an (old) codebase, the resources, budget, and potential BAU disruptions to get rid of it are too high. We need a smart way of managing debt.
- The data gleaned from source code versioning tools can offer tremendous insights on where to focus our refactoring by highlighting hotspots of significant development activities, linking modules to teams or individual developers, and identifying coupled components that tend to change together.
5.2 Guidelines for Technical Debt Reduction
Reducing technical debt can be done proactively or reactively.
- Proactive mode — In this mode, developers working on a new feature identify the need to refactor an existing code and incorporate it within their current change.
- Educating developers on code complexity may avoid accumulating technical debt for reasons other than project necessity. G. Ann Campbell from SonarSource, published an excellent paper on cognitive complexity.
- It’s essentially a heuristic for calculating a complexity score which tells us how hard it is to make sense of a code.
- This algorithm is used in SonarQube to locate sources of technical debt in a code base. For example, if developers can avoid nested loops, recursion, and multiple breaks in a method’s flow, technical debt would not increase in the first place.
- Reactive mode — Here, developers agree to add a stage to their production processes to reduce technical debt periodically. This extra process is facilitated by static analysis tools like SonarQube and integrated within their CI/CD pipelines.
- Much like any maintenance task, managing technical debt must be a legitimate process within the project delivery framework with appropriate resource allocation, valuable tools, and expertise.
- The methods pioneered by Adam Tornhill for reducing technical debt based on data from source code versioning tools are especially interesting.
- These not only recommend pragmatic ways of refactoring code smells but also address feature and module coupling, architectural debt, and single points of failure.
- The main idea is to use histograms to identify files, classes, or methods, frequently changing and becoming bottlenecks in the delivery. We expect to find a tiny percentage of our files (a Pareto distribution) constantly evolving and needing continuous maintenance.