Unit Testing: Is It Worth the Effort?
1. Overview
A few years back, I was involved in a major development project, and it was crucial to get it right.
Like all development projects, it was too short on resources. To compensate for the resource shortage, we had to be extremely efficient in what we did.
Until then, I had limited experience with topics like Test-Driven Development (TDD) or Continuous Integration (CI).
After a decent amount of research, I convinced myself that integrating some or all of these into our workflow would give us a gigantic boost in performance.
Test-Driven Development is an essential discipline that you can use to better manage resource allocation, test quality, and project deliveries.
1.1 TDD and Unit Testing
TDD relies heavily on the type of testing you use. It is generally inferred that A) Unit Testing should be the first choice, and B) the Unit of choice is the class or function.
We will challenge these two notions in this article and show that they are inefficient, costly, and counterproductive.
But before we continue, a note on Unit Testing.
1.2 Unit Testing as a Tool
Unit testing is another tool for developers to improve the outcome of their software development processes. Like all the rest, it can be very efficient when used wisely.
The idea is to recognize those situations where unit testing achieves the best results and those other situations where it’s utterly unsuitable.
The longer answer will be the focus of the next sessions.
3. What Is Unit Testing
Let’s start with the basics.
3.1 Definition
Let’s say you have a function that takes as input a variable number of characters like [‘a’, ‘b’, ‘c’] and returns all the possible combinations of these characters [‘ab’, ‘ac’, ‘bc’].
To ensure this function works as expected, you can test it by creating a set of dummy calls to that function with different inputs and validating the output for all the scenarios.
This routine is called Unit Testing.
To ensure the code works well under all circumstances (i.e. get decent test coverage), consider all the scenarios below.
| Input | Scenario |
|---|---|
| [‘a’, ‘b’, ‘c’] | Happy scenario, which is probably what you will get 99% of the time |
| [] | Calling the function with no inputs |
| [0, 1, +, “aa”] | Calling the function with an incorrect argument type |
| [”, ‘a’, ‘b’, ‘c’] | Incorrectly formatted input |
| [‘&’, ‘@’, ‘!’] | You might want to perform some sanity checks on the input, for example, accept only alpha-numeric characters. |
| [‘a’, ‘a’] | Multiple occurrences allowed? |
Many frameworks (such as JUnit for Java or Google Test for C++) have been created to provide developers boilerplate functionality, like test definition annotations, assert methods, and pass/fail to report.
This allowed users to focus more on testing the business side of things.
On top of those frameworks, as early as the 1980s, people felt the need for automation (check this article for a timeline of software testing), and test automation frameworks were developed.
3.2 Unit Tests: A Waste-Generating Practice
Writing a few unit tests now and then is never a problem, but that’s not usually how it works.
In a real-world scenario, especially with TDD, you must write massive unit tests for every new feature.
It’s up to the production process owners to specify the granularity of the units, but usually, it’s on the class or function level.
Under these settings, Unit Testing can be a notorious time-waster.
Drawbacks of Unit Testing
The two main drawbacks of massive Unit Tests on the class level are
A) High Cost-of-Ownership
B) Slower Time-to-Market
Even if you have the latest and greatest tools and infrastructure, unit testing will still require time and effort to create and maintain.
But that’s not the whole story.
The problem can be exacerbated even more if your tests are slow, inefficient, and easy to break.
Operational Excellence has made finding and eliminating waste its holy grail. If you want to achieve that, perhaps unit testing is the best place to start.
3.3 Unit Testing and TDD
Applying TDD in the context of Operational Excellence is an idea that’s well worth exploring. And as we have seen, this will lead us straight to Unit Testing.
Test-Driven Development was historically coupled with Unit Testing, and the latter was focused primarily on classes and functions as its definition of a unit.
What Went Wrong with TDD
That proved fatal as it coupled the test code with implementation details. That is a problem because the implementation can change as often as it needs to (such as for refactoring purposes), but as long as the behaviour does not change, there should be NO need to add or update your tests.
Keeping an implementation and its corresponding tests separate did not work well because Unit Tests on the class level are, by design, coupled to implementation.
This meant that heavy unit testing was slowly abandoned as unpractical. TDD was subsequently declared dead by part of the community.
That’s the opposite of what we want.
We want TDD to be used as a primary tool for achieving Operational Excellence in Software Development.
This required the “issue” of unit testing to be resolved.
4. The Challenges of Unit Tests
To fully appreciate the challenges of unit testing, we need to look at the root causes when implemented in the traditional sense.
There are many of these challenges, mainly: test coverage, test duration, testing implementation rather than functionality, database mocking, and testing legacy code.
4.1 Testing Implementation Instead of Behavior
Working on that project I mentioned earlier, I constantly refactored my code. Refactoring constituted something like 20-30% of my development time.
Refactoring keeps technical debt at bay and improves the quality of your code; it needs to be done smoothly and efficiently.
Test functionality, not implementation.
It was quite easy to notice that, being able to fully rely on existing test suites to immediately validate my code changes, I was moving at the momentum that I deemed sufficient.
I believe that what made the greatest difference was my ability to refactor the code without touching the test cases.
Functionality (or behaviour) is the system’s contract with the user. It’s the business functionality that it is supposed to deliver. Implementation, on the other hand, is transparent to the user.
To drive this point, though, I need to mention the black- vs. white-box testing issue.
If your testing requires knowing the system’s internal details, you are working with white-box testing.
On the other hand, if your testing does not require knowledge besides the functionality being tested, then you are using a black-box testing approach.
White-box testing is, by definition, implementation dependant, which is not ideal in the scheme of things we describe.
4.2 Test Coverage
From the example we used to demonstrate how Unit Testing works, it is very easy to see how one simple function may require many test cases to ensure decent coverage.
Ian Cooper, in this talk, mentions test code 2-3 times the size of the production code.
Imagine how much time was spent creating tens, if not hundreds, of thousands of test cases!
In a survey conducted by Ermira Daka and Gordon Fraser from the University of Sheffield, the following statistics were observed:
So, how do you generate decent coverage without spending an eternity writing unit tests?
Again, the answer is the definition of “unit” and the coupling of test code with implementation details.
4.3 Test Speed
Ideally, you want to run your test suites on every commit or at least once a day. That may be part of your Agile or DevOps requirements: faster iterations and quicker feedback.
If your tests cannot be completed in a few minutes, they are as good as useless.
If you run your suite once daily, during the night, imagine how many commits were done throughout the day and how difficult it would be to trace back an issue to a specific commit.
4.4 Mocking Dependencies
Mocking means replacing any dependencies that your class has on external systems.
An external system can be any of the following:
- Database
- Filesystem
- External API
- Third-party library
Why Is Mocking Bad?
Dependency mocking means you have to write test code that emulates your dependencies which are, by definition, part of the implementation.
As we have seen in the previous section, if the behaviour doesn’t change, your tests shouldn’t, too, even if the underlying implementation has changed.
In fact, in the same survey, researchers reported the following observation:
If you have not tried it before, you only need to spend very little time doing it to appreciate how difficult and time-consuming it can be.
If you change one column in a certain table, hundreds of test cases must be refactored.
Again, this directly implies the basic test “unit” choice.
4.5 Unit Testing Legacy Code
It is not unreasonable to think about moving an all-manual, massive testing exercise on legacy code to something more efficient and reliable.
Can it be automated Unit Testing?
Successfully completing a unit test project on legacy code is impossible (if not crazy). But you still need to test that software, so what do you do?
You can’t possibly write unit tests on legacy classes and functions because it would be a nightmare.
The solution, in our opinion, is to use System Integration Testing (or SIT) with as much automation as possible as part of your Testing Strategy.
5. When To Use Unit Testing
In the same survey by the University of Sheffield researchers, a very interesting fact was discovered.
Developers were writing unit test cases not because they were convinced of their utility but because it was either mandated by management or because the customer required it.
So when exactly is writing unit tests beneficial?
The table below represents, in our opinion, a reasonable list of criteria you could apply when trying to decide whether Unit Testing is the best approach to your testing problem.
| Criteria | Description |
|---|---|
| Is it self-contained | A self-contained function does not depend on other functions, which in turn might need to be tested. This does not apply to helper functions. |
| Is it self-sufficient | A self-sufficient function or class has no external dependencies like a database, filesystem, or APIs. |
| Is it part of the user contract | Does it provide any functionality to the user, or is it solely used internally within the code? |
| Is the input range big enough | In other words, can it be easily included with other tests? |
Given the above, the example of the function that generates different character combinations is a perfect candidate.
6. Unit Testing Recommendations and Best Practices
Is Unit Testing Worth the Effort
Absolutely.
Unit Testing is an integral part of your testing toolkit and must be applied before running the other, more expensive types of testing like SIT, UAT, or any performance-based tests.
Unit Tests are also an integral part of Test-Driven Development, a cornerstone practice when aiming for Operational Excellence.
7. Further Reading
Great talk on TDD by Ian Cooper.