A few years back, I was involved in a major development project, and it was crucial to get it right.
Like all development projects, it was too short on resources. To compensate for the resource shortage, we had to be extremely efficient in what we did.
After a decent amount of research, I convinced myself that integrating some or all of these into our workflow would give us a gigantic boost in performance.
TDD and Unit Testing
TDD relies heavily on the type of testing you use. It is generally inferred that A) Unit Testing should be the first choice, and B) the Unit of choice is the class or function.
We will challenge these two notions in this article and show that they are not just inefficient but also costly and very counterproductive.
But before we continue, a note on Unit Testing.
Unit Testing as a Tool
Unit testing is another tool available for developers to improve the outcome of their software development processes. Like all the rest, it can be very efficient when used wisely.
The idea is to recognize those situations where unit testing achieves the best results and those other situations where it’s utterly unsuitable.
The longer answer will be the focus of the next sessions.
2. Table of Contents
- 1. Overview
- 2. Table of Contents
- 3. What Is Unit Testing
- 4. The Challenges of Unit Tests
- 5. When To Use Unit Testing
- 6. Unit Testing Recommendations and Best Practices
- 7. Further Reading
- 8. Featured Articles
3. What Is Unit Testing
Let’s start with the basics.
Lets say you have a function that takes as input a variable number of characters like [‘a’, ‘b’, ‘c’] and returns all the possible combinations of these characters [‘ab’, ‘ac’, ‘bc’].
To ensure this function works as expected, you can test it by creating a set of dummy calls to that function with different inputs and validating the output for all the scenarios.
This routine is called Unit Testing.
To ensure the code works well under all circumstances (i.e. get decent test coverage), you must consider all the scenarios below.
|[‘a’, ‘b’, ‘c’]||Happy scenario, which is probably what you will |
get 99% of the time
|||Calling the function with no inputs|
|[0, 1, +, “aa”]||Calling the function with an incorrect argument type|
|[”, ‘a’, ‘b’, ‘c’]||Incorrectly formatted input|
|[‘&’, ‘@’, ‘!’]||You might want to perform some sanity checks |
on the input, for example, accept only alpha-numeric
|[‘a’, ‘a’]||Multiple occurrences allowed?|
Many frameworks (such as JUnit for Java or Google Test for C++) have been created to provide developers with boiler-plate functionality, like test definition annotations, assert methods, and pass/fail to report.
This allowed users to focus more on testing the business side of things.
3.2 Unit Tests: A Waste-Generating Practice
Writing a few unit tests every now and then is never a problem, but that’s not usually how it works.
In a real-world scenario, especially with TDD, you must write massive unit tests for every new feature.
It’s up to the production process owners to specify the granularity of the units, but usually, it’s on the class or function level.
Under these settings, Unit Testing can be a notorious time-waster.
Drawbacks of Unit Testing
The two main drawbacks of massive Unit Tests on the class level are
A) High Cost-of-Ownership
B) Slower Time-to-Market
Even if you have the latest and greatest tools and infrastructure, unit testing will still require time and effort to create and maintain.
But that’s not the whole story.
The problem can be exacerbated even more if your tests are slow, inefficient, and easy to break.
3.3 Unit Testing and TDD
Test-Driven Development was historically coupled with Unit Testing, and the latter was focused primarily on classes and functions as its definition of a unit.
What Went Wrong with TDD
That proved fatal as it coupled the test code with implementation details.
That is a problem because the implementation can change as often as it needs to (such as for refactoring purposes), but as long as the behaviour does not change, there should be NO need to add or update your tests.
Keeping an implementation and its corresponding tests separate did not work well for the following reason: Unit Tests on the class level are, by design, coupled to implementation.
This meant that heavy unit testing was slowly abandoned as unpractical. TDD was subsequently declared dead by part of the community.
That’s the opposite of what we want.
This required the “issue” of unit testing to be resolved.
4. The Challenges of Unit Tests
To fully appreciate the challenges of unit testing, when implemented in the traditional sense, we need to look at the root causes.
4.1 Testing Implementation Instead of Behavior
Working on that project I mentioned earlier, I constantly refactored my code. Refactoring constituted something like 20-30% of my development time.
Refactoring keeps technical debt at bay and improves the quality of your code; it needs to be done smoothly and efficiently.
It was quite easy to notice that being able to fully rely on existing test suites to immediately validate my code changes, I was moving at the momentum that I deemed sufficient.
I believe that what made the greatest difference was my ability to refactor the code without having to touch the test cases.
Functionality (or behaviour) is the system’s contract with the user. It’s the business functionality that it is supposed to deliver. Implementation, on the other hand, is transparent to the user.
To drive this point, though, I need to mention the issue of black- vs white-box testing.
If your testing requires knowing the system’s internal details, then you are working with white box testing.
On the other hand, if your testing does not require knowledge aside from the functionality being tested, then you are using a black box testing approach.
White-box testing is, by definition, implementation dependant, which is not ideal in the scheme of things we describe.
4.2 Test Coverage
From the example we used to demonstrate how Unit Testing works, it is very easy to see how one simple function may require many test cases to ensure decent coverage.
Ian Cooper, in this talk, mentions test code 2-3 times the size of the production code.
Imagine how much time was spent creating tens, if not hundreds, of thousands of test cases!
In a survey conducted by Ermira Daka and Gordon Fraser from the University of Sheffield, the following statistics were observed:
So, how do you generate decent coverage without spending an eternity writing unit tests?
Again, the answer is the definition of “unit” and the coupling of test code with implementation details.
4.3 Test Speed
If your tests cannot be completed in a few minutes, they are as good as useless.
If you run your suite once per day, during the night, imagine how many commits were done throughout the day and how difficult it would be to trace back an issue to a specific commit.
4.4 Mocking Dependencies
Mocking means replacing any dependencies that your class has on external systems.
An external system can be any of the below:
- External API
- Third-party library
Why Is Mocking Bad?
Dependency mocking means you have to write test code that emulates your dependencies which are, by definition, part of the implementation.
As we have seen in the previous section, if the behaviour doesn’t change, your tests shouldn’t too, even if the underlying implementation has changed.
In fact, in the same survey, researchers reported the following observation:
If you have not tried it before, you only need to spend very little time doing it to appreciate how difficult and time-consuming it can be.
You change one column in a certain table, and then hundreds of test cases need to be refactored.
Again, this is a direct implication of the basic test “unit” choice.
4.5 Unit Testing Legacy Code
It is not unreasonable to think about moving an all-manual, massive testing exercise on legacy code to something more efficient and reliable.
Can it be automated Unit Testing?
Successfully completing a unit test project on legacy code is impossible (if not crazy). But you still need to test that software, so what do you do?
You can’t possibly write unit tests on legacy classes and functions because it would be a nightmare.
The solution, in our opinion, is to use System Integration Testing (or SIT) with as much automation as possible as part of your Testing Strategy.
5. When To Use Unit Testing
In the same survey by the University of Sheffield researchers, a very interesting fact was discovered.
Developers were writing unit test cases not because they were convinced of their utility but because it was either mandated by management or because the customer required it.
So when exactly is writing unit tests beneficial?
The table below represents, in our opinion, a reasonable list of criteria you could apply when trying to decide whether Unit Testing is the best approach to your testing problem.
|Is it self-contained||A self-contained function does not depend on |
other functions, which in turn might need
to be tested. This does not apply to helper
|Is it self-sufficient||A self-sufficient function or class has no external dependencies like a database, filesystem, or APIs.|
|Is it part of the user contract||Does it provide any functionality to the user, or is it solely used internally within the code?|
|Is the input range big enough||In other words, can it be easily included with other tests?|
Given the above, the example of the function that generates different character combinations is a perfect candidate.
6. Unit Testing Recommendations and Best Practices
7. Further Reading
Great talk on TDD by Ian Cooper.