Unit Testing: Is It Worth the Effort?

Georges Lteif

Georges Lteif

Software Engineer

Last Updated on September 7, 2022.
Subscribe now to stay posted!
About Us
12 min read

1. Overview

A few years back, I was involved in a major development project, and it was crucial to get it right.

Like all development projects, it was too short on resources. To compensate for the resource shortage, we had to be extremely efficient in what we did.

Until then, I had very limited experience with topics like Test-Driven Development (TDD) or Continuous Integration (CI).

After a decent amount of research, I convinced myself that integrating some or all of these into our workflow would give us a gigantic boost in performance.

software testing
Different layers of software testing

Test-Driven Development is an essential discipline that you can use to better manage resource allocation, test quality, and project deliveries.

TDD and Unit Testing

TDD relies heavily on the type of testing you use. It is generally inferred that A) Unit Testing should be the first choice, and B) the Unit of choice is the class or function.

We will challenge these two notions in this article and show that they are not just inefficient but also costly and very counterproductive.

But before we continue, a note on Unit Testing.

Unit Testing as a Tool

Unit testing is another tool available for developers to improve the outcome of their software development processes. Like all the rest, it can be very efficient when used wisely.

The idea is to recognize those situations where unit testing achieves the best results and those other situations where it’s utterly unsuitable.

The longer answer will be the focus of the next sessions.


2. Table of Contents


3. What Is Unit Testing

Let’s start with the basics.

3.1 Definition

Lets say you have a function that takes as input a variable number of characters like [‘a’, ‘b’, ‘c’] and returns all the possible combinations of these characters [‘ab’, ‘ac’, ‘bc’].

To ensure this function works as expected, you can test it by creating a set of dummy calls to that function with different inputs and validating the output for all the scenarios.

This routine is called Unit Testing.

To ensure the code works well under all circumstances (i.e. get decent test coverage), you must consider all the scenarios below.

InputScenario
[‘a’, ‘b’, ‘c’]Happy scenario, which is probably what you will
get 99% of the time
[]Calling the function with no inputs
[0, 1, +, “aa”]Calling the function with an incorrect argument type
[”, ‘a’, ‘b’, ‘c’]Incorrectly formatted input
[‘&’, ‘@’, ‘!’]You might want to perform some sanity checks
on the input, for example, accept only alpha-numeric
characters.
[‘a’, ‘a’]Multiple occurrences allowed?
Unit Testing Example

Many frameworks (such as JUnit for Java or Google Test for C++) have been created to provide developers with boiler-plate functionality, like test definition annotations, assert methods, and pass/fail to report.

This allowed users to focus more on testing the business side of things.

On top of those frameworks, and as early as the 1980s, people felt the need for automation (check this article for timeline of software testing) and test automation frameworks were developed.

3.2 Unit Tests: A Waste-Generating Practice

Writing a few unit tests every now and then is never a problem, but that’s not usually how it works.

In a real-world scenario, especially with TDD, you must write massive unit tests for every new feature.

It’s up to the production process owners to specify the granularity of the units, but usually, it’s on the class or function level.

Under these settings, Unit Testing can be a notorious time-waster.

Drawbacks of Unit Testing

The two main drawbacks of massive Unit Tests on the class level are

A) High Cost-of-Ownership

B) Slower Time-to-Market

Even if you have the latest and greatest tools and infrastructure, unit testing will still require time and effort to create and maintain.

But that’s not the whole story.

The problem can be exacerbated even more if your tests are slow, inefficient, and easy to break.

Operational Excellence has made finding and eliminating waste its holy grail. If you are keen on achieving that, perhaps unit testing is the best place to start.

3.3 Unit Testing and TDD

Applying TDD in the context of Operational Excellence is an idea that’s well worth exploring. And as we have seen, this will lead us straight to Unit Testing.

Test-Driven Development was historically coupled with Unit Testing, and the latter was focused primarily on classes and functions as its definition of a unit.

What Went Wrong with TDD

That proved fatal as it coupled the test code with implementation details.


That is a problem because the implementation can change as often as it needs to (such as for refactoring purposes), but as long as the behaviour does not change, there should be NO need to add or update your tests.

Keeping an implementation and its corresponding tests separate did not work well for the following reason: Unit Tests on the class level are, by design, coupled to implementation.

This meant that heavy unit testing was slowly abandoned as unpractical. TDD was subsequently declared dead by part of the community.

That’s the opposite of what we want.

We want TDD to be used as a primary tool for achieving Operational Excellence in software development.

This required the “issue” of unit testing to be resolved.


4. The Challenges of Unit Tests

To fully appreciate the challenges of unit testing, when implemented in the traditional sense, we need to look at the root causes.

There are quite a few of these challenges, mainly: test coverage, test duration, testing implementation rather than functionality, database mocking, and testing legacy code.

Challenges unit testing worth effort
Challenges of unit testing

4.1 Testing Implementation Instead of Behavior

Working on that project I mentioned earlier, I constantly refactored my code. Refactoring constituted something like 20-30% of my development time.

Refactoring keeps technical debt at bay and improves the quality of your code; it needs to be done smoothly and efficiently.

Test functionality, not implementation.

It was quite easy to notice that being able to fully rely on existing test suites to immediately validate my code changes, I was moving at the momentum that I deemed sufficient.

I believe that what made the greatest difference was my ability to refactor the code without having to touch the test cases.

Functionality (or behaviour) is the system’s contract with the user. It’s the business functionality that it is supposed to deliver. Implementation, on the other hand, is transparent to the user.

To drive this point, though, I need to mention the issue of black- vs white-box testing.

If your testing requires knowing the system’s internal details, then you are working with white box testing.

On the other hand, if your testing does not require knowledge aside from the functionality being tested, then you are using a black box testing approach.

White-box testing is, by definition, implementation dependant, which is not ideal in the scheme of things we describe.

4.2 Test Coverage

From the example we used to demonstrate how Unit Testing works, it is very easy to see how one simple function may require many test cases to ensure decent coverage.

Ian Cooper, in this talk, mentions test code 2-3 times the size of the production code.

Imagine how much time was spent creating tens, if not hundreds, of thousands of test cases!

In a survey conducted by Ermira Daka and Gordon Fraser from the University of Sheffield, the following statistics were observed:

So, how do you generate decent coverage without spending an eternity writing unit tests?

Again, the answer is the definition of “unit” and the coupling of test code with implementation details.

4.3 Test Speed

Ideally, you want to run your test suites on every commit or at least once a day. That may be part of your Agile or DevOps requirements: faster iterations and quicker feedback.

If your tests cannot be completed in a few minutes, they are as good as useless.

If you run your suite once per day, during the night, imagine how many commits were done throughout the day and how difficult it would be to trace back an issue to a specific commit.

4.4 Mocking Dependencies

Mocking means replacing any dependencies that your class has on external systems.

An external system can be any of the below:

  1. Database
  2. Filesystem
  3. External API
  4. Third-party library

Why Is Mocking Bad?

Dependency mocking means you have to write test code that emulates your dependencies which are, by definition, part of the implementation.

As we have seen in the previous section, if the behaviour doesn’t change, your tests shouldn’t too, even if the underlying implementation has changed.

In fact, in the same survey, researchers reported the following observation:

If you have not tried it before, you only need to spend very little time doing it to appreciate how difficult and time-consuming it can be.

You change one column in a certain table, and then hundreds of test cases need to be refactored.

Again, this is a direct implication of the basic test “unit” choice.

4.5 Unit Testing Legacy Code

It is not unreasonable to think about moving an all-manual, massive testing exercise on legacy code to something more efficient and reliable.

Can it be automated Unit Testing?

Successfully completing a unit test project on legacy code is impossible (if not crazy). But you still need to test that software, so what do you do?

You can’t possibly write unit tests on legacy classes and functions because it would be a nightmare.

The solution, in our opinion, is to use System Integration Testing (or SIT) with as much automation as possible as part of your Testing Strategy.


5. When To Use Unit Testing

In the same survey by the University of Sheffield researchers, a very interesting fact was discovered.

Developers were writing unit test cases not because they were convinced of their utility but because it was either mandated by management or because the customer required it.

Unit tests influential drivers

So when exactly is writing unit tests beneficial?

The table below represents, in our opinion, a reasonable list of criteria you could apply when trying to decide whether Unit Testing is the best approach to your testing problem.

CriteriaDescription
Is it self-containedA self-contained function does not depend on
other functions, which in turn might need
to be tested. This does not apply to helper
functions.
Is it self-sufficientA self-sufficient function or class has no external dependencies like a database, filesystem, or APIs.
Is it part of the user contractDoes it provide any functionality to the user, or is it solely used internally within the code?
Is the input range big enoughIn other words, can it be easily included with other tests?
Criteria for applying unit tests

Given the above, the example of the function that generates different character combinations is a perfect candidate.


6. Unit Testing Recommendations and Best Practices


7. Further Reading

Great talk on TDD by Ian Cooper.