Unit Testing: Is It Worth the Effort?

Georges Lteif

Georges Lteif

Software Engineer

Last Updated on May 19, 2022.
Subscribe now to stay posted!
About Us
12 min read

1. Overview

A few years back, I was involved in a major development project and it was absolutely crucial to get it right.

Like all development projects, it was too short on resources. To compensate for the resource shortage, we had to be extremely efficient in what we did.

Up until that moment, I had very limited experience with topics like Test-Driven Development (TDD) or Continuous Integration (CI).

After a decent amount of research, I convinced myself that integrating some or all of these in our workflow would give us a gigantic boost in performance.

software testing
Different layers of software testing

Test-Driven Development is an essential discipline that you can use to better manage resources allocation, test quality, and project deliveries.

TDD and Unit Testing

TDD relies heavily on the type of testing you use. It is generally inferred that A) Unit Testing should be the first choice, and B) the Unit of choice is the class or function.

We will challenge these two notions in this article and show that they are not just inefficient, but also costly and very counterproductive.

But before we continue, a note on Unit Testing.

Unit Testing as a Tool

Unit testing is just another tool that’s available for developers to improve the outcome of their software development processes, and like all the rest, it can be very efficient when used wisely.

The idea is to recognize those situations where unit testing achieves the best results and those other situations where its utterly unsuitable.

The longer answer will be the focus of the next sessions.


2. Table of Contents


3. What Is Unit Testing

Lets start with the basics.

3.1 Definition

Lets say you have a function that takes as input a variable number of characters like [‘a’, ‘b’, ‘c’] and returns all the possible combinations of these characters [‘ab’, ‘ac’, ‘bc’].

To make sure this function works as expected, you can test it by creating a set of dummy calls to that functions, with different inputs, and validating the output for all the different scenarios.

This routine is called Unit Testing.

To ensure the code works well under all circumstances (i.e. get decent test coverage), you have to consider all the below scenarios.

InputScenario
[‘a’, ‘b’, ‘c’]Happy scenario, which is probably what you will
get 99% of the time
[]Calling the function with no inputs
[0, 1, +, “aa”]Calling the function with incorrect argument type
[”, ‘a’, ‘b’, ‘c’]Incorrectly formatted input
[‘&’, ‘@’, ‘!’]You might want to perform some sanity checks
on the input, for example accept only alpha-numeric
characters.
[‘a’, ‘a’]Multiple occurrences allowed?
Unit Testing Example

Many frameworks (such as JUnit for Java or Google Test for C++) have been created to provide developers with the boiler-plate functionality, like test definition annotations, assert methods, and pass/fail reporting.

This allowed users to focus more on testing the business side of things.

On top of those frameworks, and as early as the 1980s, people felt the need for automation (check this article for timeline of software testing) and test automation frameworks were developed.

3.2 Unit Tests: A Waste-Generating Practice

Writing a few unit tests every now and then is never a problem, but that’s not usually how it works.

In a real-world scenario, especially with TDD, you are required to write massive amounts of unit tests for every new feature.

It’s up to the process creators to specify the granularity of the units, but usually, it’s on the class or function level.

Under these settings, Unit Testing can be a notorious time-waster.

Drawbacks of Unit Testing

The two main drawbacks of massive Unit Tests on the class level are

A) High Cost-of-Ownership

B) Slower Time-to-Market

Even if you have the latest and greatest tools and infrastructure, unit testing will still require time and effort to create and maintain.

But that’s not the whole story.

In fact, the problem can be exacerbated even more if your tests are slow, inefficient, and easy to break.

Operational Excellence has made finding and eliminating waste its holy grail. If that is something you are keen on achieving, perhaps unit testing is the best place to start.

3.3 Unit Testing and TDD

Applying TDD in the context of Operational Excellence is an idea that’s well worth exploring. And as we have seen, this will lead us straight to Unit Testing.

Test-Driven Development was historically coupled with Unit Testing, and the latter was focused primarily on classes and functions as its definition of a unit.

What Went Wrong with TDD

That proved to be fatal as it coupled the test code with implementation details.


The reason why that is a problem is as follows: implementation can change as often as it needs to (such as for refactoring purposes), but as long as behaviour does not change, there should be NO need to add or update your tests.

Keeping an implementation and its corresponding tests separate did not work well for the following reason: Unit Tests on the class level are, by design, coupled to implementation.

This meant that heavy unit testing was slowly abandoned as unpractical. TDD was subsequently declared dead by part of the community.

That’s the opposite of what we want.

In fact, we want TDD to be used as a primary tool for achieving Operational Excellence in Software Development.

This required the “issue” of unit testing to be resolved.


4. The Challenges of Unit Tests

To fully appreciate the challenges of unit testing, when implemented in the now traditional sense, we need to have a look at the root causes.

There are actually quite a few of these challenges, mainly: test coverage, test duration, testing implementation rather than functionality, database mocking, and testing legacy code.

Challenges unit testing worth effort
Challenges of unit testing

4.1 Testing Implementation Instead of Behavior

Working on that project which I mentioned earlier, I found myself constantly refactoring my code. In fact, refactoring constituted something like 20-30% of my development time.

Refactoring is what keeps technical debt at bay and improves the quality of your code; it needs to be done smoothly and efficiently.

Test functionality, not implementation.

It was quite easy to notice that being able to fully rely on existing test suites to immediately validate my code changes, I was moving at the momentum that I deemed sufficient.

I believe that what made the greatest difference was my ability to refactor the code without having to touch the test cases.

Functionality (or behaviour) is the system’s contract with the user. It’s the business functionality that it is supposed to deliver. Implementation, on the other hand, is transparent to the user.

To drive this point through, I need to mention the issue of black- vs white-box testing.

If your testing requires knowledge of the system’s internal details, then you are working with white box testing.

On the other hand, if your testing does not require any knowledge aside from the functionality being tested, then you are using a black box testing approach.

White-box testing is, by definition, implementation dependant, and that is not ideal in the scheme of things we are describing.

4.2 Test Coverage

It is very easy to see, from the example we used to demonstrate how Unit Testing works, how one simple function may require a large number of test cases to ensure decent coverage.

Ian Cooper in this talk mentions test code 2-3 times the size of the production code.

Imagine how much time was spent creating tens, if not hundreds, of thousands of test cases!

In a survey conducted by Ermira Daka and Gordon Fraser from the University of Sheffield, the following statistics were observed:

So, how do you generate decent coverage without spending an eternity writing unit tests?

Again, the answer has to do with the definition of “unit” and the coupling of test code with implementation details.

4.3 Test Speed

Ideally, you want to run your test suites on every commit, or at least once a day. That may be part of your Agile or DevOps requirements: faster iterations and quicker feedback.

If your tests cannot be completed in a few minutes, they are as good as useless.

If you run your suite once per day, during the night, imagine how many commits were done throughout the day and how difficult it would be to trace back an issue to a specific commit.

4.4 Mocking Dependencies

Mocking means replacing any dependencies that your class has on external systems.

An external system can be any of the below:

  1. Database
  2. Filesystem
  3. External API
  4. Third-party library

Why Is Mocking Bad?

Dependency mocking means you have to write test code that emulates your dependencies which are, by definition, part of the implementation.

As we have seen in the previous section, if behaviour doesn’t change, your tests shouldn’t too, even if the underlying implementation has changed.

In fact, in the same survey, researchers reported the following observation:

If you have not tried it before, you only need to spend very little time doing it to appreciate how difficult and time consuming it can be.

You change one column in a certain table, and then some hundreds of test cases need to be refactored.

Again, this is direct implication of the choice of the basic test “unit”.

4.5 Unit Testing Legacy Code

It is not unreasonable to think about moving an all-manual, massive testing exercise on legacy code to something more efficient and more reliable.

Can it be automated Unit Testing?

Successfully completing a unit test project on legacy code is outright impossible (if not crazy). But you still need to test that software, so what do you do?

You cant possibly write unit tests on legacy classes and functions because it would be a nightmare.

The solution, in our opinion, is to use System Integration Testing (or SIT) with as much automation as possible as part of your Testing Strategy.


5. When To Use Unit Testing

In the same survey by the University of Sheffield researchers, a very interesting fact was discovered.

Developers were, in fact, writing unit test cases, not because they were convinced of their utility, but because it was either mandated by management or because the customer required it.

Unit tests influential drivers

So when exactly is writing unit tests beneficial?

The below table represents, in our opinion, a reasonable list if criteria that you could apply when trying to decide whether Unit Testing is the best approach to your testing problem.

CriteriaDescription
Is it self-containedA self-contained function does not depend on
other functions which in turn might need
to be tested. This does not apply to helper
functions.
Is it self-sufficientA self-sufficient function or class does not have any external dependencies like database, filesystem, or APIs.
Is it part of the user contractDoes it provide any functionality to the user or is it solely used internally, within the code.
Is the input range big enoughIn other words, can it be easily included with other tests?
Criteria for applying unit tests

Given the above, the example of the function that generates different character combinations is a perfect candidate.


6. Unit Testing Recommendations and Best Practices


7. Further Reading

Great talk on TDD by Ian Cooper.


Technical Risk Management and Decision Analysis — Introduction and Fundamental Principles

1. Overview I could not find a better way to start an article on Risk and Risk Management than by quoting the opening lines of Donald Lessard and Roger Miller’s 2001 paper that, briefly but lucidly, summarizes the nature of large engineering endeavours. It goes like this: This article leans heavily on three handbooks thatContinue reading “Technical Risk Management and Decision Analysis — Introduction and Fundamental Principles”

Complexity and Complex Systems From Life on Earth to the Universe: A Brief Introduction

1. Overview Dealing with complexity is an integral part of our lives, even if we do not realise it.  An organisation can be modelled as a complex system from the scale of megacorporations right down to the smallest teams. The architecture of software solutions can be equally complicated, and megaprojects and implementations are certainly involved.Continue reading “Complexity and Complex Systems From Life on Earth to the Universe: A Brief Introduction”

Book Review: Programming the Universe — A Quantum Computer Scientist Takes on the Cosmos

Synopsis Most physical theories adopt a mechanistic view when examining natural phenomena where any system can be modelled as a machine whose initial conditions and dynamics govern its future behaviour. In this book, Programming the Universe — A Computer Scientist Takes on the Cosmos, Professor Seth Lloyd proposes a radically different approach centred around aContinue reading “Book Review: Programming the Universe — A Quantum Computer Scientist Takes on the Cosmos”

From Abstract Concepts to Tangible Value: Software Architecture in Modern IT Systems

1. Overview Software design and architecture are two very elusive concepts; even Wikipedia’s entries (ref. architecture, design) are somewhat fuzzy and do not clearly distinguish between the two. The Agile manifesto’s statement on architecture and design is especially brief and raises more questions than answers. The most common definition of software architecture is as follows:Continue reading “From Abstract Concepts to Tangible Value: Software Architecture in Modern IT Systems”

Business Requirements and Stakeholder Management: An Essential Guide to Definition and Application in IT Projects

1. Overview The complexity of business requirements in IT projects has experienced exponential growth due to pressures by increasingly sophisticated client preferences, novel technologies, and fierce competition. Consider, for example, the case of financial payments. In the mid-80s, most payment transactions occurred inside bank branches, and only the biggest banks offered services on ATM orContinue reading “Business Requirements and Stakeholder Management: An Essential Guide to Definition and Application in IT Projects”

Loading…

Something went wrong. Please refresh the page and/or try again.