Test and Automation Strategy: A Deep-Dive Into an Essential Solution for Your Daily Agile Practices

1. Overview

Software Testing and Quality Assurance is an attractive topic from a software delivery perspective for a very simple reason: it’s an area where you can make substantial gains and improvements in delivery capabilities.

Why? Because of the current state of software testing, a topic we have analysed in great detail.

In summary, software testing is still a predominantly manual and labour-intensive activity. Due to human error, inadequate coverage, and software complexity, its outcomes are also subpar. You add that to the high costs involved, and you get a toxic mix that needs to be eliminated.

However, when properly conducted, software testing can be efficient and add business value to your product. To move from a state of suboptimal test quality and high cost to one where testing is a value-adding activity, you must have a Test Strategy covering automation.

We have argued in our discussions on Agile and DevOps that automation is necessary if those practices are to succeed. This mandatory requirement makes test automation a central pillar of any testing strategy.

Both of these topics, Test Strategy and Test Automation will be the central pillars of our present discussion. In particular, we will aim to answer the following question:

What constitutes a Test and Automation Strategy
Why it’s essential to have one
Why is a Test Automation Strategy crucial for Agile and DevOps?
What are the technical requirements for automating your software testing
Finally, risks and challenges to be aware of when introducing test automation

Software Testing and Quality Assurance: A Modern Analysis of Its Internal Dynamics and Impact on Delivery

2. Automation – Hype or Necessity?

2.1 Why Is Automation Vital for Your Testing

If you are still not convinced that automation is a mandatory requirement of your SDLC, perhaps looking at how counterproductive manual testing is can help you change your mind:

Firstly, manual testing is a massive drain on resources. Most of it is just legwork with little or no creativity. Ambitious and capable staff cannot be easily retained, and you will always find yourself requiring additional resources as product complexity and its testing increase over time.
Secondly, manual testing is a bottleneck in the SDLC, especially if the product is not modular and extensive regression testing needs to be made. You can parallel run development tasks, but testing can only happen once all development is completed. This constraint imposes a sequential nature on your delivery methodology, which raises the cost of change.
Thirdly, Agile sprints are typically short, making it immensely difficult to incorporate lengthy manual testing. One of the primary deviations from classic Waterfall project management in Agile and DevOps is the adoption of shorter delivery iterations with frequent software drops. Agile sprints run in 2-4 weeks, and Agile teams must design, develop, test, and deliver new software features during this relatively short time. Adopting Agile practices is one of the principles of Operational Excellence we advocate.
Fourthly, the reliability of manual testing is highly questionable. Reliability will also probably vary between teams and projects depending on the quality of the testers and processes employed. Ideally, we would like consistent processes and reliable outcomes as much as possible. Reliability can increase only if outcome variance decreases. Inconsistencies in processes and results will make process improvements much harder than they need to be.

DevOps: Finding Your Path to Successful and Continuous Improvement

2.2 Problems That Test Automation Resolves

So what can you achieve by automating all your testing? Here are some of the problems that automation can resolve:

Wider Coverage. By eliminating manual labour, QA staff can add more test cases for broader coverage.
Higher Reliability: Automation allows the reproduction of the same test environment and test results. This objective is easier today with containerization and cloud services.
Robust Results: Testing the application on multiple platforms and under different application settings (client-dependent settings) makes your test results more robust than testing it in a generic environment. You might also want to try deployment and upgrade procedures on every new release.
Shorter Iterations and Quicker Feedback can significantly improve your delivery capabilities by removing the sequential constraint from your delivery pipeline. This capability improves resource allocation, shortens delivery timeframes, and lets customers get their hands dirty as early as possible.
Increasing Business Value: Manual, time-consuming, repetitive testing has become a luxury that most, if not all, IT organizations cannot afford anymore. Companies implementing automation have a strategic edge over the competition because of their lower price-performance ratio.

2.3 Obstacles and Challenges

Transformation is never easy, but it’s also not something you can ignore if you want to stay in the game.

Here are some challenges you might face when migrating to an automated testing setup. They are inspired by firsthand experience and online sources such as the 15th State of Agile Report, which describes obstacles to adopting Agile.

An initial investment in Skills, Tools, and Infrastructure is probably the first barrier to overcome. Most organizations would have to build up the skills and infrastructure (sometimes from scratch), which may require a significant investment in time and effort.
Migration to automated testing can also be risky and disruptive for the business. Depending on the organization’s risk appetite and how much psychological safety it has for executing transformations, senior management may opt out of automation instead of taking on any risks it might involve.
Commitment from Leadership: as with any major change, securing the budget and resources is essential for success. You also need continuous backup and support if results are not evident in short timeframes.
Legacy Products and Technologies: Building a test suite with decent coverage for massive legacy systems can be daunting to kick off simply due to the size of the initial effort, especially if coupled with poor documentation and distributed or lost knowledge. Another potential problem with legacy products is that they were never written to be tested, a concept that we return to later under Coding with Automation in Mind.
Getting Everybody On Board: Organisational culture plays a large part in driving or inhibiting change. People can feel threatened by new technologies, which they believe might render their roles obsolete. Part of introducing automation, for example, requires re-inventing the QA role.

2.4 Addressing Some of the Challenges

Addressing the challenges accompanying significant change would need to start with a profound appreciation for the risk to the business’s survival by not implementing that change. Below are some thoughts on how the challenges described earlier can be met:

The initial cost can be distributed over future projects regarding time and effort saved on manual testing and additional test cycles that could be avoided.
It is far easier and smoother to think of how an automated test system can be implemented around a new versus an existing product. If you have a greenfield project underway, immediately implement test automation.
Automation must be marketed and positioned as a genuine opportunity for growth for all stakeholders and not merely a cost-saving exercise (although that should follow when the system is fully operational). It’s also an opportunity to transform a bleeding cost into added value.

3. Test and Automation Strategy

3.1 Definition

A Test and Automation Strategy is a documented, published, and internal operations guide that describes how software testing must be conducted within the applied framework of the Software Development Lifecycle.

More specifically:

A Test and Automation strategy outlines the overall objective of software testing activities and how this help to produce valuable products that customers are happy to acquire. This first goal paints the big picture and helps people discover meaning in their actions.
The strategy defines the roles and responsibilities of the development and testing team, the processes that govern the testing, the different types of testing involved, the outcomes of test exercises, and the artefacts produced. Designing efficient and effective methods for software testing is integral to success.
A Test Automation Strategy provides a definite position on the different software testing methodologies and why they have been selected or excluded from the chosen processes. A perfect example is a position on Unit Testing, which we have amply described in this article.
The strategy also defines the rules of engagement between the development and testing teams. These rules are essential, primarily if you use techniques such as Test-Driven Development that require high collaboration between cross-functional teams.

It is important to note that there is no one-size-fits-all plan that works equally well across different industries (e-commerce, automotive, financial), platforms (web apps, desktop apps, embedded software, legacy software), technologies, or deployment options (cloud, hosted, onsite, embedded). You simply have to fill in the blanks as per your custom settings.

3.2 Properties of a Great Test and Automation Strategy

Now that we know a Test and Automation Strategy let’s see if we can articulate a few critical requirements for its success.

Clarity and Accessibility. A test strategy (or any such guides) must be published. Publishing internal processes remove any temptation to improvise and provides a solid baseline for continuous improvement.
Well Articulated: An excellent test and automation strategy uses unified terminology and precise language when addressing key issues. Critical elements of software testing such as type (User Acceptance, System Integration, Unit Testing…), approach (Black, Grey, or White box), and automation (Full, Semi, or None) need to be clearly articulated to avoid ambiguity and misunderstandings.
Finely Engineered Processes that would answer the below questions:
- How to prepare and execute a test plan
- What tools are used, and when
- What constitutes a successful/failed test case
- Who is involved in the test preparation, execution, and approval

3.3 What Does a Test Automation Strategy Cover

We have touched upon the key areas of what a Test and Automation Strategy includes. Let’s see if we can clarify the operational aspects of those areas a bit further.

Roles and Responsibilities of the staff involved: this describes how developers, testers, and team leads execute the testing processes and contribute to continuous improvement.

Techniques and Best Practices: This is a tabulated list of selected methods and a mapping to specific cases where these can be applied. This mapping table prescribes what tests need to be run and when. For example:
1. What are the advantages and limitations of Test-Driven Development or Black / White Box testing, and what is the best scenario for each use case?
2. Under what conditions and at which stage would Stress (Performance) Testing be applicable in the SDLC process?
3. What are the criteria to determine the Success or Failure of the test run?

Tools and Technologies: This ensures consistency across the different teams and products. Examples of tools or libraries include:
1. Unit Test libraries for Java-based code (JUnit or TestNG)
2. Optical Character Recognition (OCR) libraries for image validation of printed data
3. Automation frameworks, Robot Framework, are one example.
4. Online collaboration or software versioning tools: JIRA, GitLab, Bitbucket, etc.
5. Hardware tools (such as robots)

Process Flows. For example:
1. Preparation, execution, and documentation of test plans
2. Reporting bugs (steps to reproduce, gathering logs, success criteria)
3. Test approval processes
4. Ensuring reusability of test cases across products or versions
5. Business requirement validation and verification

4. Implementing the Test and Automation Strategy

4.1 Overview

Now that we know what the Test and Automation Strategy wishes to achieve, let’s look at some practical thoughts on how best this could be done.

4.2 Tools and Infrastructure

Today’s market offers a wide variety of tools essential to implementing a test and automation strategy. Without going into too many details about those tools, let’s look at the broad categories they fall into and how these could make your implementation faster and easier.

Sophisticated tools like LoadRunner or SoapUI that help you generate custom requests suitable for regression of stress testing are already available as premium and commercial products. If it makes sense, you can also write your tools with high-level programming languages like Python or Robot Framework.
Containerization is a significant help for organising and managing your test environments. Containerization tools like Docker allow you to write scripts that can be used to generate lab environments ready for immediate use. Because Docker uses a command-line interface and text files for its scripts, it can be easily integrated with other tools like Robot Framework.
Cloud technology and Software as a Service (SaaS) offerings have brought a wealth of powerful and affordable tools to your doorstep. You can set up your DevOps CI/CD pipelines on Gitlab or AWS and access them from any workstation with an internet connection.

If you haven’t worked with these tools before, they might sound scary, but you will notice how intuitive and valuable they are as soon as you start using them.

4.3 Custom Framework

Having all the tools and infrastructure you need to kick off your test and automation strategy, you still need one ingredient to glue these components together and provide a working solution: a custom framework of rules, code, and scripts.

Rules govern the development and testing processes to achieve the objectives outlined in the strategy.
You will inevitably need to write custom code specifically for testing purposes, including code that supports building and executing unit tests. As we shall see a bit further, you might have to write code that, for example, facilitates the validation of function outputs.
Scripts will form the core of your test scripts. For example, they might also be used to provide generic utilities like creating transaction requests.

4.4 Methods and Techniques

4.4.1 Test-Driven Development

We have dedicated an article to the crucial topic of Test-Driven Development (TDD), and we invite you to read it before creating or implementing a Test and Automation Strategy. In essence, TDD allows you to:

Optimize your resource usage by allowing testers to start working as soon as the solution design is signed off and in parallel with development work.
Ensure that testing does not lag behind development. This requirement is crucial if you want your test coverage to remain adequate for the product’s whole life.
Incentivize developers to write code that can be easily tested.

4.4.2 Test Functionality, Not Implementation

It is crucial to write test cases that do not depend on implementation. Generally, test cases should only change if the behaviour changes.

This rule is vital for reducing the cost of ownership of your test suites. As you refactor your original code to keep it clean and up-to-scratch, you will use your battery of test cases to ensure functionality is intact. This advantage will disappear if code refactoring induces a refactoring of test cases.

The Test Functionality Not Implementation motto is most applicable to unit testing, on which we have written a complete article. Another great example of avoiding implementation testing is dependency mocking, especially in databases. Database mocking is a deep rabbit hole that will skyrocket your ownership cost.

4.4.3 Code and Test Coverage

Code coverage is a smart proxy to test coverage. Plugins exist today, allowing you to see which code areas were adequately covered by unit tests. This information is vital for constructing additional test cases to fill testing gaps.

4.4.4 Continuous Integration (CI)

The following graph from Google Trends shows a constant increase in interest in the term “CI/CD pipeline” for the past 5 years.

If you are unsure what Continuous Integration means, we recommend reading our DevOps and Continuous Delivery article. The below definition of CI by Atlassian will probably do for now:

Continuous integration (CI) is the practice of automating the integration of code changes from multiple contributors into a single software project. The CI process comprises automatic tools that assert the new code’s correctness before integration. A source code version control system is the crux of the CI process. The version control system is supplemented with other checks like automated code quality tests, syntax style review tools, and more.

DevOps (with its CI/CD pipelines) is beyond Agile project delivery.

Coupled with Test Automation, CI is a powerful device that dramatically improves your software delivery capabilities by allowing smaller features to be automatically built and rigorously tested with minimal effort.

As we have argued repeatedly in our articles, Agile and DevOps can never work without test automation. Therefore, it is essential to remember that the project delivery methodology you intend to adopt will closely work with any test automation framework you will put in place.

In summary, Agile and DevOps, on the one hand, and automation, on the other, must be closely compatible and designed under one strategy.

4.5 Coding With Automation in Mind

We have repeatedly mentioned the importance of coding with automation in mind in the previous sections; now, it’s time to put some details into the discussion.

Automation requires active participation from developers, not just testers or DevOps engineers. Ultimately, automation requires a machine to execute tests and validate outcomes that need to be in an accessible, machine-readable format.

We can think of any code unit, such as a method, as a transformation that takes inputs, applies business rules to these inputs, and produces outputs.

Automation involves having a third-party orchestrator fire up the application(s) under test, push inputs into that service path, collect outputs, and apply validation rules on those outputs. Advanced automation features can also generate automated reports, email them, and open bug tickets when test cases fail.

software automation testing — Coding with Automation in Mind

The figure above shows an example of inputs that can be manipulated by automation scripts and outputs that can be easily validated.

A properly designed test automation framework should allow developers or testers to effortlessly produce test cases with different inputs (simulating different scenarios) and validation rules around any artifact produced by the software component (log file, database, system state, etc.).

Let’s take a few examples to demonstrate the idea.

test automation scripts — Test Automation Scripts

Output: probably the first item to inspect. Typically easy to interpret and validate.
Log files of a semi-structured nature (do not have to be rigid or constrained) for log messages would make it much easier to look for errors and abnormalities. System issues (memory, hardware, or database) are often logged in a file and provide an interesting information source.
Database records: transactions running on a system may change records, which could be queried for validation. Transaction logs are great sources of information on system behaviour.
System state: this is beneficial to inspect when actions or inputs to the system would change its internal conditions. For example, exposing the system’s internal state through APIs would make this type of information available for validation by test units.
Integrations with third-party systems: it is not uncommon for a service to interact on a transaction basis with other systems in the ecosystem. This is generally done via a request/response paradigm. The response can take different shapes and forms (JSON, XML, character stream, etc.). A good design will find a way to expose these message pairs for validation.

5. Final Words

It is undeniable that automation (in testing and otherwise) is a mandatory requirement for companies to stay in the game; it is just impossible to keep ramping up manual efforts to maintain product quality on par with progress.

With the proliferation of methods, tools, and even philosophies on software delivery in general and testing and automation in particular, we hope this article has helped you build a solid groundwork for formulating a good testing strategy.

We believe that overarching software delivery concepts exist and serve as a fine comb to groom your thoughts and organize your priorities. We grouped these concepts under Operational Excellence in Software Development, and we believe implementing those concepts must include an automation component.

We hope these ideas have helped you formulate your opinions on the best course for your organization.

Principles of Operational Excellence in Software Development