Software Testing and Quality Assurance is an attractive topic from a software delivery perspective for a very simple reason: it’s an area where you can make substantial gains and improvements in delivery capabilities.
In summary, software testing is still a predominantly manual and labour-intensive activity. Due to human error, inadequate coverage, and software complexity, its outcomes are also subpar. You add that to the high costs involved, and you get a toxic mix that needs to be eliminated.
However, when properly conducted, software testing can be efficient and add business value to your product. To move from a state of suboptimal test quality and high cost to one where testing is a value-adding activity, you must have a Test Strategy covering automation.
We have argued in our discussions on Agile and DevOps that automation is necessary if those practices are to succeed. This mandatory requirement makes test automation a central pillar of any testing strategy.
Both of these topics, Test Strategy and Test Automation will be the central pillars of our present discussion. In particular, we will aim to answer the following question:
- What constitutes a Test and Automation Strategy
- Why it’s essential to have one
- Why is a Test Automation Strategy crucial for Agile and DevOps?
- What are the technical requirements for automating your software testing
- Finally, risks and challenges to be aware of when introducing test automation
Software Testing and Quality Assurance: A Modern Analysis of Its Internal Dynamics and Impact on Delivery
2. Automation – Hype or Necessity?
2.1 Why Is Automation Vital for Your Testing
If you are still not convinced that automation is a mandatory requirement of your SDLC, perhaps looking at how counterproductive manual testing is can help you change your mind:
2.2 Problems That Test Automation Resolves
So what can you achieve by automating all your testing? Here are some of the problems that automation can resolve:
2.3 Obstacles and Challenges
Transformation is never easy, but it’s also not something you can ignore if you want to stay in the game.
Here are some challenges you might face when migrating to an automated testing setup. They are inspired by firsthand experience and online sources such as the 15th State of Agile Report, which describes obstacles to adopting Agile.
2.4 Addressing Some of the Challenges
Addressing the challenges accompanying significant change would need to start with a profound appreciation for the risk to the business’s survival by not implementing that change. Below are some thoughts on how the challenges described earlier can be met:
3. Test and Automation Strategy
- A Test and Automation strategy outlines the overall objective of software testing activities and how this help to produce valuable products that customers are happy to acquire. This first goal paints the big picture and helps people discover meaning in their actions.
- The strategy defines the roles and responsibilities of the development and testing team, the processes that govern the testing, the different types of testing involved, the outcomes of test exercises, and the artefacts produced. Designing efficient and effective methods for software testing is integral to success.
- A Test Automation Strategy provides a definite position on the different software testing methodologies and why they have been selected or excluded from the chosen processes. A perfect example is a position on Unit Testing, which we have amply described in this article.
- The strategy also defines the rules of engagement between the development and testing teams. These rules are essential, primarily if you use techniques such as Test-Driven Development that require high collaboration between cross-functional teams.
It is important to note that there is no one-size-fits-all plan that works equally well across different industries (e-commerce, automotive, financial), platforms (web apps, desktop apps, embedded software, legacy software), technologies, or deployment options (cloud, hosted, onsite, embedded). You simply have to fill in the blanks as per your custom settings.
3.2 Properties of a Great Test and Automation Strategy
Now that we know a Test and Automation Strategy let’s see if we can articulate a few critical requirements for its success.
- Clarity and Accessibility. A test strategy (or any such guides) must be published. Publishing internal processes remove any temptation to improvise and provides a solid baseline for continuous improvement.
- Well Articulated: An excellent test and automation strategy uses unified terminology and precise language when addressing key issues. Critical elements of software testing such as type (User Acceptance, System Integration, Unit Testing…), approach (Black, Grey, or White box), and automation (Full, Semi, or None) need to be clearly articulated to avoid ambiguity and misunderstandings.
- Finely Engineered Processes that would answer the below questions:
- How to prepare and execute a test plan
- What tools are used, and when
- What constitutes a successful/failed test case
- Who is involved in the test preparation, execution, and approval
3.3 What Does a Test Automation Strategy Cover
We have touched upon the key areas of what a Test and Automation Strategy includes. Let’s see if we can clarify the operational aspects of those areas a bit further.
- Roles and Responsibilities of the staff involved: this describes how developers, testers, and team leads execute the testing processes and contribute to continuous improvement.
- Techniques and Best Practices: This is a tabulated list of selected methods and a mapping to specific cases where these can be applied. This mapping table prescribes what tests need to be run and when. For example:
- What are the advantages and limitations of Test-Driven Development or Black / White Box testing, and what is the best scenario for each use case?
- Under what conditions and at which stage would Stress (Performance) Testing be applicable in the SDLC process?
- What are the criteria to determine the Success or Failure of the test run?
- Tools and Technologies: This ensures consistency across the different teams and products. Examples of tools or libraries include:
- Unit Test libraries for Java-based code (JUnit or TestNG)
- Optical Character Recognition (OCR) libraries for image validation of printed data
- Automation frameworks, Robot Framework, are one example.
- Online collaboration or software versioning tools: JIRA, GitLab, Bitbucket, etc.
- Hardware tools (such as robots)
- Process Flows. For example:
- Preparation, execution, and documentation of test plans
- Reporting bugs (steps to reproduce, gathering logs, success criteria)
- Test approval processes
- Ensuring reusability of test cases across products or versions
- Business requirement validation and verification
4. Implementing the Test and Automation Strategy
Now that we know what the Test and Automation Strategy wishes to achieve, let’s look at some practical thoughts on how best this could be done.
4.2 Tools and Infrastructure
Today’s market offers a wide variety of tools essential to implementing a test and automation strategy. Without going into too many details about those tools, let’s look at the broad categories they fall into and how these could make your implementation faster and easier.
If you haven’t worked with these tools before, they might sound scary, but you will notice how intuitive and valuable they are as soon as you start using them.
4.3 Custom Framework
Having all the tools and infrastructure you need to kick off your test and automation strategy, you still need one ingredient to glue these components together and provide a working solution: a custom framework of rules, code, and scripts.
4.4 Methods and Techniques
4.4.1 Test-Driven Development
We have dedicated an article to the crucial topic of Test-Driven Development (TDD), and we invite you to read it before creating or implementing a Test and Automation Strategy. In essence, TDD allows you to:
4.4.2 Test Functionality, Not Implementation
This rule is vital for reducing the cost of ownership of your test suites. As you refactor your original code to keep it clean and up-to-scratch, you will use your battery of test cases to ensure functionality is intact. This advantage will disappear if code refactoring induces a refactoring of test cases.
The Test Functionality Not Implementation motto is most applicable to unit testing, on which we have written a complete article. Another great example of avoiding implementation testing is dependency mocking, especially in databases. Database mocking is a deep rabbit hole that will skyrocket your ownership cost.
4.4.3 Code and Test Coverage
Code coverage is a smart proxy to test coverage. Plugins exist today, allowing you to see which code areas were adequately covered by unit tests. This information is vital for constructing additional test cases to fill testing gaps.
4.4.4 Continuous Integration (CI)
The following graph from Google Trends shows a constant increase in interest in the term “CI/CD pipeline” for the past 5 years.
Continuous integration (CI) is the practice of automating the integration of code changes from multiple contributors into a single software project. The CI process comprises automatic tools that assert the new code’s correctness before integration. A source code version control system is the crux of the CI process. The version control system is supplemented with other checks like automated code quality tests, syntax style review tools, and more.
DevOps (with its CI/CD pipelines) is beyond Agile project delivery.
Coupled with Test Automation, CI is a powerful device that dramatically improves your software delivery capabilities by allowing smaller features to be automatically built and rigorously tested with minimal effort.
As we have argued repeatedly in our articles, Agile and DevOps can never work without test automation. Therefore, it is essential to remember that the project delivery methodology you intend to adopt will closely work with any test automation framework you will put in place.
In summary, Agile and DevOps, on the one hand, and automation, on the other, must be closely compatible and designed under one strategy.
4.5 Coding With Automation in Mind
We have repeatedly mentioned the importance of coding with automation in mind in the previous sections; now, it’s time to put some details into the discussion.
Automation requires active participation from developers, not just testers or DevOps engineers. Ultimately, automation requires a machine to execute tests and validate outcomes that need to be in an accessible, machine-readable format.
We can think of any code unit, such as a method, as a transformation that takes inputs, applies business rules to these inputs, and produces outputs.
Automation involves having a third-party orchestrator fire up the application(s) under test, push inputs into that service path, collect outputs, and apply validation rules on those outputs. Advanced automation features can also generate automated reports, email them, and open bug tickets when test cases fail.
The figure above shows an example of inputs that can be manipulated by automation scripts and outputs that can be easily validated.
A properly designed test automation framework should allow developers or testers to effortlessly produce test cases with different inputs (simulating different scenarios) and validation rules around any artifact produced by the software component (log file, database, system state, etc.).
Let’s take a few examples to demonstrate the idea.
5. Final Words
It is undeniable that automation (in testing and otherwise) is a mandatory requirement for companies to stay in the game; it is just impossible to keep ramping up manual efforts to maintain product quality on par with progress.
With the proliferation of methods, tools, and even philosophies on software delivery in general and testing and automation in particular, we hope this article has helped you build a solid groundwork for formulating a good testing strategy.
We believe that overarching software delivery concepts exist and serve as a fine comb to groom your thoughts and organize your priorities. We grouped these concepts under Operational Excellence in Software Development, and we believe implementing those concepts must include an automation component.
We hope these ideas have helped you formulate your opinions on the best course for your organization.