Software Testing and Quality Assurance is an attractive topic from a software delivery perspective for a very simple reason: it’s an area where you can make substantial gains and improvements in delivery capabilities.
In summary, software testing is still a predominantly manual and labour-intensive activity. Due to human error, inadequate coverage, and software complexity, its outcomes are also subpar. You add that to the high costs involved, and you get a toxic mix that needs to be eliminated.
However, when properly conducted, software testing can be efficient and can add business value to your product.
To move from a state of suboptimal test quality and high cost to one where testing is a value-adding activity, you need to have a Test Strategy, which needs to cover automation.
We have argued in our discussions on Agile and DevOps that automation is a necessary ingredient if those practices were to become successful. This mandatory requirement makes test automation a central pillar of any testing strategy.
Both of these topics, Test Strategy and Test Automation, will be the central pillars of our present discussion. In particular, we will aim to answer the following question:
- What constitutes a Test and Automation Strategy
- Why it’s essential to have one
- Why is a Test Automation Strategy crucial for Agile and DevOps
- What are the technical requirements for automating your software testing
- Finally, risks and challenges to be aware of when introducing test automation
2. Table of Contents
- 1. Overview
- 2. Table of Contents
- 2. Automation – Hype or Necessity?
- 3. Test and Automation Strategy
- 4. Implementing the Test and Automation Strategy
- 5. Final Words
- 6. Featured Articles
2. Automation – Hype or Necessity?
2.1 Why Is Automation Vital for Your Testing
If you are still not convinced that automation is a mandatory requirement of your SDLC, perhaps looking at how counterproductive manual testing is can help you change your mind:
- Firstly, manual testing is a massive drain on resources. Most of it is just leg-work with little or no creativity. Ambitious and capable staff cannot be easily retained, and you will always find yourself requiring additional resources as product complexity and its testing increase over time.
- Secondly, manual testing is a bottleneck in the SDLC, especially if the product is not modular and extensive regression testing needs to be made. You can parallel run development tasks, but testing can only happen once all development is completed. This constraint imposes a sequential nature on your delivery methodology, which inherently raises the cost of change.
- Thirdly, Agile sprints typically are of short duration, making it immensely difficult to incorporate lengthy manual testing. One of the primary deviations from classic Waterfall project management in Agile and DevOps is the adoption of shorter delivery iterations with frequent software drops. Agile sprints run in 2-4 weeks, and Agile teams need to design, develop, test, and deliver new software features during this relatively short time. Adopting Agile practices is one of the principles of Operational Excellence we advocate.
- Fourthly, the reliability of manual testing is highly questionable. Reliability will also probably vary between teams and projects depending on the quality of the testers and processes employed. Ideally, we would like to have consistent processes and reliable outcomes as much as possible. Reliability can increase only if outcome variance decreases. Inconsistencies in processes and results will make any process improvements much harder than they need to be.
2.2 Problems That Test Automation Resolves
So what can you achieve by automating all your testing? Here are some of the problems that automation can resolve:
- Wider Coverage. By eliminating manual labour, QA staff can add more test cases for broader coverage.
- Higher Reliability: Automation allows the reproduction of the same test environment and test results all the time. This objective is easier today with containerization and cloud services.
- Robust Results: Testing the application on multiple platforms and under different application settings (client-dependent settings) makes your test results more robust than testing it in a generic environment. You might also want to try deployment and upgrade procedures on every new release.
- Shorter Iterations and Quicker Feedback can significantly improve your delivery capabilities by removing the sequential constraint from your delivery pipeline. This capability improves resource allocation, shortens delivery timeframes, and allows customers to get their hands dirty as early as possible.
- Increasing Business Value: Manual, time-consuming, repetitive testing has become a luxury that most, if not all, IT organizations could not afford anymore. Companies implementing automation have a strategic edge over the competition because of their lower price-performance ratio.
2.2 Obstacles and Challenges
Change is never easy, but it’s also not something you can ignore if you want to stay in the game.
Here are some of the challenges you might face when migrating to an automated testing setup. They are inspired by firsthand experience and online sources such as the 15th State of Agile Report which describes obstacles to the adoption of Agile.
- An initial investment in Skills, Tools, and Infrastructure is probably the first barrier to overcome. Most organizations would have to build up the skills and infrastructure (sometimes from scratch), which may require a significant investment in time and effort.
- Migration to automated testing can also be risky and disruptive for the business. Depending on the organization’s risk appetite and how much psychological safety it has for executing transformations, senior management may decide to opt-out of automation instead of taking on any risks it might involve.
- Commitment from Leadership: as with any major change, securing the budget and resources is essential for success. You also need continuous backup and support if results cannot be evident in short timeframes.
- Legacy Products and Technologies: Building a test suite with decent coverage for massive legacy systems can be pretty daunting to kick off simply due to the size of the initial effort, especially if coupled with poor documentation and distributed or lost knowledge. Another potential problem with legacy products is that they were never written to be tested, a concept which we return to later under Coding with Automation in Mind.
- Getting Everybody On Board: Organizational culture plays a large part in driving or inhibiting change. People can feel threatened by new technologies, which they believe might render their roles obsolete. Part of introducing automation, for example, requires re-inventing the QA role.
2.3 Addressing Some of the Challenges
Addressing the challenges accompanying significant change would need to start with a profound appreciation for the risk to the business’ survival from not implementing that change.
Below are some thoughts on how the challenges described earlier can be met:
- The overall initial cost can be distributed over future projects in terms of time and effort saved on manual testing and additional test cycles that could be avoided.
- It is far easier and smoother to think of how an automated test system can be implemented around a new versus an existing product. If you have a greenfield project underway, immediately incorporate test automation into its implementation.
- Automation would need to be marketed and positioned as a genuine opportunity for growth for all stakeholders and not merely a cost-saving exercise (although that should follow when the system is fully operational). It’s also an opportunity to transform a bleeding cost into added value.
3. Test and Automation Strategy
- A Test and Automation strategy outlines the overall objective of software testing activities and how this help to produce valuable products that customers are happy to acquire. This first goal paints the big picture and helps people discover meaning in what they do.
- The strategy defines the roles and responsibilities of the development and testing team, the processes that govern the testing, the different types of testing involved, the outcomes of test exercises, and the artefacts produced. Designing efficient and effective methods for software testing is integral to success.
- A Test Automation Strategy provides a definite position on the different software testing methodologies and the reasons as to why they have been selected or excluded from the chosen processes. A perfect example is the position on Unit Testing, which we have amply described in this article.
- The strategy also defines the rules of engagement between the development and testing teams. These rules are essential, primarily if you use techniques such as Test-Driven Development that require a high degree of collaboration between cross-functional teams.
It is important to note that there is no one-size-fits-all plan that works equally well across different industries (e-commerce, automotive, financial), platforms (web apps, desktop apps, embedded software, legacy software), technologies, or deployment options (cloud, hosted, onsite, embedded). You simply have to fill in the blanks as per your custom settings.
3.2 Properties of a Great Test and Automation Strategy
Now that we know what a Test and Automation Strategy is, let’s see if we can articulate a few critical requirements for its success.
- Clarity and Accessibility. A test strategy (or any such guides) must be published. Publishing internal processes removes any temptation to improvise and provides a solid baseline for continuous improvement.
- Well Articulated: An excellent test and automation strategy uses unified terminology and precise language when addressing key issues. Critical elements of software testing such as type (User Acceptance, System Integration, Unit Testing…), approach (Black, Grey, or White box), automation (Full, Semi, or None) need to be clearly articulated to avoid ambiguity and misunderstandings.
- Finely Engineered Processes that would answer the below questions:
- How to prepare and execute a test plan
- What tools are used and when
- What constitutes a successful/failed test case
- Who is involved in the test preparation, execution, and approval
3.3 What Does a Test Automation Strategy Cover
We have touched upon the key areas of what a Test and Automation Strategy includes. Let’s see if we can clarify the operational aspects of those areas a bit further.
- Roles and Responsibilities of the staff involved: this describes how developers, testers, and team leads execute the testing processes and contribute to continuous improvement.
- Techniques and Best Practices: This is essentially a tabulated list of selected methods and a mapping to specific cases where these can be applied. This mapping table prescribes what tests need to be run and when. For example:
- What are the advantages and limitations of Test Driven Development or Black / White Box testing, and what is the best scenario for each use case?
- Under what conditions and at which stage in the SDLC process would Stress (Performance) Testing be applicable?
- What are the criteria to determine the Success or Failure of the test run?
- Tools and Technologies: This ensures consistency across the different teams and products. Examples of tools or libraries include:
- Unit Test libraries for Java-based code (JUnit or TestNG)
- Optical Character Recognition (OCR) libraries for image validation of printed data
- Automation frameworks, Robot Framework, being one example.
- Online collaboration or software versioning tools: JIRA, GitLab, Bitbucket, etc.
- Hardware tools (such as robots)
- Process Flows. For example:
- Preparation, execution, and documentation of test plans
- Reporting bugs (steps to reproduce, gathering logs, success criteria)
- Test approval processes
- Ensuring reusability of test cases across products or versions
- Requirement validation and verification
4. Implementing the Test and Automation Strategy
Now that we know what the Test and Automation Strategy wishes to achieve let’s look at some practical thoughts on how best this could be done.
4.2 Tools and Infrastructure
Today’s market offers a wide variety of tools that are essential to implementing a test and automation strategy.
Without going into too many details about those tools, let’s look at the broad categories they fall into and how these could make your implementation faster and easier.
- Sophisticated tools like LoadRunner or SoapUI that help you generate custom requests suitable for regression of stress testing are already available as premium and commercial products. You can also write your own tools if it makes sense with high-level programming languages like Python or Robot Framework.
- Containerization is a significant help for organizing and managing your test environments. Containerization tools like Docker allow you to write scripts that can be used to generate lab environments ready for immediate use. Because Docker uses a command-line interface and text files for its scripts, it can be easily integrated with other tools like Robot Framework.
- Cloud technology and Software as a Service (SaaS) offerings have brought a wealth of powerful and affordable tools to your doorstep. You can set up your DevOps CI/CD pipelines on Gitlab or AWS and access them from any workstation with an internet connection.
If you haven’t worked with these tools before, they might sound scary, but you will notice how intuitive and valuable they are as soon as you start using them.
4.3 Custom Framework
Having all the tools and infrastructure that you need to kick off your test and automation strategy, you still need one ingredient to glue these components together and provide you with a working solution: a custom framework of rules, code, and scripts.
- Rules govern the development and testing processes to achieve the objectives outlined in the strategy.
- You will inevitably need to write custom code specifically for testing purposes, including code that supports building and executing unit tests. As we shall see a bit further, you might have to write code that, for example, facilitates the validation of function outputs.
- Scripts will form the core of your test scripts. For example, they might also be used to provide generic utilities like creating transaction requests.
4.4 Methods and Techniques
4.4.1 Test-Driven Development
We have dedicated an article to the crucial topic of Test-Driven Development (TDD), and we invite you to read it before creating or implementing a Test and Automation Strategy.
In essence, TDD allows you to:
- Optimize your resource usage by allowing testers to start working as soon as the solution design is signed off and in parallel with development work.
- Ensure that testing does not lag behind development. This requirement is crucial if you want your test coverage to remain adequate for the product’s whole life.
- Incentivize developers to write code that can be easily tested.
4.4.2 Test Functionality Not Implementation
It is crucial to write test cases that do not depend on implementation. As a general rule, test cases should only change if the behaviour changes.
This rule is vital for reducing the cost of ownership of your test suites. As you refactor your original code to keep it clean and up-to-scratch, you will use your battery of test cases to ensure that functionality is intact. This advantage will disappear if code refactoring induces a refactoring of test cases.
The Test Functionality Not Implementation motto is most applicable to unit testing, on which we have written a complete article.
Another great example of avoiding implementation testing is dependency mocking, especially databases. Database mocking is a deep rabbit hole that will skyrocket your ownership cost.
4.4.3 Code and Test Coverage
Code coverage is a smart proxy to test coverage. Plugins exist today that will allow you to see which areas of the code were adequately covered by unit tests. This information is vital for constructing additional test cases to fill any testing gaps.
4.4.4 Continuous Integration (CI)
The following graph from Google Trends shows a constant increase in interest in the term “CI/CD pipeline” for the past 5 years.
If you are not quite sure what Continuous Integration means, we recommend reading our article on DevOps and Continuous Delivery. The below definition of CI by Atlassian will probably do for now:
Continuous integration (CI) is the practice of automating the integration of code changes from multiple contributors into a single software project. The CI process is comprised of automatic tools that assert the new code’s correctness before integration. A source code version control system is the crux of the CI process. The version control system is also supplemented with other checks like automated code quality tests, syntax style review tools, and more.
Coupled with Test Automation, CI is a powerful device that dramatically improves your software delivery capabilities by allowing smaller features to be automatically built and rigorously tested with minimal effort.
As we have argued repeatedly in our articles, Agile and DevOps can never work without test automation. Therefore, it is essential to remember that the project delivery methodology you intend to adopt will closely work with any test automation framework you will put in place.
In summary, Agile and DevOps, on one hand, and automation on the other need to be closely compatible and designed under one strategy.
4.5 Coding With Automation in Mind
We have repeatedly mentioned the importance of coding with automation in mind in the previous sections; now, it’s time to put some details into the discussion.
Automation requires active participation from developers and not just testers or DevOps engineers to work. In the end, automation requires a machine to execute tests and validate outcomes that need to be in an accessible, machine-readable format.
We can think of any code unit such as a method as a transformation that takes inputs, applies business rules to these inputs, and produces outputs.
Automation involves having a third-party orchestrator fire up the application(s) under test, push inputs into that service path, collect outputs, and apply validations rules on those outputs. Advanced automation features can also generate automated reports, email them, open bug tickets when test cases fail.
The figure above shows an example of inputs that can be manipulated by automation scripts as well as outputs that can be easily validated.
A properly designed test automation framework should allow developers or testers to effortlessly produce test cases with different inputs (simulating different scenarios) and validation rules around any artifact produced by the software component (log file, database, system state, etc.).
Let’s take a few examples to demonstrate the idea.
- Output: probably the first item to inspect. Typically easy to interpret and validate.
- Log files of a semi-structured nature (does not have to be rigid or constrained) for log messages would make it much easier to look for errors and abnormalities. System issues (memory, hardware, or database) are often logged in a file and provide an interesting information source.
- Database records: transactions running on a system may result in a change in database records, which could be queried for validation purposes. Transaction logs are great sources of information on system behaviour.
- System state: this is beneficial to inspect when actions or inputs to the system would change its internal conditions. Exposing the system’s internal state through APIs, for example, would make this type of information available for validation by test units.
- Integrations with third-party systems: it is not uncommon for a service to interact on a transaction basis with other systems in the ecosystem. This is generally done via a request/response paradigm. The response can take different shapes and forms (JSON, XML, character stream, etc.). A good design will find a way to expose these message pairs for validation.
5. Final Words
It is undeniable that automation (in testing and otherwise) is a mandatory requirement for companies to stay in the game; it is just not possible to keep ramping up manual efforts to maintain product quality on par with progress.
With the proliferation of methods, tools, and even philosophies on software delivery in general and testing and automation in particular, we hope that this article has helped you build a solid groundwork on which to formulate a good testing strategy.
We believe that overarching concepts of software delivery exist and that they serve as a fine comb to groom your thoughts and organize your priorities. We grouped these concepts under the term Operational Excellence in Software Development, and we believe that any implementation of those concepts must necessarily include an automation component.
We hope these ideas have helped you formulate your opinions on the best course for your organization.