Best Practices for Test Automation | David Tzemach
Updated: Jan 26, 2022
There is no doubt about the importance of automated frameworks in the Agile environment and as part of the day-to-day testing process. These are some insights to guide any automation project.
Write Clear and Efficient Code
Just like any other development project, you must write clear and efficient code that you can run in a reasonable time. The number of tests run even on a minor automation project is often hundreds and sometimes thousands that need to run over and over again. Therefore, if the code is not efficient, it will affect the timelines and reduce the number of executions.
Uncertainty Is Automation’s Worst Enemy
One of the main criteria to determine the quality of a good automation project is the ability to provide consistent results every time we run it. If automation passes successfully in the first run and fails in the second run without any changes made, we cannot be certain if the failure is due to an actual product bug or a bug in the automation itself.
How can we remove uncertainty in an automated environment?
Remove unstable tests.
Do not automate unstable functionality.
Make sure that each test is verified manually.
Make sure that your code is up to date.
Make sure that you change the test data when requirements change.
Make sure that you run in stable environments.
Make sure that you use reliable builds of the application.
Make sure that each test case and code implementation is reviewed.
Please ensure that the new code is approved before releasing it to the main branch.
Determining Failure Gates
The automation code is designed to find bugs in the tested application. For that, we must make sure that in the case of a failure, it is easy to understand the root cause of the failed test without spending a lot of time diagnosing the failure.
Therefore, when writing automation, we must think about ways to reduce the debugging time, such as automated bug reporting, logs, and any other information that improves the debugging process.
Make Sure the Code Is Maintainable
After the first implementation of the automation code, validate that the code is constantly updated to support new features added, modified, or removed from the application.
It is crucial that the code is easy to understand and is written to allow others to know why it was written that way. Otherwise, a lot of time is wasted in investigating the reasons and logic used by the original coder.
After every test execution, generate a report that explains the test results. These guidelines help keep it efficient:
In case of failure, provide a link to the debugging information collected during the execution.
The report is sent automatically to the relevant people.
The report contains a link to the labs used to run the tests.
The report should be clear to the reader.
The report contains details about the tests that failed and passed.
The report shows who the automation owner is.
The report shows the execution start and end times.
Know Which Test Cases to Automate
There are endless test cases that we can run during a testing cycle, but in reality, we cannot afford to write and execute them. Just like in manual testing, we need to determine which test cases should be automated based on a few simple factors:
Automate the test cases that increase the automation ROI.
Automate any test case that requires high precision.
Automate any test that is part of the regression cycle.
Automate test cases that you cannot perform manually.
Automate any test case that is executed repeatedly.
Automate any time-consuming test case.
Automate any test case that runs per build.
Automate high-risk scenarios.
Test cases that you should not automate:
Do not automate test cases for which the requirements are still changing. The maintainability of such cases will reduce the automation ROI.
Do not automate test cases that are part of usability testing.
Do not automate the application GUI if its objects are frequently modified.
Do not automate test cases that are not verified manually.
Do not automate ad-hoc random tests.
Know the Answer to “Why Are We Automating?“
Do not start any automation project without knowing the answer to the question above. It is key to understanding the drive behind the decision to automate and, in addition, helps reduce the expectation gap between the team and management.
Automation Will Not Fix All Problems
The primary reason to implement an automation process is to free up manual testers’ time to design more complex cases, increase their knowledge in the application being tested, and perform exploratory testing, which helps increase confidence in the product.
Automation does not fix all problems. Do not expect automation to reveal all bugs and issues. The number of bugs found by automation is always lower than those found by exploratory and manual testing.
Automation will not help you if you have chaos in the current testing effort.
Automated testing consumes a lot of time and resources.
Automation will not help you if you are already behind schedule.
Automated testing does not replace manual testing.
Automated testing does not eliminate the basic testing process of test design, planning, and writing.
Automate Only What Is Necessary
When designing an automation project, always consider the time and investment factors. Adding an unnecessary test case affects the ROI of the entire project, so automation should cover only the necessary test cases. There is no benefit in writing tests that will not contribute to the test coverage.
In addition, before determining the test cases to automate, ask yourself a few basic questions:
What maintenance will be needed to adjust this test in future versions?
What is the preparation required in creating the test?
Can the test case be used in future versions?
Are the test’s expected results defined?
Do you have the test data for this test?
What is the objective of the test case?
Do you have the pass/fail criteria?
When a test fails, we want to find the root cause of the problem as fast as possible. The last thing we want is to spend time investigating numerous automation steps that affect the failed test. Independent tests allow you to understand the cause of the failure faster and remove the dependency on other automation steps. Another advantage of independent tests is the ability to run each test separately as a single unit without any relation to the other tests.
You Must Be Familiar with the Application Under Test
Automated testing should be treated in the same way as manual testing; the people responsible for building and designing the test scenarios must be familiar with the application being tested:
Know the technologies used to develop the software.
Know the main flows of the end-to-end scenarios.
Know the architecture of the application.
Know which platforms are supported.
Planning Before Implementation
Just like any other project, we cannot write any code before we know the basic parameters:
Determine the testing tools and programming language to use.
Understand the deliverables of the automation project.
Select the appropriate automation framework.
Understand the expectations.
Prepare the automation design.
Expected test coverage.
Build the Right Automation Team
Selecting the right automation tool is essential, but it is more important to hire the right people to use it during the development phase:
Hire an automation architect to design the automation architecture.
Manual testers should focus on manual testing. If you want them to automate, make sure that they are dedicated only to the automated testing work.
The team must include at least one authority to lead the technical aspects and guide the other automation engineers.
Manual testers are not programmers. Please do not ask them to program without appropriate training.
Make sure that your team has enough people to meet the project requirements.
Manual Verification Before Automation
Before automating your tests, you must verify that each test case is tested and verified manually. If you decide to skip this phase, you will have trouble understanding the source of failures during the automated process.
Another consequence of skipping the manual verification is that it takes more time to stop the coder’s work and investigate the failure. Doing the manual verification reduces the investigation time, which allows the coder to keep their focus.
Choose an Automation Tool Which Is Familiar to Your Team Members
You can spend thousands of dollars on the best automation tool, but if your team is not familiar with it, they will not get the most out of it and will probably spend more time learning it than on actual development.
When selecting the automation tool, follow a few basic guidelines:
If your team members have previous experience with an automation tool that meets the project requirements, select it. Their expertise will help you meet targets faster.
When selecting the tool, ask yourself if you can use it in future projects.
Select the tool that is most suitable to meet the project requirements.
Select the automation tool that allows your team to develop a programming language that they are familiar with.