There is no doubt about the importance of automated frameworks in the Agile environment and as part of the day-to-day testing process. These are some insights to guide any automation project.
Write Clear and Efficient Code
Just like any other development project, you must write clear and efficient code that you can run in a reasonable time. The number of tests run even on the smallest automation project is often hundreds and sometimes thousands that need to run over and over again. Therefore, if the code is not efficient it will affect the timelines and reduce the number of executions.
Uncertainty Is Automation’s Worst Enemy
One of the main criteria to determine the quality of a good automation project is the ability to provide consistent results every time we run it. If automation passes successfully in the first run and fails in the second run without any changes made, we cannot be certain if the failure is due to a real product bug or a bug in the automation itself.
How can we remove uncertainty in an automated environment?
Remove unstable tests.
Do not automate unstable functionality.
Make sure that each test is verified manually.
Make sure that your code is up to date.
Make sure that you change the test data when requirements change.
Make sure that you run on stable environments.
Make sure that you use reliable builds of the application.
Make sure that each test case and code implementation is reviewed.
Make sure that new code is approved before releasing it to the main branch.
Determining Failure Gates
The automation code is designed to find bugs in the tested application. For that, we must make sure that in the case of a failure, it is easy to understand the root cause of the failed test without spending a lot of time diagnosing the failure.
Therefore, when writing automation, we must think about different ways to reduce the debugging time, such as automated bug reporting, logs and any other information that improves the debugging process.
Make Sure the Code Is Maintainable
After the first implementation of the automation code, validate that the code is always updated to support new features added, modified or removed from the application.
It is crucial that the code is easy to understand and is written so that it allows others to understand why it was written that way. Otherwise, a lot of time is wasted in investigating the reasons and logic used by the original coder.
After every test execution, generate a report that explains the test results. These guidelines help keep it efficient:
In case of failure, provide a link to the debugging information collected during the execution.
The report is sent automatically to the relevant people.
The report contains a link to the labs used to run the tests.
The report should be clear to the reader.
The report contains details about the tests that failed and passed.
The report shows who the automation owner is.
The report shows the execution start and end times.
Know Which Test Cases to Automate
There are endless test cases that we can run during a testing cycle, but in reality, we just cannot afford to write and execute them. Just like in manual testing, we need to determine which test cases should be automated based on a few simple factors:
Automate the test cases that increase the automation ROI.
Automate any test case that requires high precision.
Automate any test that is part of the regression cycle.
Automate test cases that you cannot perform manually.
Automate any test case that is executed repeatedly.
Automate any test case that is time-consuming.
Automate any test case that runs per build.
Automate high-risk scenarios.
Test cases that you should not automate:
Do not automate test cases for which the requirements are still changing. The maintainability of such cases will reduce the automation ROI.
Do not automate test cases that are part of usability testing.
Do not automate the application GUI if its objects are frequently modified.
Do not automate test cases that are not verified manually.
Do not automate ad-hoc random tests.
Know the Answer to “Why Are We Automating?“
Without knowing the answer to the question above, do not start any automation project. It is key to understanding the drive behind the decision to automate and in addition, helps reduce the expectation gap between the team and management.
Automation Will Not Fix All Problems
The primary reason to implement an automation process is to free up manual testers’ time, so they can design more complex cases, increase their knowledge in the application being tested and perform exploratory testing which helps increase confidence in the product.
Automation does not fix all problems. Do not expect automation to reveal all bugs and issues. The number of bugs found by automation is always lower than the number of bugs found by exploratory and manual testing.
Automation will not help you if you have chaos in the current testing effort.
Automated testing consumes a lot of time and resources.
Automation will not help you if you are already behind schedule.
Automated testing does not replace manual testing.
Automated testing does not eliminate the basic testing process of test design, planning and writing.
Automate Only What Is Necessary
When designing an automation project, always consider the time and investment factors. Adding an unnecessary test case affects the ROI of the entire project, so automation should cover only the necessary test cases. There is no benefit in writing tests that will not contribute to the test coverage.
In addition, before determining the test cases to automate, ask yourself a few basic questions:
What maintenance will be needed to adjust this test in future versions?
What is the preparation required in creating the test?
Can the test case be used in future versions?
Are the test’s expected results defined?
Do you have the test data for this test?
What is the objective of the test case?
Do you have the pass/fail criteria?
When a test fails, we want to find the root cause of the problem as fast as we can. The last thing we want is to spend time investigating numerous automation steps that affect the failed test. Independent tests allow you to understand the cause of the failure faster and remove the dependency on other automation steps. Another advantage of independent tests is the ability to run each test separately as a single unit without any relation to the other tests.
You Must Be Familiar with the Application Under Test
Automated testing should be treated in the same way as manual testing, the people responsible for building and designing the test scenarios must be familiar with the application being tested:
Know the technologies used to develop the software.
Know the main flows of the end-to-end scenarios.
Know the architecture of the application.
Know which platforms are supported.
Planning Before Implementation
Just like any other project, we cannot write any code before we know the basic parameters:
Determine the testing tools and programming language to use.
Understand the deliverables of the automation project.
Select the appropriate automation framework.
Understand the expectations.
Prepare the automation design.
Expected test coverage.
Build the Right Automation Team
Selecting the right automation tool is important, but it is more important to hire the right people to use it during the development phase:
Hire an automation architect to design the automation architecture.
Manual testers should focus on manual testing. If you want them to automate, make sure that they are dedicated only to the automated testing work.
The team must include at least one authority who can lead the technical aspects and guide the other automation engineers.
Manual testers are not programmers. Do not ask them to program without appropriate training.
Make sure that your team has enough people to meet the project requirements.
Manual Verification Before Automation
Before automating your tests, you must verify that each test case is tested and verified manually. If you decide to skip this phase, you will have trouble understanding the source of failures during the automated process.
Another consequence of skipping the manual verification is that it takes more time to stop the coder’s work and start the investigation about the failure. If you do the manual verification, you reduce the investigation time, which allows the coder to keep their focus.
Choose an Automation Tool Which Is Familiar to Your Team Members
You can spend thousands of dollars on the best automation tool, but if your team is not familiar with it they will not get the most out of it and will probably spend more time in learning it than on actual development.
When selecting the automation tool, follow a few basic guidelines:
If your team members have previous experience with an automation tool that meets the project requirements, select it. Their experience will help you meet targets faster.
When selecting the tool, ask yourself if you can use it in future projects.
Select the tool that is most suitable to meet the project requirements.
Select the automation tool that allows your team to develop in a programming language that they are familiar with.