What shouldn't we automate in an agile environment? | David Tzemach
If you're familiar with my blog, you already know that I'm a great believer in automation frameworks that allow the team to create a more productive, efficient and reliable process of coding and testing. However, some aspects of the testing process still need human eyes, common sense, and intelligence.
Usability Testing
Usability testing is very different from other test types that determine the quality of the software. It cannot be automated because it requires someone to work with the software to determine how it was experienced and where the gaps are in the user experience.
GUI Testing (Is it worth the ROI?)
GUI testing is one of the most challenging areas to automate. I’ve seen just too many organizations that invested thousands of human hours in automating the GUI of their products but, in the end, found it was a waste of time that didn't provide the expected ROI. Some GUI tests can be used to ensure there are no unexpected changes in the GUI. Still, you should ask yourself whether it’s worth the costs and investment instead of improving other quality issues that will provide a better ROI and reduce the risks in that area.
Tests that are not worth the investment
I used to be involved in an automation project where the test team automated a thousand provided tests (at least on paper). So what is wrong here? They automated almost all the available test scenarios without really thinking about the ROI.
The team invested weeks in automating tests marked as low risk, tests that would never fail, and tests whose failure had a meager chance of impacting the software. The entire automation process was based on the spirit of "let's automate everything" instead of asking the simple question of the ROI that this automation project provides.
In some cases, some tests are written without real thought as to whether they are essential or not. Once the automation project starts, the team will automate these tests just because they never want to rerun them (because they know there is a 0% chance that it will make a difference).
Tests that need to be executed only once
The main goal of automated testing is to allow the team to focus on essential things in the software development lifecycle. Automating test scenarios that will run only once are not worth the team's time to invest in the design, creation, and execution of these tests.
Exploratory Testing
In my opinion, exploratory testing is the best and most efficient method that agile teams use in any testing process. Exploratory testing can be used for learning purposes (you learn more about the software when testing it) or to provide a fast way to evaluate the overall quality of the software. However, when it comes to real testing effort, a skilled tester must design and execute tests.
Exploratory testing should be done by humans and not by automated scripts because automated scripts will not let the tester take in new information he generated from the exploratory session and use it to improve future testing and development processes.
In addition, although exploratory testing is a great testing approach, there is a real need for automated tests that will allow the team to focus on their exploratory sessions without worrying about any regressions that automated tests should cover.