Exploratory Testing (ET) | David Tzemach
Updated: Jan 26, 2022
Exploratory testing is probably one of the most efficient and effective testing methods for Agile teams to use. Exploratory testing differs from all other testing methods, as it requires a different state of mind.
In exploratory testing, the test design, learning, and actual testing are parallel. This is the opposite of all other test methods, which separate these activities into different phases, from learning to design and test execution.
Today, where the Agile frameworks are more popular than ever before, testers have less room to work with comprehensive test documentation and weeks of planning. To stay relevant, testers must meet this challenge and master lightweight test approaches, including risk-based testing and exploratory testing.
Exploratory testing and Agile
Exploratory testing is one of the most crucial testing practices for Agile teams. In Agile software development, the team bases most of their test strategy on automated frameworks that minimize technical debt and automate the regression tests.
Using automated frameworks in an Agile environment is essential for obtaining fast feedback. Agile teams need to deliver incremental releases frequently. However, automated testing is not enough to ensure that the system meets all the requirements.
To guarantee that the system is built as desired by the customer, we must ensure that someone uses it and adds the sort of testing that automated frameworks cannot give. This does not mean we will need a comprehensive test document with detailed scripts for manual testing, as that does not make sense in an Agile environment.
Exploratory testing is the perfect solution to this, as it is not based on heavy testing documentation but the critical thinking and creativity of the testers. The exploration process is done on the vital flows of the application and changed based on the information generated during the process. When using exploration sessions, the tester is free from the constraints of test scripts and can reveal unintended consequences of poor coding and design practices not considered in advance.
When should we use exploratory testing?
Exploratory testing can be an excellent platform for many testing challenges. It is beneficial in the following scenarios:
When you want to move beyond the ordinary scripts
The usual testing process includes the execution of pre-written test scenarios. In some cases, we need to go further and use other paths without the constraints of tests that have been used repeatedly through the development process.
When the pressure is high
Exploratory testing is a lightweight method that allows Agile teams to work under pressure in a narrow time frame and still achieve efficient test coverage, mitigate risks and ensure that each delivery meets the quality standards.
When you want to get another perspective
Once the team has completed the scenario-based tests and is still not convinced that all areas are covered, exploratory testing can be the best method to get the actual status without assumptions.
A non-technical person does tests.
Another possibility is to let a non-technical person (customer, Product Owner, project manager etc.) run an exploratory session on the system. By doing so, the team gains the end user’s perspective on the current usability of the product.
One of the most successful quality initiatives in my experience was the product owners' exploratory sessions. As part of a sprint, the Product Owners took two hours to run a short exploratory session and ensure that the system performed exactly as intended.
Besides the obvious technical benefits that such sessions add to the overall quality of the product, they increase the team's unity as the Product Owners contribute to an area under the development team's responsibility.
Feedback and estimations
How many times has your manager asked you to provide a time estimation on a new feature that you need to test? Well, using ET, the tester can start working with the app and get a sense of what it will take to test it or share feedback about its current state.
To manage bug activities.
Exploratory testing is excellent when you need to handle different quality aspects related to bugs, for example:
Conducting further tests to verify a bug.
Looking for a specific reproduction scenario.
Finding critical bugs in the shortest time.
There are no or partial requirements.
The team can run a short ET session on the application. Using their experience and knowledge, they will most likely find significant issues not correctly defined in the user stories. Sometimes, there is no other way besides diving into the water.
Exploratory testing should be controlled.
Although ET provides solid ground for testers to use imagination and skill, the testing session should still be controlled and monitored. Here is a simple template:
Before starting the test session
Exploratory testing is not ad-hoc testing. It should always start with some planning. A good start is to ask your testers to provide information about the main testing flows that they want to explore, their ideas for testing and the main goals they want to achieve.
During the test process
Exploratory sessions should be limited in time. If it takes more than a few hours, a review should be done to understand its progress and how effective it is.
Note that the review process should be quick, not distract the tester from the primary goal. Here are some essential tips for conducting an adequate review:
What type of bugs were found, and how did they affect the session?
Are there any impediments to the tester’s ability to conduct the session?
What is the current progress of the tests?
Are there any findings that change the original goals?
What adjustments should be made to achieve the initial targets of the session?
At the End of the Session
Like any other testing process, a review must be done to understand the impact of the session. Here are some essential tips for conducting an adequate review:
A general discussion of the quality of the session.
Review bugs found and analyze their impact on the product.
Determine the remaining risks and those which have been removed.
Compare the results with the initial goals.
How you can get more from exploratory testing
Here are some basic things that can generate more value from exploratory sessions:
Who should run it?
It depends on the goal you want to achieve during the session, as both experienced and inexperienced testers can do it. If the goal is to learn a new feature, both are relevant. However, when it comes to a testing effort where the complexity, challenges, and pressure are high, I always go with the testers familiar with the product, its technology and the testing risks it possesses.
A scripted exploratory is an option.
Exploratory testing can be done without a preliminary test design, but it can also be executed using specifically created test scripts as direction for the tester. This approach is best when we want more control of the execution, as we set boundaries and let the testers explore without losing focus.
Next steps based on findings
The main idea of exploratory testing is to explore. A good exploration process is never based only on the preliminary plan. It is based on the findings during the session. A tester should adapt their testing efforts based on what they find.
During the ET session, the tester uses different inputs that create positive and negative behaviours, reflected in the outputs generated by the application. An excellent software tester can use this data to choose the next step in the execution flow to increase the efficiency of the tests.
Generate live documentation
The documentation process is conducted during the testing session. To make it more efficient, screen recorders can record the testing scenarios. Another option is to use a simple notepad to document the main testing flows and a general idea of the tests covered during the session.
Don’t ignore the environment.
The exploratory session will never be efficient if it isn’t in an environment with no complexity. A common mistake is using simple test environments that don’t allow the tester to explore and test critical functionalities of the application.
Keep the testing artifacts.
New test data (scripts, inputs, etc.) are created during the exploratory testing session and can be re-used in future tests. Keep this data to save time and effort in future sprints.