Agile Testing Quadrants: Exploratory Testing (ET) | David Tzemach

Exploratory testing is probably one of the most efficient and effective testing methods for Agile teams to use. Exploratory testing differs from all other testing methods, as it requires a different state of mind.

In exploratory testing, the test design, learning, and actual testing are done in parallel. This is the opposite of all other test methods, which separate these activities into different phases, from learning to design and finally test execution.

In the industry today, where the Agile frameworks are more popular than ever before, there is simply less room for testers to work with comprehensive test documentation and weeks of planning. To stay relevant, testers must be able to meet this challenge and master lightweight test approaches, which include risk-based testing and especially exploratory testing.

Exploratory testing and Agile

Exploratory testing is one of the most important testing practices for Agile teams. In Agile software development, the team bases most of their test strategy on automated frameworks that minimize technical debt and automate the regression tests.

Using automated frameworks in an Agile environment is essential for obtaining the fast feedback that Agile teams need in order to succeed in their effort of delivering incremental releases frequently. However, the use of automated testing is not enough to ensure that the system meets all the requirements.

To guarantee that the system is built as desired by the customer, we must ensure that someone actually uses it, and adds the sort of testing that automated frameworks cannot give. This does not mean that we will need a comprehensive test document with detailed scripts for manual testing, as that does not make sense in an Agile environment.

Exploratory testing is the perfect solution to this, as it is not based on heavy testing documentation but on the critical thinking and creativity of the testers. The exploration process is done on the critical flows of the application and changed based on the information generated during the process. When using exploration sessions, the tester is free from the constraints of test scripts and can reveal unintended consequences of poor coding and design practices not considered in advance.

When should we use exploratory testing?

Exploratory testing can be a great platform for many testing challenges. It is especially helpful in the following scenarios:

When you want to move beyond the ordinary scripts

he usual testing process includes the execution of pre- written test scenarios. In some cases, we need to go further and use other paths without the constraints of tests that have been used repeatedly through the development process.

When the pressure is high

Exploratory testing is a lightweight method that allows Agile teams to work under pressure in a narrow time frame, and still to achieve efficient test coverage, mitigate risks and ensure that each delivery meets the quality standards.

When you want to get another perspective

Once the team has completed the scenario-based tests and you are still not convinced that all areas are covered, exploratory testing can be very handy.

Tests done by a non-technical person

Another possibility is to let a non-technical person (customer, Product Owner, project manager etc.) run an exploratory session on the system. By doing so, the team gains the end user’s perspective on the current usability of the product.

In fact, one of the most successful quality initiatives in my experience were the exploratory sessions executed by the Product Owners. As part of a sprint, the Product Owners took two hours to run a short exploratory session and ensure that the system is performing exactly as intended.

Besides the obvious technical benefits that such sessions add to the overall quality of the product, they increase the unity of the team as the Product Owners contribute to an area that is under the responsibility of the development team.

Feedback and estimations

How many times have you been asked by your manager to provide a time estimation on a new feature that you need to test? Well, using ET, the tester can start working with the app and get a sense of what it will take to test it or simply share feedback about its current state.

To manage bug activities

Exploratory testing is great when you need to handle different quality aspects related to bugs, for example:

  • Conducting further tests to verify a bug.

  • Looking for a specific reproduction scenario.

  • Finding critical bugs in the shortest time.

There are no or partial requirements

The team can run a short ET session on the application. Using their experience and knowledge they will most likely find major issues that are not correctly defined in the user stories. Sometimes, there is no other way besides diving into the water.

Exploratory testing should be controlled

Although ET provides solid ground for testers to use imagination and skill, the testing session should still be controlled and monitored. Here is a simple template:

Before starting the test session

Exploratory testing is not ad-hoc testing. It should always start with some planning. A good start is simply to ask your testers to provide information about the main testing flows that they want to explore, their ideas for testing and the main goals they want to achieve.

During the test process

Exploratory sessions should be limited in time. If it takes more than a few hours, a review should be done to understand its progress and how effective it really is.

Note that the review process itself should be quick so it does not distract the tester from the main goal. Here are some basic tips for conducting an effective review:

  • What type of bugs were found and how do they affect the session?

  • Are there any impediments to the tester’s ability to conduct the session?

  • What is the current progress of the tests?

  • Are there any findings that change the original goals?

  • What adjustments should be made to achieve the preliminary targets of the session?

At the End of the Session

Like any other testing process, a review must be done to understand the impact of the session. Here are some basic tips for conducting an effective review:

  • A general discussion of the quality of the session.

  • Review bugs found and analyze their impact on the product.

  • Determine the remaining risks and those which have been removed.

  • Compare the results with the preliminary goals.

How you can get more from exploratory testing

Here are some basic things that can generate more value from exploratory sessions:

Who should run it?

It depends on the goal you want to achieve during the session as both experienced and inexperienced testers can do it. If the goal is to learn a new feature, then it is relevant for both. However, when it comes to a testing effort where the complexity, challenges, and pressure are high, I always go with the testers that are familiar with the product, its technology and the testing risks it possesses.

Scripted exploratory is an option

Exploratory testing can be done without a preliminary test design, but it can also be executed using specifically created test scripts to be used as direction for the tester. This approach is best when we want more control of the execution, as we set boundaries and let the testers explore without losing focus.

Next steps based on findings

The main idea of exploratory testing is to explore. A good exploration process is never based only on the preliminary plan. It is based on the findings during the session. A tester should adapt their testing efforts based on what they find.

During the ET session, the tester uses different inputs that create both positive and negative behaviors, reflected in the outputs generated by the application. A great software tester can use this data to choose the next step in the execution flow in order to increase the efficiency of the tests.

Generate live documentation

The documentation process is conducted during the testing session. To make it more efficient, screen recorders can record the testing scenarios. Another option is to use a simple notepad for the tester to document the main testing flows and a general idea of the tests covered during the session.

Don’t ignore the environment

The exploratory session will never be efficient if it isn’t on an environment with the right complexity. A common mistake is using simple test environments to that don’t allow the tester to explore and test critical functionalities of the application.

Keep the testing artifacts

During the exploratory testing session, new test data (scripts, inputs etc.) are created and can be re-used in future tests. Make sure to keep this data to save time and effort in future sprints.