A\B Testing Overview | David Tzemach
Updated: Jan 25, 2022
A/B testing is a test technique that is quite common in the online industry. It is used to test different product versions to see which one delivers the best results for consumers. When using A/B tests, the team compares two versions of the product and tests how much a particular variable affects the product in different aspects like usability, user experience, and performance.
How is it used?
Let’s say that the organization needs to initiate a marketing campaign on the company website and still doesn't know how different variables such as the font used or the size of the title influence customer behaviour. In the past, the organization had to invest in a deep analysis of these aspects, which consumed days and even weeks from staff.
With A/B testing, the organization can run two variations of the campaign with a controlled group of consumers that share their feedback regarding which of the two versions is the most successful. The organization can adjust the different variables to fine-tune the campaign until it has reached maximum effectiveness.
A/B Testing Process
The following is a basic A/B testing process you can use to start running tests:
Phase 1: Define the variable you want to test
The process starts with the selection of one or more variables to test. The key in testing these variables is to test them one by one to measure their performance. Otherwise, we cannot be sure which of those variables was responsible for the change in measurements.
This does not mean that you cannot test multiple elements simultaneously. If you decide to test multiple variables, you can use a different process, called multivariate testing, that uses the same core aspects of A/B testing but provides a comparing mechanism that reveals more information about how other variables interact and affect one another.
Phase 2: Identify goals
Goals are driven by business objectives, for example:
Increase traffic to the site.
Improve user experience and usability.
Clear any objects that are not contributing.
Evaluate the revenues earned in each variation.
Reduce the response time of the site.
Measure how much traffic from visitors is generated via a specific URL.
Phase 3: Definition of testing ideas and hypotheses
Once your goals are identified, it’s time to begin generating A/B testing hypotheses and ideas for changes that will be made to add more value than the current version. Once you have generated such a list, you must prioritize it based on its importance, expected impact on the customer, and business value.
Phase 4: Create the test variations
This phase focuses on creating the desired changes for one or more variables, such as changing object locations, colours or the order of elements on the page. In addition, the team needs to understand the change and whether there is any impact on the system to ensure that it works as expected before revealing it to customers.
Phase 5: Start measuring through experiments and campaigns
This is the time to kick off the project by using a limited experiment with focus groups or cross campaign with real customers. At this point in the process, users will access the site and interact with each variant, which is measured, monitored and compared to determine how each performs.
Phase 6: Result Analysis
Once the execution phase is complete, it’s time to analyze the execution results and how they fit the ideas and hypotheses determined in the early phases of the process. To conduct an effective analysis, the execution results should provide a clear view of the difference between the tested variables and whether that difference is significant.