Data is updated regularly in your mediation group page. At any point during the 90 days, you can make a decision based on the data and commit to a variant. Committing to a variant means that you’re choosing to use one of the variants as your mediation group configuration for 100% of your users.
To access and monitor your A/B test:
- Sign in to your AdMob account at https://admob.google.com.
- Click Mediation in the sidebar.
- Select the mediation group with the A/B test you want to view.
To help you analyze the results of your A/B test, AdMob reports the scaled monthly earnings of Variant A and Variant B. Scaled monthly earnings is an estimate of how much you’ll earn in 28 days if you were to apply this variant to 100% of your users. This provides a fair comparison and a more accurate picture of how each variant is doing.
Check the scaled monthly earnings difference at the top of the A/B test page to understand the magnitude of difference between the variants.
Charts illustrate how each ad source performs in each variant so that you can easily compare the performance of an ad source across both variants. These charts make it clearer and easier for you to interpret the experiment results. The Summary chart shows you an overview of the key metrics. You can hover over a metric in the Summary chart to determine how each variant is performing for that key metric.
You’ll find the scaled revenue, eCPM, match rate, and scaled impressions of Variant A and Variant B, which will be updated throughout the test. You can click a tab for each metric to review a line chart in greater detail. Each line chart shows daily data and plotted Variant B as a percentage change compared to Variant A.
Understand the results
Once a test begins, AdMob compares the scaled monthly earnings in both variants to determine which variant will perform best. AdMob shows the leading variant in the A/B test page once it has determined that a variant has performed consistently and substantially better than the other variant. We’ll also provide a recommended action: “Keep original” (if variant A is performing better) or “Apply variant” (if variant B is performing better).
To make adjustments to a test while it’s active, click Edit test. This will end the current test and create a draft of a new test with the same settings that you can edit. To ensure accurate results and recommendations, tests cannot be edited in real time. Remember it takes time to achieve results, so avoid prematurely ending your test based on initial results.
AdMob determines the leading variant and the estimated percent chance using the following criteria:
- How consistently the leading variant performs better than the other.
- The magnitude of the difference between both variants.
- The volume of data available. We recommend choosing an ad unit with substantial traffic to increase the amount of data available during the test. Note that the test must receive at least 10,000 ad requests in order for AdMob to determine the results.
A higher consistency and a larger difference in performance, as well as a high volume of data, helps AdMob increase the estimated percent chance.
If any of these criteria aren’t met, AdMob will not be able to determine a leading variant. In this case, you’ll have no conclusive results in the A/B test page. However, you can still view the metrics from the test for each variant.
About setting user allocation for Variant B
When the user allocation is set to less than 50%, the data for Variant B is a scaled calculation. This means the data is a fair comparison because AdMob calculates the data for Variant B as if it were receiving an equal user split.
For example, if the user allocation for Variant B is set to 10%, the estimated earnings will be scaled by 9. This allows you to do a fair comparison with Variant A (which has a user allocation of 90%). So if Variant B earns $10, the scaled calculation of Variant B’s earnings is $90.
Take action
We recommend letting your test run for at least two weeks before taking action. You can commit to a variant at any time during the 90 days. Committing to a variant means that you’re choosing to use one of the variants as your mediation group configuration for 100% of your users. Consider the difference in estimated earnings and the percent estimated chance when you decide to commit to a variant.
You can choose to keep Variant A or apply Variant B:
- To select Variant A, click Keep Variant A. This means your original mediation group configuration is retained.
- To select Variant B, click Apply Variant B. This means that we apply the settings of the variation to your account.
If you don’t commit to a variant by the end of the 90 days, the test stops and Variant A is used on 100% of your users, even if it wasn’t the leader of the test.
You have 30 days after the experiment ends to choose a variant before the test expires and Variant B is deleted.