Overview
Price experiments allow you to run A/B tests and use the results to confidently and accurately adjust your app's prices according to purchasing power in different markets globally. You can also use price experiments to test different price points against a control price in your core markets to drive more revenue. Setting the optimal price sets for your one-time products in different countries/regions ensures you're applying the best pricing strategy for your app to continue driving revenue upwards while still attracting new users. As this feature is based on one-time product pricing and sales, you must be a merchant developer to use it.
Prepare and run a price experiment
The sections below provide guidance and important information about preparing and running price experiments and understanding their results.
Before you start
Expand and read each of the sections below before you start setting up a price experiment in Play Console.
Prerequisites- You can only use active one-time products for experiments.
- Users must have the Manage store presence and View financial data permissions to use price experiments.
General information
- Experiments are at the app-level only; you cannot run experiments across multiple apps.
- The maximum length an experiment can run for is six months, after which prices revert to original price. Prices will also revert to the original price after 14 days of becoming statistically significant.
- You can test a maximum of 1000 purchase options in a single experiment.
- You cannot run the same experiment with overlapping one-time products and overlapping countries until at least 30 days after the end of the original experiment.
- You cannot pause experiments. You can only run them or stop them entirely. You cannot restart a stopped experiment.
Country/region limits
- You can only run one experiment at a time in any given country/region. For example, if you're already running an experiment in Norway, you can't run another experiment there on different products unless you stop the first experiment and create another.
- The tested price must be within the defined price ranges for in-app purchases. If a price percentage is outside the defined price range for that country/region as part of an experiment, the price percentage will automatically stop at the boundary value.
- You can have a maximum of two price variants and a control for each experiment.
- Experiments are only implemented in the selected country/region and not in its associated location overrides. For example, if you run an experiment in France, this excludes associated French territories (French Guiana, French Polynesia, Guadeloupe, Martinique, Mayotte, New Caledonia, Réunion, Saint Barthélemy, Saint Martin, Saint Pierre and Miquelon, Wallis and Futuna).
Changes to one-time products
- You cannot change the price of a one-time product featured in a running experiment. To change the price, you must stop the experiment. Note the following:
- The only exceptions are when you run two experiments and apply the results of the first experiment.
- If you configure a new one-time product or purchase option during an active experiment, it'll be omitted from the experiment.
Running experiments and applying results
- You can apply experiment prices at any time but we recommend waiting until your results are statistically significant and to only apply price changes that perform better than your control group.
- You cannot add additional countries/regions or products to an experiment which is currently running.
Experiments heavily feature the use of statistical terminology. If you are unfamiliar with these terms, you may find it helpful to refer to the glossary below.
| Term | Definition | Notes |
|---|---|---|
| Confidence interval |
A range of values expressing the uncertainty around whether the result is statistically significant or not. |
This is a key signal with which to interpret the experiment outcome. Statistical significance is determined when the confidence interval doesn't intersect with 0. This may not show up for the first few days due to low data volumes. |
| Confidence level |
The probability for which the observed difference between variant and control is true. Example: A confidence level of 90% would mean that 10% of the time the observed difference is likely due to chance. In other words, there is a less than 10% chance that the data being tested could have occurred if there’s no difference between control and variant. |
|
| Control | The original testing variable (containing the originally priced in-app purchase) in an experiment arm. | The control price is seen by a portion of the experiment audience defined during setup, and all users outside of the experiment. |
| Experiment outcome | The result of your experiment. | Possible experiment outcomes are listed in the following section. |
| False positive | When a negative event is incorrectly categorized as positive. |
When we detect that the spend difference between the control(s) and test is statistically significant (positive) but the spend difference between the control(s) and test is actually not statistically significant (negative), it's considered a false positive. The false positive rate is the ratio of the number of false positives and the total number of actual negative events. |
| One-time products |
Items for which users are charged on a one-time basis. One-time products can include items like virtual goods (for example, game levels or potions) and premium services within your app on Google Play.
|
Experimenting with a price change across many products may decrease the potential of cannibalization. |
| Purchase options |
A purchase option defines how the entitlement is granted to the user, its price, and where (region) the product will be available. A single one-time product can have multiple purchase options and a purchase option can have different prices in different regions. |
You can choose a maximum of 1000 purchase options. In general, more data means less time to detect a difference between the control and variant(s), if there is a difference. The price experiments are done at the purchase option level and not at the product level. For more information, see the object hierarchy in the one-time product object model. |
| Minimum detectable effect (MDE) |
An input you choose when setting up the experiment that indicates the level of improvement you want your experiment to detect. You can think of this as deciding how sensitive you want your experiment to be, thus affecting the estimated time until statistical significance. Once you choose your MDE, you'll see an estimate of the time it'll take to determine a statistically significant result based on historical data of your selection. For example, a smaller MDE means a more sensitive experiment which leads to a longer estimated time to statistical significance calculation. |
|
| Novelty effect | A phenomenon in which, users may prefer or try out a new feature (such as a new product or price point), even if the feature is not better or more appealing than the original. In this case, the uplift usually phases out over time. | Experiments that conclude quickly may be subject to novelty effects. We recommend using the estimated weeks until statistical significance calculations to consider this effect. |
| Statistical significance | In an A/B test, this is the determination of whether or not the difference between the control and variant is real or due to chance. |
If the confidence interval doesn't include the value of zero in the effect, we consider this a statistically significant result. In our case, the effect is the difference between control and variant spend. You can view some visual examples of confidence intervals in the Frequently asked questions section. |
| Variant | A variation of the original testing variable for an experiment arm. | The variant is either a price increase or a price decrease on the originally priced in-app purchase. |
Set up an experiment
When setting up an experiment, we recommend you include all interchangeable one-time product with all purchase options in your app (such as in-game currency) in the experiment to limit cannibalization.
If you're working with low data volumes, you likely won’t have enough data to draw a conclusion from the experiment. For example, if you're running an experiment with a new product, our estimated experiment length and warnings may not be accurate. In this case, you may see a warning in Play Console that a very low data volume is predicted within your chosen time frame. If you get a low data volume message, you may not have enough data to run an experiment or you may be able to fine-tune your experiment parameters. Here are some suggested actions to troubleshoot low data volume issues.
Conversely, if you're running a longer experiment, it’s important to consider that results may be impacted by economic factors like global currency fluctuations over time.
Part 1: Add details
To add your experiment details, do the following:
- Open Play Console and go to the Price experiments page (Monetize with Play > Price experiments).
- Click Create experiment.
- In the "Add details" section, enter an experiment name and a short description of your experiment (optional).
- Note: Your experiment name and short description aren't seen by users.
- Select the countries/regions you want to experiment with.
- Important: Price experiments are only implemented in the selected country/region and not in its associated locations. For example, if you select France, your price experiments will only be implemented in France and not in its associated territories like French Guiana.
- Click + Products. This opens the Select purchase options dialog. The Active purchase options section lists the available one-time product names with their IDs, purchase options and their IDs and the number of orders for that product in the last 30 days. Select the purchase options of the products that you want to experiment with and then click Apply.
- Select a start date. Your experiment will begin at 00:00 PT on the date you select. You can schedule your experiment to launch in the future.
- Click Next to continue setting up your experiment by adding variants.
Based on your experiment setup input, if we predict your data volumes will be too low to reach statistical significance, you'll see a warning. In this case, we recommend making adjustments to ensure you gather enough data to reach statistical significance. To view possible next steps, see Recommended actions for low data volumes.
Recommended actions for low data volumesThe following recommendations will help to change the data volumes required to reach statistical significance so that the experiment duration is shortened.
|
Experiment setup step |
Recommended action |
|---|---|
| Part 1: Add details | Increase the number of countries/regions in the experiment. |
| Part 1: Add details | Test across all one-time products or increase the number of one-time products. |
| Part 2: Add variants | Decrease the number of variants from two to one. |
| Part 2: Add variants |
Experiment with decreasing your price by a larger percentage (lower price). Note: This will not affect the initial estimation but will help during the live experiment. |
| Part 3: Manage settings | Increase audience size per variant (for example, 50% control and 50% treatment). |
| Part 3: Manage settings |
Increase the minimum detectable effect. Note: This only affects experiments in which the variant(s) and the control performed the same. |
| Part 3: Manage settings | Decrease the confidence level. |
Part 2: Add variants
In the "Add variants" section, you can add and remove price variants. Your price experiment must have at least one variant. To add variants:
- Using the drop-down, choose Increase price or Decrease price, and enter the percentage you want to apply to your choice. Note the following:
- You must enter a whole number.
- If you select Decrease price, you must enter a value between 1 and 99.
- If you select Increase price, you must enter a value between 1 and 999.
- After adding a variant, your control price range and variant price range are listed on the page. You can click View product prices for more detail, including product names, purchase options and their respective IDs, location overrides (if applicable), and tax information. Here you can see the actual percentage price changes after using the exchange rate and country-specific pricing patterns.
- Click + Add another variant and repeat the previous steps if you want to include multiple variants in your experiment. The control price range and variant price range are also listed on the page.
- Click Next to continue setting up your experiment by fine-tuning your experiment settings.
Part 3: Manage settings
In the "Manage settings" section, you can set your targeting parameters to fine-tune your experiment. To manage your experiment settings:
- Enter your experiment audience. This is the percentage of users who'll be in the experiment. These users will be split equally across your variant(s) and control. Note that users outside of the experiment will also see the original (control) price but won't be included in the experiment analysis.
- Note: You must enter a whole number between 1 and 100.
- Enter your confidence level. Decreasing the confidence level will increase the likelihood of a false positive, but it also will decrease the experiment's runtime.
- Enter a minimum detectable effect. This indicates the level of improvement you want your experiment to detect. You can think of this as deciding how sensitive you want your experiment to be. This will adjust your estimated time until statistical significance calculation.
- Refine your targeting parameters to fine-tune your experiment. You can see the estimated time until statistical significance calculation as a guideline based on your setup.
Part 4: Launch your experiment
You are now ready to launch your experiment. During the experiment, prices will change for selected purchase options and countries/regions. You can end your experiment at any time.
- Before launching your experiment, confirm that you understand the following:
- The experiment may affect your app’s revenue. Note that without a statistically significant result, any negative or positive revenue result may not be meaningful, so we recommend waiting until you reach statistical significance.
- You won’t be able to edit prices for the selected purchase options while the experiment is live.
- Applying the results of an experiment will affect all included countries/regions and purchase options.
- Click Confirm and launch.
Optional: End your experiment
You can end your experiment at any time.
- Open Play Console and go to the Price experiments page (Monetize with Play > Price experiments).
- Open the experiment you want to end.
- Click End experiment.
After you end your experiment, experiment prices will revert to the original price for the relevant countries/regions and products. The experiment and analysis will no longer be active.
If you have scheduled a future experiment, you can also cancel that by following the previous steps. Experiments that are canceled within 24 hours of their scheduled start date occasionally go live temporarily.
View experiment results and apply variants
It's important that you're familiar with different possible experiment outcomes and what they may mean for your app and its pricing strategy. Read this section carefully before applying your results.
View and analyze your experiment results
When your results are considered to be statistically significant, they'll be made available on your experiment's analysis page and you'll be notified via an Inbox message.
View your result
Visit your experiment to view the result and analysis. Go to the Price experiments page (Monetize with Play > Price experiments) and click the right arrow next to your experiment to view the experiment analysis page.
The outcome is displayed near the top of the page. Expand the Possible experiment outcomes section to view the different outcomes and what they mean. Underneath the experiment outcome, you can view a short description explaining the result. For example, you might see, "Variant [X] performed the best" above a short explanation of the experiment's result, which might say something like " Variant [X] achieved the greatest revenue." In cases where a variant has matched or outperformed a control, you can click Apply variant [X] to apply the variant and update your purchase options pricing accordingly. To learn more, see Apply variants.
Note: Experiments end automatically 14 days after a result is determined, or you can click End experiment to end it immediately.
View supporting data
Below the outcome, a table shows the data that the statistical result is based on. The table lists the variant(s), revenue, revenue versus control (and revenue versus control as a percentage), and the confidence interval for your variant(s). If you want to understand the different metrics on your experiment's analysis page, expand the Metric definitions section.
Below the table in the "Supporting data" section, you can view more granular data. The chart displays revenue versus control information to the latest available date as default. In this view, the shaded area represents the confidence interval; the chart allows you to view how the confidence interval changed through time. You can adjust the selected duration and dates using the duration and date filters at the top right of the chart.
You can also display the following metrics using the metric filter at the top left of the chart: Revenue, Orders, Buyers, Buyer ratio, ARRPU (these metrics are also described in Metric definitions). If you choose another metric, the chart won't display a confidence interval, as this is based on revenue.
To view more product-level details, you can also export your experiment results as a CSV file. You can view the fields, format, and examples for the CSV export by expanding the Metric definitions (CSV export) section.
Apply variants
Before applying your variant, note that you can only apply the variant to all purchase options and countries/regions in your experiment. We calculate statistical significance across the whole setup and therefore cannot guarantee the same outcome and effect if only part of the setup countries/regions and one-time products were applied.
If your variant(s) matched or outperformed your control, you can update your pricing by applying the variant:
- Open Play Console and go to the Price experiments page (Monetize with Play> Price experiments).
- Click Apply variant next to the variant you want to apply.
Your app's pricing will be updated as configured in the variant. Your experiment will end automatically 14 days after the result was determined, or you can click End experiment to end it immediately.
Possible experiment outcomesThere are multiple possible outcomes for an experiment. Your experiment outcome determines what your next steps should be.
| Experiment outcome | Meaning | Notes/recommendations |
|---|---|---|
| Variant [X] performed best | Variant [X] has become statistically significant by producing a meaningful result with the largest increase in revenue. | Apply variant [X] , as it significantly outperformed your control and the other variant(s). |
| Both variants performed better than the control | Both variants achieved greater revenue than the control. Review the results and decide which variant to apply. | You decide which variant to apply. |
| Variant(s) and the control performed the same | No variant achieved greater revenue than the control. | Your experiment has collected enough data and yields a draw between variant(s) and the control. |
| The control performed best | The control achieved greater revenue than both variants. | This suggests that your current price points for the experiment countries and product are already optimized. Continue to use the control price for the experiment products and countries. |
| More data needed | Your experiment is in progress. More data is needed to determine a statistically significant result. | Go to recommended actions for low data volumes to determine next steps. |
| Inconclusive result | Your experiment has been stopped prematurely or has reached the maximum experiment duration of six months. More data is needed to determine a statistically significant outcome. Try running a new experiment with a different setup. | Go to recommended actions for low data volumes to determine next steps. This may be an indication that buyers are indifferent about the one-time product prices in the experiment countries/regions. |
This table lists the metrics from your experiment's analysis page.
| Metric | Definition |
|---|---|
| All products revenue | The gross revenue from experiment users who purchased one-time products in and outside the experiment over the specified time period. |
| Average revenue per paying user (ARPPU) | The total gross revenue of one-time products divided by the number of unique buyers (users who make at least one purchase of one-time products from in and outside of the experiment) in the specified time period for the experiment. This metric will help you understand the value of buyers to your business. |
| Buyer ratio (28-day) |
The percentage of monthly active users who made at least one one-time product purchase during the experiment, including one-time products from outside of the experiment. Note: This is a key metric for understanding buyer conversion and increasing the breadth of your paying user base. |
| Buyers | Unique users who purchase at least one one-time product at the experimented price over the specified time period. |
| New installer revenue | Gross revenue from users who installed the app for the first time on any device after the experiment start date and saw the experimented price for the first time. Note that some users opt out of sharing this data with Google. |
| Orders | The number of experimented one-time product purchases made during the experiment over the specified time period. |
| Revenue | Gross revenue from experiment users who purchased one-time products in the experiment over the specified time period. |
This table lists the fields, format, and examples for metric definitions exported as a CSV file.
|
Field |
Format | Examples and notes |
|---|---|---|
| Date | String |
Date of the orders based on the Pacific Time Zone (in MMM DD, YYYY format). |
| SKU ID | String |
Developer-specified unique ID of the one-time product. |
| Product Title | String |
Developer-specified name of the one-time product. |
| Country | String |
Unique country code for relevant one-time product metrics. |
| Experiment Arm | String |
Experiment arm for relevant one-time product metrics. |
| Developer Currency |
String |
Currency for which the orders were converted. This is the local currency you're paid in. |
| Revenue | Numeric |
Total gross revenue for the specified one-time product (given the date, country and experiment arm). |
| New Installer Revenue | Numeric |
Gross revenue from users who installed the app for the first time on any device after the experiment start date and saw the price for the first time (given the specified one-time product, given the date, country and experiment arm). Note that some users opt out of sharing this data with Google. |
| Orders | Numeric |
Total orders for the specified one-time product (given the date, country, and experiment arm). |
| Buyers | Numeric |
Total buyers for the specified product (given the date, country and experiment arm). Buyers are users who purchased the specific one-time product during the experiment. |
Frequently asked questions
What countries/regions can I run experiments in?Countries/regions are eligible for price experiments only if they meet the following criteria:
- Countries/regions you've already rolled out a release in (including on internal test tracks).
- Countries/regions that support local currency.
If data volumes are very low, no data is available. Additionally, some data displayed on your experiment's analysis page may have a delay of up to seven days.
Can I test prices that are much lower or higher than the current set limits per country?
No; experiment prices can only be tested within the minimum and maximum price ranges defined per country/region. When entering a price change during the experiment setup, if the price change is at either limit, it'll automatically assume the minimum or maximum price. To review our list of price ranges and currencies allowed by country, see Supported locations for distribution to Google Play users.
No; if you're integrated with any version of Google Play's billing system, you can run price experiments.
All experiment orders are visible in your downloadable monthly financial reports (estimated sales report and earnings report). You can see which experimented one-time products are bought at the control and variant prices for the countries/regions you're running your price experiment in.
We recommend that you don't run any concurrent A/B experiments to ensure meaningful results. For example, Firebase works independently from price experiments in Play Console and any interference with specific one-time products may affect the reliability of experiment results.
We use jackknife (a statistical tool that allows an estimation of the variance through resampling) to calculate confidence intervals and then apply mixture sequential probability testing to control an inflated false positive rate from continuous monitoring. Statistical significance is determined when the confidence interval doesn't intersect with zero.
You must have the Manage store presence and View financial data permissions to use price experiments.
The experiment stops automatically 14 days after it reaches statistical significance or at the maximum experiment length of six months. We don't stop the experiment at the estimated duration given at the experiment setup; instead, we stop it after a statistically significant difference has been detected, with 14 days given to apply the new pricing before prices revert back to the original.
No, this isn't possible. A user will only see one price for the duration of the experiment.
No, this isn't possible because the experiment has already started collecting results with the parameters you initially selected. You can either run a second experiment in parallel with the additional countries/regions selected, or you can end the current experiment and set up a new experiment. Note that if there are overlapping purchase options and countries/regions, you'll need to wait 30 days before you can set up a new experiment.
Statistical significance is calculated across all countries/regions combined with experimented products as opposed to independently for each combination. If you’re interested in the price adjustment in a particular country/region, we recommend only selecting that country/region in the experiment setup.
Yes, you can apply a variant even after prices have reverted back to the original.
If your experiment's purchase options don’t generate revenue that’s significantly different to your control or there isn’t enough data, the result is considered to be inconclusive. For this outcome, it's possible that buyers are indifferent to the two prices.
In general, a larger sample size gives greater power. If the single one-time product you plan to test is one of the most popular products with many buyers, then testing that one-time product alone may be worthwhile. Otherwise, we encourage testing multiple one-time products together to increase the power of your experiment.
If you have multiple one-time products that are interchangeable to some degree (for example, USD 0.99 for 60 in-app currency units and USD 4.99 for 300 in-app currency units), then we encourage you to test both products together to avoid cannibalization.
We generally recommend one-variant experiments with your originally-priced one-time product(s) as the control and your discounted one-time product(s) as the variant. A two-variant experiment has your originally-priced one-time product(s) as the control and the same one-time product(s) at two different price points. Results from such experiments can be difficult to interpret, and two-variant experiments take more time than single-variant experiments to reach statistical significance.
If the confidence interval doesn't include the value of zero in the effect, we consider this a statistically significant result. In our case, the effect is the difference between control and variant spend.
Here's an example of how your confidence interval might look with a conclusive positive result:
Here's an example of how your confidence interval might look with a conclusive negative result:
Here's an example of how your confidence interval might look with an inconclusive result that needs more data:
Yes, we continuously monitor user location spoofing and take measures to mitigate the impact on our developers and their products.
Related content
- Learn more about price experiments and related features.
- Complete the price experiments course on Google Play Academy.