A/B Testing in Mobile Apps: How to Avoid Costly Mistakes
An A/B test often looks like a simple task: build two versions of a screen, split the audience in half, and see which one performs better. In practice, a single mistake in experiment design or analytics leads the team to make a product decision based on data that measured something other than what they intended.
The cost of that mistake isn’t just the team’s time. If a test falsely shows a winner, a variant that is no better than the current one, and sometimes worse, gets rolled out to 100% of your audience. The promised growth never shows up in revenue, and any conversion loss shows up there too, not in the experiment report. If the test turns out to be broken after launch, it has to be restarted: that’s several weeks lost, or even months for apps with low traffic.
This article isn't about any particular experimentation or analytics platform. We've collected lessons from A/B tests we ran in mobile applications across our projects: how to design an experiment, where the reliability of results is most often lost, and what to check before launch. It's written for founders, product owners and product leaders who make decisions based on test results. As engineers, our responsibility is not only to implement different variants but also to make sure that the experiment measures what the team thinks it measures.
Why A/B testing matters for your business
Let's imagine you're running an online business. You're selling products, operating in a competitive market, and there are plenty of other companies offering similar things. You have to compete for users' attention, keep them engaged, and ultimately convince them to buy from you.
In this environment, every percentage point matters. A small improvement in conversion rates or customer retention can have a significant impact on your revenue.
A simple example: 20,000 users start checkout every month, and the average order value is $60. A 4-percentage-point increase in conversion means 800 additional orders and around $48,000 in extra revenue per month. The same math works the other way: if the "winning" variant actually lowers conversion by 2 points, you lose 400 orders, or $24,000 every month.
Naturally, you want to understand how users interact with your application. Do they find it easy to use? Do they complete purchases? At what point do they drop off? Do they come back?
That's where analytics comes in. It gives you visibility into what's happening in your application: user behavior, key metrics, bottlenecks, and trends. But here's the thing: knowing what's happening doesn't necessarily tell you what to do about it.
Let's say you notice that the number of purchases has been steadily declining. You can see the numbers going down, but what is actually causing the decline? Maybe the checkout process is too complicated. Maybe users have trouble finding the products they're looking for. Maybe a particular UI change has made the experience less convenient. Or perhaps the problem has nothing to do with the interface at all. (We wrote more about finding the cause of a decline in Why Your Mobile App Is Underperforming and How to Diagnose the Cause Before Spending More.)
This is where A/B testing can help. Instead of rolling out a change to everyone and hoping for the best, we can experiment. We start with a hypothesis, implement the proposed change for a specific group of users, keep another group on the existing experience, and then analyze the results.
For the business, this means three things:
- product decisions are based on how users actually behave, not on opinions inside the team;
- a risky change is first seen by a limited part of the audience;
- only changes whose effect has been tested on real users make it to production.
What makes a good experiment?
Start with a hypothesis
Before starting implementation, we need a clear hypothesis. In other words, what do we believe will happen, and why?
Let's take the example we'll follow throughout this article. We want to reduce the number of checkout steps from four to two. Our hypothesis is that simplifying the process will reduce friction, make the purchasing journey faster and more enjoyable, and encourage more users to complete their purchases. Ultimately, we expect this to increase our conversion rate.
Before writing UI code, we define exactly what's changing, who can participate, and how success will be measured. The definition should be specific enough that another engineer can implement the same experiment without guessing. In our projects, this is a short document that the product owner and the team agree on before work starts. It saves weeks: arguing about what counts as success after launch is too late.
| Item | Example decision |
|---|---|
| Hypothesis | Combining address and shipping reduces checkout friction and increases completed purchases. |
| Unit of assignment | One stable customer account. |
| Eligible population | Signed-in customers with a cart containing an item eligible for delivery and an app version that supports the experiment. |
| Variants | four_step (Control) and two_step (Treatment). |
| Primary metric | Purchase conversion: users with complete_purchase within seven days after start_checkout, divided by users assigned at start_checkout. |
| Guardrails | Payment errors, checkout crashes, refund rate, support contacts, and checkout latency. |
| Exposure | The first time the assigned checkout variant actually becomes visible. |
| Test duration | Determined by the sample size calculation, at least two full weeks. |
| Conversion window | Seven days after start_checkout. Users assigned in the last seven days of the test are counted until their window closes. |
Guardrail metrics deserve a separate mention. A new checkout can raise conversion and at the same time increase payment errors or support contacts. Without guardrails, a "successful" test like that is easy to roll out to everyone.
Control and treatment groups
To evaluate an experiment, we need two groups: the control group keeps using the existing experience, and the treatment group receives the proposed change.
Let me explain this with an analogy I like to use. Imagine a hospital with 100 patients, and we want to test a new medicine. Only 40 patients give their consent to participate. We randomly divide them into two groups and give each patient a mark on their medical chart: yellow for the control group on the existing medicine, blue for the treatment group on the new one. The remaining 60 patients shouldn't have either of these marks.
The same principle applies to our application. Only users who actually belong to the experiment receive the variant attribute. Users outside the experiment should either have no such attribute or have a clearly distinct value, such as not_in_experiment. The important thing is to avoid mixing users who aren't participating in the experiment with our actual control group, otherwise, the comparison loses its meaning.
Keep eligibility, assignment, exposure, and outcome separate
These four concepts are easy to confuse:
- Eligibility: the user is allowed to participate.
- Assignment: the system selects a variant for the eligible user.
- Exposure: the user actually sees the tested experience.
- Outcome: the user completes or abandons the measured action.
For example, a user might be assigned to the treatment group but never navigate to checkout. They belong to the treatment group, but they haven't been exposed to the new checkout experience.
We should also distinguish users who couldn't be assigned because of a configuration failure from users successfully assigned to control.
When these concepts get mixed up, the effect of the change is either diluted or artificially inflated. Below we'll look at how we keep them apart in the implementation.
Experiment assignment: who gets which variant?
Random assignment and traffic share
Users should be assigned randomly to control and treatment groups to avoid selection bias. A common approach is a 50/50 distribution, which is generally the most statistically efficient for comparing two variants. If a change is risky, we can limit the share of participants: for example, we include only 10% of eligible users in the experiment and split them equally between Control and Treatment. The remaining 90% simply don't participate.
This limits the risk, but we pay for it with time, and that cost is easy to underestimate. Let's say 1,000 users start checkout every day. At 10% of traffic, each group gets 50 users a day:
| Effect to detect | Users needed per group | Duration at 10% of traffic | Duration at 50% of traffic |
|---|---|---|---|
| Conversion grows from 40% to 44% | ≈ 2,400 | ≈ 48 days | ≈ 10 days* |
| Conversion grows from 40% to 42% | ≈ 9,500 | ≈ 190 days | ≈ 38 days |
Calculated for a two-sided test at a 5% significance level and 80% power. * Even if the sample is collected faster, the test runs for at least two full weeks to account for weekly seasonality.
That's why we choose the traffic share together with the product owner, based on two questions: what minimum effect matters to the business, and how long the business is willing to wait for an answer.
Assignment should be stable
Once a user has been assigned to a variant, they should stay in that group throughout the experiment. Imagine a user gets assigned to Treatment today but receives Control tomorrow. This makes our results unreliable.
Most experimentation platforms handle variant assignment using a process called bucketing: they use a user identifier to consistently determine which group the user belongs to. However, we shouldn't just assume this works correctly. We verify that users stay in the same variant across sessions and that the variant doesn't change when they log in, log out, or switch accounts. If we're testing purchasing behavior, an account-level identifier is more appropriate than an installation identifier.
Analytics: designing measurement together with the experiment
The important thing is to design analytics together with the experiment, rather than adding tracking as an afterthought. If the events are set up incorrectly, there's no fixing it after launch: the data for the test period is lost, and the experiment has to be restarted.
Design events around the user journey, not screens
For our checkout experiment (four_step as Control, two_step as Treatment), we track six milestones:
| Event | When should it be triggered? |
|---|---|
start_checkout | The user enters the checkout flow. |
complete_address | The user successfully saves a valid address. |
select_shipping | The user successfully selects a shipping method. |
select_payment | The user successfully selects a payment method. |
submit_order | The user submits the order after validation. |
complete_purchase | The backend confirms that the purchase is completed. |
In addition to these funnel milestones, we record checkout_exposed when the assigned checkout variant becomes visible. This event is kept out of the purchase funnel calculation.
You can measure conversion using just two events, start_checkout and complete_purchase. However, from experience, I can say that when something goes wrong, you'll want to understand what happened in between. Separate submit_order and complete_purchase events show the difference between "users didn't like the new checkout" and "the new checkout has technical issues." And even though address and shipping are handled on the same screen in the treatment variant, we still track complete_address and select_shipping separately: this way, the variants can be compared regardless of how the UI is structured.
Screen-view events are useful for understanding navigation behavior and investigating UI-specific problems. However, we shouldn't use them as our primary conversion funnel: the two variants have different screen structures, and the comparison would be incorrect.
Define a simple analytics event contract
Before implementing tracking, we agree on which properties each event needs:
| Property | Purpose |
|---|---|
event_name | Identifies the user action. |
variant | Identifies the assigned experiment group (four_step, two_step, or not_assigned). |
experiment_id | Identifies the experiment and its version. |
user_id | Identifies the customer account. |
The important experiment-specific property is variant. Don't assume that the analytics SDK automatically knows your application's account ID: the user needs to be identified explicitly.
Track the outcome, not the button tap
In our projects, events aren't triggered by button taps but by the successful completion of a business operation. Tapping Continue doesn't necessarily mean that the address was successfully saved, so complete_address is emitted only after the address has been saved. And complete_purchase is triggered only after the backend confirms the purchase, not simply when the user taps the Pay button.
Separate assignment from exposure
We assign the variant immediately after the user taps Start checkout, before rendering the checkout UI, and attach it to the start_checkout event. This helps us avoid initially rendering the default variant and then switching to another one.
We track exposure separately, when the assigned checkout variant actually becomes visible. A screen on Android can be recreated, so the exposure event is sent exactly once per user and experiment.
What happens when assignment fails?
Experiment configuration can fail to load: the user is offline, the request times out, or the configuration is missing or invalid. These failures shouldn't break the application experience:
- Safe fallback. The user sees the existing checkout, but isn't automatically classified as Control: we mark them as
not_assignedand exclude them from the experiment comparison. Otherwise, the control group gets contaminated by users who had network problems, and the result is skewed. - Cached assignments. If the user was successfully assigned during a previous session, we reuse the cached assignment when offline.
- No UI flickering. The assignment is resolved before entering the tested flow, rather than switching a screen the user is already interacting with.
The main principle is that experiment infrastructure should never become a single point of failure for the application.
Mixed app versions
The minimum app version that contains the experiment code is checked during eligibility evaluation: users on older versions aren't assigned to either group.
How to read A/B test results
The funnel and the right denominator
We define the analysis rules in advance: count unique users rather than raw event occurrences, group them by their assigned variant, and require milestones to occur in the correct order.
Everyone assigned at start_checkout stays in the denominator, even if they never reach payment. Only users who complete a purchase within the conversion window go into the numerator.
| Metric | four_step (Control) | two_step (Treatment) |
|---|---|---|
Assigned users at start_checkout | 5,000 | 5,000 |
Users with complete_purchase | 2,000 | 2,200 |
| Conversion rate | 40% | 44% |
Difference: +4 percentage points. With 5,000 users per group, this difference is statistically significant (p < 0.001), so it can't be explained by chance. In money terms, using the example from the beginning of the article, that's about $48,000 in extra revenue per month.
Notice that the percentages use the original assigned population as their denominator, not the users who reached the previous funnel step, and not the users who saw the variant. With the wrong denominator, a test can show growth where there is none. We use the exposure event for diagnostics: if there are noticeably fewer exposures than assignments, that signals a technical problem.
Sample size and statistical significance
Let's say 4 out of 10 users completed a purchase in the control group and 6 out of 10 in the treatment group. Looks like a great improvement, right? But with such a small number of users, a few additional purchases could completely change the results. Statistical significance helps us understand whether the difference we're seeing is likely to be a real effect of our change or just a result of random variation. For a simple explanation, see the Scribbr guide "Statistical Significance: A Simple Introduction".
Peeking
Imagine we're running an experiment for two weeks, but after the first day, we see that the treatment group is performing better and decide to stop the experiment. The problem is that repeatedly checking results and stopping as soon as we see statistical significance can increase our chances of a false positive. That's why we define the experiment duration, analysis approach, and required sample size in advance.
Running overlapping experiments
Let's say we're testing a new product details page layout on the same user journey as the new checkout. Their effects might interact, and we may not be able to attribute the observed change to one experiment alone. In such cases, we separate overlapping experiment populations or design the analysis to account for interactions.
The most common A/B testing mistakes we see
| Mistake | What it leads to | What it means for the business |
|---|---|---|
| Users outside the experiment end up in the control group | The effect is diluted, the comparison is invalid. | The real effect isn't visible, and a good change gets rejected. |
| Assignment failures are counted as Control | The control group is skewed by users with technical problems. | The new variant looks better than it is. |
| Assignment isn't stable across sessions or accounts | Users see both variants, the results are invalid. | The test has to be restarted. |
| Events are triggered by button taps | The funnel shows intentions, not actual results. | Payment errors look like "user behavior." |
| Conversion is calculated from screen views | Variants with different screen structures can't be compared. | The conclusion depends on the layout, not on behavior. |
| Assignment is treated as exposure | The analysis includes users who never saw the change. | The effect is underestimated, and the decision gets delayed. |
| The test is stopped at the first good result | Higher risk of a false positive and a failed release. | A variant that doesn't actually work is rolled out to everyone. |
| Parallel tests on the same user journey | The effect can't be attributed to a specific change. | Resources go into a change that had no effect. |
| Older app versions aren't accounted for | The groups include users for whom the experiment doesn't technically work. | Part of the data is noise, and the conclusions are skewed. |
| Nobody checked that the groups actually split 50/50 (SRM) | An imbalance points to a technical error in assignment. | The decision is based on a technically broken test. |
Recognize your project? If you see at least two of these mistakes, the results of your past tests shouldn't be trusted by default. As part of a mobile app audit, we check events, assignment logic and discrepancies between analytics and backend data, and then show you which conclusions from past tests you can rely on and which need to be re-checked.
Pre-launch A/B test checklist
We go through these questions with the team before an experiment starts. They also help when choosing an experimentation platform:
Ready to launch?
How we run experiments in Gilzor projects
When developing mobile apps, we build analytics and experimentation infrastructure into the feature design stage and define the hypothesis and metrics together with the product owner. Assignment scenarios (logging in and out, working offline, older app versions, configuration failures) are tested as part of QA, before users see the experiment.
If you're only planning your first tests, we'll join at the design stage: we'll set up events, assignment and failure handling so that your very first experiment gives you a result you can trust.
If your tests are already running but you don't trust the results, or the numbers in your analytics don't match your backend data, start with a mobile app audit: we'll check your instrumentation, events and assignment logic.
Built by Gilzor
Results we’ve shipped
Talk to the people who build it. Tell us about your project and get a free estimate of scope, timeline and cost.
Key takeaway
Reliable experiment analytics starts with a well-designed event contract. Different platforms may offer different features and implementation approaches, but the key things remain the same: stable user assignment, consistent analytics, separate exposure tracking, and a correct calculation of the result.
The goal isn't to collect as many analytics events as possible. It's to make sure that, when the experiment ends, we can confidently answer the question we started with: Did simplifying checkout actually help more users complete their purchases?
No sales pitch
Want to check whether your tests measure what they should?
Send us the plan for your next experiment, or tell us where the numbers don’t add up.
Tatiana Bulavko, Android Developer at Gilzor.
Gilzor · Mobile App Audit partner
Need a team for your app?
Services
Mobile App AuditIndependent review of your app’s code, architecture and UX.→By company type
Selected projects
The team behind them
