An A/B test compares two versions of something (a page, an email, a price display) by showing each to a random group of users and measuring which performs better. It is the standard way companies find out whether a change actually helps.
A running example
A fictional online grocery app, BasketBuddy, wants to know whether showing delivery time on the product page increases orders. Version A is the current page; version B adds the delivery time.
Step 1: Define the question and the metric
Decide before the test:
- Primary metric: the one number that decides the result, for example the share of visitors who place an order (conversion rate).
- Guardrail metrics: numbers that must not get worse, such as average order value or cancellations.
Choosing the metric after seeing results invites cherry-picking.
Step 2: Randomise
Each user is randomly assigned to A or B and stays in that group. Randomisation makes the groups comparable, so any difference in outcome can be attributed to the change rather than to who saw it.
Non-random splits (for example, Android users see B and iOS users see A) mix the effect of the change with differences between the groups.
Step 3: Decide the sample size
Small tests are noisy: random variation alone can make B look better or worse. Before starting, estimate how many users you need based on:
- The current conversion rate.
- The smallest improvement worth detecting.
- How much risk of false alarms and missed effects you accept.
Online calculators and statistics libraries do this calculation. The key idea: detecting small differences needs many more users than detecting large ones.
Step 4: Run the test for the planned duration
Run for full weeks if behaviour differs by weekday, and do not stop early because the result looks good on day two. Checking repeatedly and stopping at the first "significant" moment greatly increases false wins.
Step 5: Analyse
Compare the primary metric between groups and test whether the difference is larger than chance would explain. For conversion rates, a two-proportion test is common.
from statsmodels.stats.proportion import proportions_ztest
orders = [conv_a, conv_b] # number of converting users in A and B
visitors = [n_a, n_b] # number of users in A and B
stat, p_value = proportions_ztest(orders, visitors)Report three things:
- The difference in the metric (absolute and relative).
- The uncertainty, ideally as a confidence interval.
- The p-value, compared against the threshold you set beforehand.
Step 6: Decide
- Clear improvement and guardrails fine: ship B.
- No clear difference: the change probably has little effect at this scale; decide on other grounds (cost, simplicity).
- Worse: keep A, and you have learned something.
Common mistakes
- Peeking and stopping early.
- Testing many metrics and declaring victory on whichever one moved.
- Sample ratio mismatch: if you planned a 50/50 split but got a noticeably different ratio, something in the assignment is broken; investigate before trusting results.
- Novelty effects: users click a new design because it is new; the effect can fade.
- Ignoring segments entirely, or the opposite: slicing into many segments until one looks significant by chance.
When you cannot run an A/B test
Sometimes randomising is impossible (a price change by law, a change that affects everyone). Analysts then compare before and after carefully, or compare similar regions, while being clear that the evidence is weaker.
Learn the foundations
Read hypothesis testing and p-values explained for the theory behind step 5, and simulate random experiments yourself in our NumPy random and simulation set.
Practise what you just read
Reading helps; writing the code yourself is what sticks. Start here: