Hypothesis testing answers one question: is the difference I see in my data likely to be real, or could it easily be random noise? Here is the idea without heavy maths.
A business example
A fictional bakery chain, CrumbCo, trains staff at some branches in upselling. Average bill size at trained branches looks a little higher than at untrained ones. Is training really working, or are the trained branches just having a lucky month?
Two hypotheses
- Null hypothesis (H0): there is no real difference; any gap is due to chance.
- Alternative hypothesis (H1): there is a real difference.
The test does not prove H1. It asks how surprising your data would be if H0 were true.
The p-value
The p-value is the probability of seeing a difference at least as large as yours if there were truly no effect.
- Small p-value: data like yours would be unusual under "no effect", which is evidence against H0.
- Large p-value: data like yours is quite ordinary under "no effect"; you cannot rule out chance.
What a p-value is not:
- Not the probability that H0 is true.
- Not the size or importance of the effect.
- Not proof of anything on its own.
The significance level
Before testing, choose a threshold, often 0.05. If p is below it, you call the result statistically significant and reject H0. The threshold is a convention, not a law of nature; some decisions need stricter thresholds.
Two kinds of errors
| H0 actually true | H0 actually false | |
|---|---|---|
| Reject H0 | Type I error (false positive) | Correct |
| Do not reject H0 | Correct | Type II error (false negative) |
Lowering the threshold reduces false positives but increases misses. Larger samples reduce both.
Common tests and when to use them
| Question | Test |
|---|---|
| Do two groups have different averages? | Two-sample t-test |
| Did the same units change before vs after? | Paired t-test |
| Do more than two groups differ in averages? | ANOVA |
| Are two categorical variables related? | Chi-square test |
| Do two groups differ in a proportion? | Two-proportion z-test |
from scipy import stats
t_stat, p_value = stats.ttest_ind(trained_bills, untrained_bills, equal_var=False)equal_var=False (Welch's t-test) is a safer default when the groups may have different spreads.
Statistical vs practical significance
With enough data, even a tiny difference becomes "significant". Always report the size of the difference and a confidence interval, and ask whether it matters to the business: does the extra revenue cover the cost of training?
Watch out for
- Multiple testing: run enough tests and some will be significant by chance.
- Non-random groups: if the best branches were chosen for training, the comparison is biased regardless of the p-value.
- Data dredging: forming the hypothesis after looking at the data.
Where to use it
A/B tests, before-and-after comparisons, comparing regions or segments, and checking whether a model's improvement is real. See the A/B testing guide for the full workflow.
Practise what you just read
Reading helps; writing the code yourself is what sticks. Start here: