Analysts rarely have data on everyone. A customer survey, a sample of support calls, a pilot in two cities: these are samples, and conclusions are only as good as the way the sample was chosen.
Population and sample
- Population: everyone or everything you want to draw conclusions about.
- Sample: the subset you actually measure.
A good sample is representative: it looks like the population in the ways that matter.
Sampling methods
Simple random sampling
Every member has an equal chance of selection. In pandas: df.sample(n=500, random_state=42). It is the baseline for fairness.
Stratified sampling
Split the population into groups (strata), such as city tiers or customer segments, and sample from each. It guarantees that small but important groups are included and can make estimates more precise.
sample = df.groupby("city_tier", group_keys=False).sample(frac=0.05, random_state=42)Systematic sampling
Take every kth item from a list, such as every 20th invoice. Simple, but risky if the list has a repeating pattern that lines up with k.
Cluster sampling
Randomly choose whole groups (some stores, some schools) and measure everything inside them. Cheaper to collect, but results can vary more if clusters differ from each other.
Convenience sampling
Using whoever is easiest to reach: the customers who answered an email, the stores near the office. Quick, but often biased.
Common biases in business data
Selection bias
The sample differs systematically from the population. Analysing only customers who stayed a year ignores everyone who left early.
Non-response bias
People who answer surveys differ from those who do not. Very happy and very unhappy customers often respond more than the indifferent middle.
Survivorship bias
Looking only at survivors: studying only successful products or only active accounts leads to over-optimistic conclusions.
Measurement bias
The way data is collected distorts it: leading survey questions, a sensor that reads high, a field filled in only when something goes wrong.
Time bias
A sample from one week, a festival period or a sale may not represent normal behaviour.
How analysts reduce bias
- Ask how the data was collected and who might be missing.
- Compare sample characteristics with known population figures (share by city, by age, by channel).
- Use random or stratified sampling where you control it.
- Weight results when some groups are over- or under-represented.
- State limitations clearly in the report.
Sample size
Larger samples reduce random error, but they do not fix bias. A huge biased sample is still biased. Precision depends on sample size; accuracy depends on how the sample was chosen.
Practise
Simulate random and stratified samples and watch estimates vary in the NumPy random set. Experiments are the strongest defence against bias; see the A/B testing guide.
Practise what you just read
Reading helps; writing the code yourself is what sticks. Start here: