Studying stats in college, I remember freaking out a bit when I saw that people kept using z-tests (or t-tests) on non-normal distributions in A/B tests.
I’d had it hammered into me that before running a z-test, you need to check that your data points are independent and identically distributed, from a population with known variance, and that the population itself is normally distributed.
It feels like it should be a problem to run a z-test on a non-normal distribution. But it usually isn’t, because an A/B test doesn’t test individual raw values; it doesn’t even test a single mean. It tests the difference in means between two groups.
Because of this, the z-test doesn’t actually assume your data is normal. It assumes the sampling distribution of that difference is normal. Thanks to the central limit theorem, that assumption holds even when the raw data in both groups is wildly non-normal, as long as your sample sizes are large enough.
CLT for the win
The central limit theorem states that for independently and identically distributed (i.i.d.) observations with finite variance, the distribution of the sample mean approaches a normal distribution as n grows, regardless of the shape of the original distribution. This applies to each group’s mean individually.
Since the difference of two independent, approximately normal random variables is itself approximately normal, the sampling distribution of the treatment effect (mean of B minus mean of A) converges to normal, too.
With the sample sizes typical in online experimentation (thousands to millions of units), the mean of a Bernoulli variable (conversion), a Poisson-ish count, or a nasty long-tailed revenue metric all have sampling distributions that are effectively normal in each group. So does their difference. The skew in the raw data gets averaged out on both sides before the comparison ever happens.
Z-test away
Most experimentation platforms (like ours) run tests on metrics aggregated at the user level, and sample sizes are usually large. With that kind of scale, the central limit theorem kicks in hard, and the normality assumption on the sampling distribution of the difference holds up well even when the raw metric distribution is ugly.
So don’t worry, your Stats 101 flashbacks are just flashbacks. You can run z-tests on non-normal distributions, and it will be OK.
Bonus: Try it out
If you are an R snob like myself, here is a snippet of code you can run off your A/B test to see the central limit theorem in action on your own data:
# =====================================================================
# Central Limit Theorem (CLT) demo for an A/B TEST
# ---------------------------------------------------------------------
# Shows how the DIFFERENCE IN MEANS between two groups (control vs.
# treatment) is approximately normal — the basis for A/B test stats.
#
# Packages required: none (uses base R only)
# =====================================================================
# ---- 1. Point this at the customer's data ----------------------------
# Your data frame needs two columns:
# - a GROUP column identifying the group (e.g. "control" / "treatment")
# - a numeric VALUE column (the metric being measured)
#
# Example (delete this and use your own):
# df <- read.csv("your_ab_test.csv")
df <- data.frame(
group = rep(c("control", "treatment"), each = 5000),
value = c(rexp(5000, rate = 1.0), # control (mean = 1.00)
rexp(5000, rate = 0.9)) # treatment(mean ~ 1.11)
) # <- sample data; replace me
group_column <- "group" # <- column that holds the group labels
value_column <- "value" # <- numeric column to compare
control_label <- "control" # <- value in group_column for group A
treatment_label <- "treatment" # <- value in group_column for group B
# ---- 2. Settings you can tweak ---------------------------------------
sample_size <- 30 # size of each per-group sample (n)
n_samples <- 2000 # how many A/B "replays" to simulate
# ---- 3. Split into the two groups ------------------------------------
control <- df[[value_column]][df[[group_column]] == control_label]
treatment <- df[[value_column]][df[[group_column]] == treatment_label]
control <- control[!is.na(control)]
treatment <- treatment[!is.na(treatment)]
# ---- 4. Repeatedly sample each group, record the difference in means --
set.seed(42) # reproducible; remove for different results each run
mean_diffs <- replicate(n_samples, {
s_ctrl <- sample(control, size = sample_size, replace = TRUE)
s_trt <- sample(treatment, size = sample_size, replace = TRUE)
mean(s_trt) - mean(s_ctrl) # treatment minus control
})
# ---- 5. Plot the distribution of the difference in means -------------
hist(mean_diffs,
breaks = 40,
freq = FALSE,
col = "lightblue",
border = "white",
main = sprintf("Difference in Means: treatment - control (n = %d/group)", sample_size),
xlab = "Sample mean(treatment) - mean(control)")
# CLT: the difference in means is ~Normal, centered on the true
# difference, with standard error sqrt(SE_ctrl^2 + SE_trt^2).
true_diff <- mean(treatment) - mean(control)
se_diff <- sqrt(var(control) / sample_size + var(treatment) / sample_size)
curve(dnorm(x, mean = true_diff, sd = se_diff),
col = "darkblue", lwd = 2, add = TRUE)
abline(v = 0, col = "grey40", lwd = 2, lty = 3) # "no effect" line
abline(v = true_diff, col = "red", lwd = 2, lty = 2) # true difference
legend("topright",
legend = c("No effect (0)", "True difference"),
col = c("grey40", "red"),
lty = c(3, 2), lwd = 2, bty = "n")
# ---- 6. Quick numeric summary ----------------------------------------
cat("Control mean: ", round(mean(control), 4), "\n")
cat("Treatment mean: ", round(mean(treatment), 4), "\n")
cat("True difference (trt - ctrl): ", round(true_diff, 4), "\n")
cat("Mean of simulated diffs: ", round(mean(mean_diffs), 4), "\n")
cat("Theoretical standard error: ", round(se_diff, 4), "\n")
cat("Observed SD of diffs: ", round(sd(mean_diffs), 4), "\n")
# Share of simulated experiments that detected a positive effect
cat("Share of samples with diff > 0:", round(mean(mean_diffs > 0), 3), "\n")
