A practical guide to running rigorous experiments — from statistical foundations through pitfalls and decision frameworks.

Part 1

Foundations

Core concepts: hypotheses, conversion rates, and random variation

Part 2

Statistical Significance

P-values, confidence intervals, and Type I/II errors

Part 3

Power and Sample Size

How to design experiments that can actually detect real effects

Part 4

Pitfalls Part 1

Lucky day traps, SRM, base-rate mismatch, and data loss

Part 5

Pitfalls Part 2

Twyman's Law, underpowered tests, peeking, and overdue experiments

Part 6

Decision Framework

From trustworthy data to clear rollout decisions