An A/B test is a controlled experiment: you show one part of your users one version of a page or element (variant A, usually the current one) and another part a different version (variant B). You compare a metric, such as the conversion rate, and decide whether the difference is real or just the result of chance.
Why “before and after” isn’t enough
If you change a button on Monday and see more sales on Tuesday, you don’t know whether it was the button, a campaign, the day of the week, or pure chance. In an A/B test, both versions run at the same time with users assigned at random, so all other factors affect both equally. That is the key: randomization lets you attribute the difference to the change.
When it makes sense to run a test
- You have enough traffic. As you’ll see in lesson 5, detecting small improvements requires tens of thousands of visitors per variant.
- There is a decision that depends on the result. If you’re going to launch the change no matter what, a test adds nothing.
- You can measure the goal. A purchase, a sign-up, a click. Without a clear metric there is no experiment.
- There is real uncertainty. If it’s an obvious error (a broken button, a form that doesn’t submit), you fix it, you don’t test it.
When not to do it
With low traffic, a test will take months and will probably be inconclusive. In that case, it’s better to rely on qualitative research (interviews, session recordings, surveys), proven best practices, and bigger changes, and to save A/B testing for when your volume allows it.
The building blocks of an experiment
- Hypothesis: what you think will happen and why.
- Variants: control (A) and one or more alternatives.
- Primary metric: the one that decides the result.
- Sample and duration: how many people and for how long.
- Decision criteria: which result will make you implement, discard, or keep testing.
Something to think about: think of a change you’re about to make on your website. Could you argue that there is real uncertainty and that you have enough traffic to measure it?