How to use the statistical significance calculator
Why calculating statistical significance in A/B tests matters
Calculating statistical significance is crucial in A/B tests to make sure your results are not the product of coincidence or random variability. Without this calculation, your results may not reflect the real impact of the changes you implemented.
In an A/B test, we compare two variants: version A (control) and version B (modified). The goal is to determine which of the two versions has superior performance, whether in terms of conversion rate, clicks, purchases or other key metrics.
Calculating statistical significance helps us answer one key question:
Is the change we made big enough for us to trust the results and act on them?
If we don’t calculate significance correctly, we risk making decisions based on faulty data, which could hurt the performance of our strategy.
What does a p-value mean in an A/B test?
In an A/B test, the p-value is a crucial indicator that helps us determine whether the observed results are statistically significant or could have occurred by chance. A low p-value (generally below 0.05) indicates that the results are statistically significant.
A low p-value suggests that it is very unlikely that the difference observed between variants A and B was due to chance. Therefore, we can be more confident that the change we implemented had a real effect.
High p-value
A high p-value, on the other hand, means there isn’t enough evidence to reject the null hypothesis, that is, there is no significant difference between versions A and B.
Important factors to consider in an A/B test
When you run an A/B test, several factors influence the validity and effectiveness of your results. Here are the most important ones:
1. Sample size
It’s essential to have a large enough sample for your results to be representative. A small sample size can make results less reliable, since there may be variability in the data.
If the sample is too small, the test may not be statistically significant, even if the real difference between variants is substantial.
2. Test duration
An A/B test must run long enough to capture real variability in the data. If the test runs for too short a period, it may not reflect users’ usual behavior.
For example, if you run the test for a single day, you might miss important data if that day was atypical. In general, it’s recommended that tests last at least one week to make sure the results are representative.
3. Controlling external variables
It’s important to control external factors that may affect the results. This includes:
- Seasonal changes
- Promotions or special events
- Disruptions to the user experience (for example, website bugs)
These variables can influence the performance of version A or B, so they should be minimized or taken into account when interpreting the results.
4. The null hypothesis
Before starting an A/B test, you should always define a null hypothesis. The null hypothesis assumes that there is no difference between the two versions you’re testing.
For example, the null hypothesis could be:
“There is no difference in conversion rate between version A and version B.”
Rejecting the null hypothesis means you’ve found a significant difference between the two variants, which gives you the confidence to make informed decisions about the direction of your strategy.
5. Considering margins of error
In an A/B test, there is always an inherent margin of error. It’s important to be aware of the probability of making Type I errors (false positives) or Type II errors (false negatives). These errors can influence how you interpret the results.
- Type I error (false positive): Concluding there is a difference when there actually isn’t one.
- Type II error (false negative): Failing to detect a difference when one really exists.
Make sure you calculate and account for these margins before making final decisions.
Conclusion
Calculating statistical significance is a critical step in any A/B test. It helps us determine whether the results we obtained are relevant or simply the product of random variability. Additionally, taking other factors into account, such as sample size, test duration and external variables, ensures your results are reliable and that the decisions you make based on them are the right ones.
Although traditional significance is the starting point, many specialists prefer to cross-check this data with probability models to reduce uncertainty. If you’re looking for an analysis that quantifies the risk of loss and the real probability of winning, you can use this Bayesian A/B test calculator, which offers a view much more oriented toward business decision-making than the simple p-value.
Making decisions based on sound statistics is essential to optimize performance and achieve your goals effectively and efficiently.
Frequently Asked Questions about Significance in A/B Tests
Click on each question to expand the answer about the statistical significance calculator in an A/B test:
What does it mean for a result to be “statistically significant”?
It means the performance difference observed between Version A and Version B is very unlikely to be due to chance. Technically, it indicates that we’ve gathered enough evidence to reject the null hypothesis at a given confidence level (usually 95%).
Why is the p-value so important in my calculator?
The p-value is the probability that the difference you see between your variants is pure random noise. If your p-value is below 0.05, the calculator will tell you the result is significant, giving you the green light to implement the change on a solid foundation of confidence.
Which significance level should I choose: 90%, 95% or 99%?
The standard in the Growth & CRO industry and web optimization is 95%. If you choose 99%, you’ll be much stricter and avoid false positives, but you’ll need a much larger user sample. 90% is sometimes used in fast-testing environments where a slightly higher risk of error is accepted in exchange for speed.
Can I trust the calculator if I have few conversions?
It’s not recommended. Statistics requires a minimum sample size to be reliable. If you enter very low numbers (e.g. fewer than 30 conversions per variant), the margin of error will be so high that the result, even if it comes out “significant,” could be a false positive caused by the high volatility of small data sets.
What is a “False Positive” (Type I Error) in an A/B Test?
A false positive occurs when the calculator says there is a winner, but in reality there is no real difference in user behavior over the long term. This usually happens when the test is stopped too early or when external factors such as temporary promotions or seasonality are ignored.
Why isn’t my test reaching statistical significance?
There are three common reasons: 1) The difference between the variants is too small to be detected, 2) Traffic is insufficient for the chosen confidence level, or 3) The test hasn’t run long enough. If there’s no significance after 14 days, consider making a more drastic change in Version B or increasing the experiment’s traffic.
For more information about A/B testing, web optimization and other performance improvement strategies, visit our web optimization section, where you’ll find related articles and resources.
Haven’t launched your experiment yet?
Before you start, it’s vital to know how long you need to collect data for your results to be valid.
