---
title: "A/B testing: when it pays, what to fix first | PostCliq"
description: "A/B testing and split testing for marketing teams: why only 12% of tests win, the traffic you need, and which leaks to fix before you test."
url: "https://postcliq.com/blog/ab-testing-for-marketing-teams"
---

# A/B testing for marketing teams: when it pays and what to fix first

A/B testing pays when a page has enough conversions to reach a reliable answer. Most B2B pages do not, and only about one test in eight wins anyway. Here is the arithmetic, and what to fix before you test.

By [Nirmal Gyanwali](https://postcliq.com/blog/author/nirmal-gyanwali) Updated 30 September 2026 8 min read

On this page

1. The short answer
2. What is A/B testing?
3. How often do A/B tests actually win?
4. How much traffic do you need for A/B testing?
5. Why should you fix obvious leaks before you A/B test?
6. What should marketing teams do instead of testing on low traffic?
7. When is A/B testing worth it for a marketing team?
8. Does PostCliq run A/B tests?
9. What to do next
10. Frequently asked questions

## The short answer

Key takeaways

- A/B testing is a controlled experiment that splits traffic between two versions of a page to see which one converts better.
- Most experiments do not win: Optimizely reports that only 12% produce a statistically significant improvement on the primary metric.
- A page converting at 2% needs roughly 42,000 visitors to reliably detect a 20% relative lift, which is over a year of traffic for many B2B pages.
- Low-traffic sites get more from fixing broken forms, slow pages and missing proof than from testing button colours.
- PostCliq does not run A/B tests; it finds the leaks worth fixing before a test is worth running.

A/B testing is worth it for a marketing team when a page gets thousands of conversions a year and the question is genuinely uncertain. Otherwise it is slow and often inconclusive. [Optimizely's analysis of 127,000 experiments](https://www.optimizely.com/insights/top-10-takeaways-from-running-127000-experiments/) found only 12% produce a statistically significant win on the primary metric, and those are mostly enterprise programmes with large audiences. Fix what is plainly broken first.

## What is A/B testing?

A/B testing is a method of comparing two versions of a page, email or ad by randomly showing each to a share of visitors and measuring which produces more of a chosen outcome, such as enquiries. Because the only systematic difference between the groups is the change, a large enough difference can be attributed to it rather than to chance.

Split testing usually means the same thing. Some teams use "split URL testing" for comparing two separate pages, and "multivariate testing" for changing several elements at once and measuring each combination. Multivariate tests need far more traffic than a simple A/B test, because the visitors are divided among many more versions.

What is statistical significance in an A/B test?

Statistical significance is the confidence that a difference between versions is not random noise. The usual threshold is 95%, which means accepting a 5% chance of declaring a winner that is not really better. Significance alone says nothing about whether the difference is big enough to matter.

The upside of testing is real, which is why large companies invest in it. In their [2017 Harvard Business Review article](https://hbr.org/2017/09/the-surprising-power-of-online-experiments) , Ron Kohavi and Stefan Thomke described a Bing change to how ad headlines were displayed that increased revenue by 12%, worth more than USD 100 million a year in the United States. That is the case for testing at scale. The question is whether your site has scale.

## How often do A/B tests actually win?

Most A/B tests do not produce a winner. Optimizely's data shows only 12% of experiments deliver a statistically significant improvement on the primary metric, and even healthy programmes see most tests come back inconclusive. Testing is a volume game: the value comes from running many well-researched tests, not a handful of hunches.

> Only 12% of experiments produce a statistically significant improvement on the primary metric.

Optimizely, analysis of 127,000 experiments

The same analysis adds context. The median company in its data runs 34 experiments a year, and healthy programmes see roughly 35 to 40% of tests reach a conclusive result either way. These are Optimizely customers, mostly large organisations with the traffic and teams to run a steady cadence.

What experimentation programmes really look like

**12%** of experiments produce a significant win on the primary metric [Optimizely, 127,000 experiments](https://www.optimizely.com/insights/top-10-takeaways-from-running-127000-experiments/)

**34** experiments a year run by the median company [Optimizely, 127,000 experiments](https://www.optimizely.com/insights/top-10-takeaways-from-running-127000-experiments/)

**12%** revenue lift from one Bing headline test, over USD 100 million a year [Kohavi and Thomke, Harvard Business Review](https://hbr.org/2017/09/the-surprising-power-of-online-experiments)

Now combine the two numbers. If roughly one test in eight wins, a team running four tests a year should expect about one winner in two years. That is before you ask how long each test takes on your traffic.

## How much traffic do you need for A/B testing?

You need enough visitors to detect the size of change you care about. As a rule, a page converting at 2% needs about 21,000 visitors per version, around 42,000 in total, to reliably detect a 20% relative lift at 95% confidence and 80% power. Smaller lifts need far more. Many B2B pages simply do not have that traffic.

Here is the arithmetic, so you can run it on your own page. The standard sample-size formula for comparing two conversion rates is:

n per version = (1.96 + 0.84)² × \[p1(1 − p1) + p2(1 − p2)\] ÷ (p2 − p1)²

where 1.96 gives 95% confidence, 0.84 gives 80% power, p1 is your current conversion rate and p2 is the rate you hope to reach.

**Worked example.** A B2B services page gets 3,000 sessions a month and converts 2% of them into enquiries, which is 60 enquiries a month. You want to know whether a new headline lifts that by 20%, to 2.4%.

- p1 = 0.02, p2 = 0.024, difference = 0.004
- Variance term = 0.02 × 0.98 + 0.024 × 0.976 = 0.0196 + 0.0234 = 0.0430
- n per version = 7.85 × 0.0430 ÷ 0.000016 ≈ 21,100 visitors
- Total ≈ 42,200 visitors, which at 3,000 a month is about 14 months

Fourteen months is too long. Seasonality, campaigns, a site update or a change in the sales team will have muddied the result long before the end. And if the real effect is 10% rather than 20%, you need about 161,000 visitors: roughly four and a half years.

Months to reach a reliable A/B test result at 3,000 sessions a month

- 2% baseline, detect a 50% lift 2.5months
- 5% baseline, detect a 20% lift 5.4months
- 2% baseline, detect a 20% lift 14.1months
- 2% baseline, detect a 10% lift 53.8months

Source: [PostCliq analysis, standard two-proportion sample size at 95% confidence and 80% power](https://postcliq.com/methodology)

Two things follow. First, low-traffic sites can only detect large effects, so test bold changes, not button colours. Second, for many B2B pages the honest answer is not to test at all, but to fix what is obviously broken and measure before and after.

## Why should you fix obvious leaks before you A/B test?

Fix obvious leaks first because they need no experiment to justify. A form that fails to deliver, a page that takes eight seconds to load on a phone, or a case study link that returns an error is simply broken. Testing a new headline on a page with a broken form measures noise on top of a known loss.

The obvious leaks are also common. The [HTTP Archive Web Almanac 2025](https://almanac.httparchive.org/en/2025/performance) found only 48% of websites pass all Core Web Vitals on mobile. The [WebAIM Million 2026](https://webaim.org/projects/million/) found missing form labels on 51% of the top million home pages. And contact forms convert worse than any other form type in [Zuko's vendor benchmark](https://zuko.io/blog/8-surprising-insights-from-zukos-benchmarking-data) , with only 9.09% of viewers submitting.

| Type of change | Examples | Fix or test? |
| --- | --- | --- |
| Something is broken | Form not delivering, 404 proof links, mobile layout errors | ✓ Fix now. No test needed |
| Something is slow | Mobile load time, heavy images, render-blocking scripts | ✓ Fix now. Measure before and after |
| Something is missing | No response promise, no case study near the claim, no phone number | ✓ Fix, then watch the trend |
| A clear best practice | Unlabelled inputs, required phone field, vague button text | ✓ Fix. Evidence already exists |
| A genuine judgement call | Two different value propositions, pricing page structure | ◐ Test, if traffic allows |
| A cosmetic preference | Button colour, image choice, minor copy tweaks | ✕ Rarely worth testing on low traffic |

Source: PostCliq analysis

The logic is the same as any investment. A test costs time, tooling and traffic. Spend that on questions where the answer is genuinely uncertain and the stakes are high. Everything in the top four rows of the table is a known problem with a known direction.

## What should marketing teams do instead of testing on low traffic?

On low traffic, make changes that are well supported by evidence, then measure the trend before and after using the same period and channel mix. Pair that with qualitative evidence: session recordings, user feedback and sales-call notes. This is not as rigorous as a controlled test, but it is far faster and usually points the same way.

A practical sequence for a B2B marketing team:

1. **Audit the page.** List what is broken, slow, missing or confusing. Our [CRO audit guide](https://postcliq.com/blog/cro-audit) sets out a one-hour method.
2. **Watch real sessions.** Free behaviour tools show where people hesitate or give up. See our guide to [the best CRO tools and which question each one answers](https://postcliq.com/blog/best-cro-tools) .
3. **Fix in order of likely impact.** Start with the enquiry path: the form, the call to action and the proof beside it. Our [contact form best practices](https://postcliq.com/blog/contact-form-best-practices) cover the most common form fixes.
4. **Measure honestly.** Compare the same weeks, the same channels and enquiry quality as well as volume. Note anything else that changed.
5. **Test only when you have a real question and enough traffic.** Use the formula above before you start, not after.

## When is A/B testing worth it for a marketing team?

A/B testing is worth it when four conditions hold: the page gets enough conversions to detect the change you care about within about four to eight weeks, the question is genuinely uncertain, the decision is costly to get wrong, and someone owns the programme. Paid landing pages with steady budgets and ecommerce checkouts most often meet all four.

A quick test of readiness:

- **Volume:** at least a few hundred conversions per version within the test window. The worked example needed about 420 enquiries per version (21,100 × 2%), a useful sanity check on any plan.
- **Stability:** no major campaign, redesign or pricing change planned during the test.
- **Tooling:** a testing platform that does not slow the page. Google retired its free testing tool, Google Optimize, on [30 September 2023](https://support.google.com/optimize/answer/12979939) , so most teams now need a paid tool or a landing page builder with testing built in.
- **Discipline:** a written hypothesis, a primary metric chosen in advance, and a fixed sample size. Stopping a test the moment it looks good is the most common way to declare a false winner.

For context, [Ruler Analytics' 2026 benchmark](https://ruleranalytics.com/blog/insight/conversion-rate-by-industry/) puts the average conversion rate across 13 industries at 5.13%. A page well below the average for its industry usually has a leak to fix before it has a question to test. Our guide to [what a good conversion rate looks like](https://postcliq.com/blog/what-is-a-good-conversion-rate) has the benchmarks by industry.

## Does PostCliq run A/B tests?

No. PostCliq does not run A/B tests, change your site or split your traffic. It is a free website audit that reads your public pages the way a careful visitor would and finds what stops visitors becoming enquiries, with evidence and three quick wins to make first. Many of those findings are fixes that need no test.

That makes it a sensible step before an experimentation programme, not a replacement for one. It checks speed and page experience, forms and calls to action, content clarity, search access, accessibility, trust signals and AI readiness across up to 50 public pages, then re-audits weekly so you can see whether fixes held. If you later run tests, you will be testing on a page that already works. The [how it works page](https://postcliq.com/how-it-works) shows the process.

## What to do next

1. Run the sample-size formula on your highest-value page before committing to any test. If the answer is more than two months, do not test that change.
2. Submit your own enquiry form and time your page on a phone. Fix anything broken before anything else.
3. Work through a [CRO audit](https://postcliq.com/blog/cro-audit) and sort each finding into the fix-or-test table above.
4. Choose one genuine, high-stakes question to test only when a page has the traffic to answer it.
5. Get a prioritised list of the leaks to fix first with a [free PostCliq website audit](https://postcliq.com/#try-postcliq) .

## Frequently asked questions

### What is the difference between A/B testing and split testing?

In practice they mean the same thing: comparing two versions by randomly dividing visitors between them. Some tools use "split URL testing" specifically for comparing two separate pages at different addresses, while "A/B testing" often refers to changing elements on one page. Both need the same sample-size arithmetic to produce a reliable result.

### How long should an A/B test run?

Run a test until it reaches the sample size you calculated before starting, and for at least one or two full weeks so weekday and weekend behaviour are both included. Do not stop early because one version looks ahead, as early leads often reverse. If the required sample would take more than two months, the test is probably not worth running.

### What is a good A/B test win rate?

Lower than most teams expect. Across 127,000 experiments, Optimizely reports that only 12% produce a statistically significant improvement on the primary metric. A programme where most tests come back flat is normal. The goal is learning quickly and cheaply, which is why research quality and testing volume matter more than any single test.

### Can you A/B test with low traffic?

You can, but only for large effects. A page converting at 2% with 3,000 monthly sessions needs about 14 months to reliably detect a 20% lift. Low-traffic teams should test bold changes, use higher-traffic pages, or skip testing and fix evidence-backed problems, measuring before and after with qualitative evidence from recordings and feedback.

### What should you A/B test first?

Test the biggest genuine uncertainty on your highest-traffic, highest-value page: usually the headline and offer, the call to action, or the structure of the enquiry path. Do not test things that are simply broken or against clear best practice, such as unlabelled form fields. Fix those first, because the answer is already known.

### Does Google still offer a free A/B testing tool?

No. Google says Google Optimize and Optimize 360 are no longer available as of 30 September 2023. Teams that relied on it now use paid experimentation platforms, landing page builders with built-in testing, or server-side testing. GA4 can still measure results, but it does not split traffic between versions on its own.

## Fix the leaks first. Free, in minutes.

PostCliq does not run tests. It finds what stops visitors becoming enquiries, so the fixes that need no test come first.

Free. No credit card.

![Nirmal Gyanwali, founder and chief executive of WP Creative.](https://postcliq.com/authors/nirmal-gyanwali.jpg)

Written by

[Nirmal Gyanwali](https://postcliq.com/blog/author/nirmal-gyanwali)

Founder and CEO, WP Creative · Sydney, Australia

Nirmal Gyanwali has spent more than twenty years building websites, and most of that time watching them fail at the one job the business actually bought them for, which is turning a visitor into an enquiry. He founded WP Creative in Sydney in January 2016.

- [LinkedIn](https://www.linkedin.com/in/nirmalgyanwali/)
- [YouTube](https://www.youtube.com/@nirmalgyanwali)
- [Instagram](https://www.instagram.com/nirmalgyanwali)
- [TikTok](https://www.tiktok.com/@nirmalwpc)

## Keep reading

- [**CRO audit: how to run a conversion audit in an hour**](https://postcliq.com/blog/cro-audit)
- [**The best CRO tools in 2026, and which question each one answers**](https://postcliq.com/blog/best-cro-tools)
- [**How to increase your website conversion rate: fixes ranked by effort and impact**](https://postcliq.com/blog/how-to-increase-conversion-rate)

[More on Conversion rate optimisation](https://postcliq.com/blog/topic/conversion) · [All articles](https://postcliq.com/blog)