What Is A/B Testing? How It Works, Examples, Pitfalls

A/B testing means showing two versions of something — a headline, a button, a pricing page — to different visitors at random, then measuring which one leads to more of what you want (sign-ups, purchases, clicks).

Version A — the current one (the "control").
Version B — the change you're testing (the "variant").

Because visitors are split randomly, a difference in results is likely caused by the change, not by who happened to visit.

A simple example

Your landing page's sign-up rate is 3%. You think a clearer headline will help.

Half of visitors see: "The modern invoicing platform"
Half see: "Get paid on time — automatic invoice reminders for freelancers"

After enough visitors, B converts at 4.1% and A at 3.0%. If that difference is big enough to be unlikely by chance, B wins. (How to build a landing page)

What to test

Changes that could plausibly make a real difference:

Headlines and value propositions
Call-to-action wording ("Start free" vs "Create account")
Pricing page layout, plan names, which plan is highlighted (How to price your SaaS)
Sign-up flow length (fewer fields)
Onboarding steps
Free trial length (Freemium vs free trial)

Button colour tests are famous, but rarely matter much. Test bigger ideas.

You may not have enough traffic (yet)

This is the uncomfortable truth for most new products. To reliably detect a modest improvement — say, 3% to 3.6% — you typically need thousands of visitors per version. With 200 visitors a week, a test could take months, and small apparent differences are mostly noise.

With little traffic:

Make bold changes — big differences are detectable with fewer visitors.
Talk to users instead. Five conversations often teach more than a month-long test. (How to validate your app idea)
Make the change and watch — an imperfect before/after comparison is still useful.

Use an online sample-size calculator before starting, to see how long a test would really need.

Statistical significance, in plain English

"Significant at 95%" roughly means: if there were really no difference between A and B, you'd see a gap this big less than 5% of the time by luck. It does not mean "95% chance B is better".

Practical rules:

Decide the sample size in advance and run the test until you reach it.
Don't stop early because B looks ahead after two days — "peeking" and stopping at the first good-looking result produces lots of false winners.
Run for full weeks — behaviour differs between weekdays and weekends.
One main metric per test, decided up front.

How to run one

No-code / marketing tools: many landing page builders and analytics tools include A/B testing.

Feature-flag tools: PostHog, GrowthBook, LaunchDarkly, Statsig and others split users between variants and measure results. (Feature flags)

DIY: assign each visitor (or logged-in user) to A or B randomly, store the assignment so they always see the same version, and record conversions per group:

const variant = getStoredVariant(user.id) ?? (Math.random() < 0.5 ? 'A' : 'B')
saveVariant(user.id, variant)
track('pricing_viewed', { variant })

Make sure the same person always sees the same version, or the results are meaningless. (Analytics for beginners)

Common mistakes

Testing with too little traffic, then believing the noise.
Stopping the test early.
Testing many things at once and not knowing which mattered.
Optimising a small metric (clicks) that hurts a bigger one (retention, revenue).
Forgetting cookie consent rules if your testing tool sets cookies. (Do you need a cookie banner?)

EasySpawn runs your app on its own server, where Claude Code can add feature flags, variant tracking and the reporting queries to see which version won. See how it works or join the waitlist.

What Is A/B Testing? A Beginner's Guide for App Builders