What Is A/B Testing? A Beginner's Guide for App Builders
A/B testing shows two versions of something to different users at random and measures which performs better. How it works, what to test, how much traffic you need (often more than you have), statistical significance in plain English, tools, and the mistakes that produce false winners.
A/B testing means showing two versions of something — a headline, a button, a pricing page — to different visitors at random, then measuring which one leads to more of what you want (sign-ups, purchases, clicks).
- Version A — the current one (the "control").
- Version B — the change you're testing (the "variant").
Because visitors are split randomly, a difference in results is likely caused by the change, not by who happened to visit.
A simple example
Your landing page's sign-up rate is 3%. You think a clearer headline will help.
- Half of visitors see: "The modern invoicing platform"
- Half see: "Get paid on time — automatic invoice reminders for freelancers"
After enough visitors, B converts at 4.1% and A at 3.0%. If that difference is big enough to be unlikely by chance, B wins. (How to build a landing page)
What to test
Changes that could plausibly make a real difference:
- Headlines and value propositions
- Call-to-action wording ("Start free" vs "Create account")
- Pricing page layout, plan names, which plan is highlighted (How to price your SaaS)
- Sign-up flow length (fewer fields)
- Onboarding steps
- Free trial length (Freemium vs free trial)
Button colour tests are famous, but rarely matter much. Test bigger ideas.
You may not have enough traffic (yet)
This is the uncomfortable truth for most new products. To reliably detect a modest improvement — say, 3% to 3.6% — you typically need thousands of visitors per version. With 200 visitors a week, a test could take months, and small apparent differences are mostly noise.
With little traffic:
- Make bold changes — big differences are detectable with fewer visitors.
- Talk to users instead. Five conversations often teach more than a month-long test. (How to validate your app idea)
- Make the change and watch — an imperfect before/after comparison is still useful.
Use an online sample-size calculator before starting, to see how long a test would really need.
Statistical significance, in plain English
"Significant at 95%" roughly means: if there were really no difference between A and B, you'd see a gap this big less than 5% of the time by luck. It does not mean "95% chance B is better".
Practical rules:
- Decide the sample size in advance and run the test until you reach it.
- Don't stop early because B looks ahead after two days — "peeking" and stopping at the first good-looking result produces lots of false winners.
- Run for full weeks — behaviour differs between weekdays and weekends.
- One main metric per test, decided up front.
How to run one
No-code / marketing tools: many landing page builders and analytics tools include A/B testing.
Feature-flag tools: PostHog, GrowthBook, LaunchDarkly, Statsig and others split users between variants and measure results. (Feature flags)
DIY: assign each visitor (or logged-in user) to A or B randomly, store the assignment so they always see the same version, and record conversions per group:
const variant = getStoredVariant(user.id) ?? (Math.random() < 0.5 ? 'A' : 'B')
saveVariant(user.id, variant)
track('pricing_viewed', { variant })
Make sure the same person always sees the same version, or the results are meaningless. (Analytics for beginners)
Common mistakes
- Testing with too little traffic, then believing the noise.
- Stopping the test early.
- Testing many things at once and not knowing which mattered.
- Optimising a small metric (clicks) that hurts a bigger one (retention, revenue).
- Forgetting cookie consent rules if your testing tool sets cookies. (Do you need a cookie banner?)
EasySpawn runs your app on its own server, where Claude Code can add feature flags, variant tracking and the reporting queries to see which version won. See how it works or join the waitlist.
Related: Feature Flags for Small Teams · Analytics for Beginners · How to Build a Landing Page · How to Price Your SaaS
Keep reading
What Is v0? Vercel's AI App Builder Explained
v0 is Vercel's AI app builder: describe a UI or an app and it generates a real Next.js project with React, Tailwind and shadcn/ui, with a live preview and one-click publishing to Vercel. What it's good at, how it differs from Lovable and Bolt, and what to know before shipping.
What Is UTF-8? Character Encoding and Garbled Text (Mojibake) Explained
UTF-8 is the character encoding that lets computers store every language and emoji as bytes — and it's the default almost everywhere now. How encoding works, why you see garbled text like ’ or é, and how to fix it in HTML, databases, CSV files and code.