Topic

A/B testing: guides, agencies and tools

A/B testing compares a change against the current experience with real visitors and measures the difference. It is the backbone of most CRO programs, but it is not the only method. Research, usability testing and analytics decide what is worth testing.

Key points

  • Check test feasibility (traffic, baseline conversion, effect size) before committing
  • Decide the primary metric and stopping rule before launch
  • Revenue per visitor usually beats conversion rate as an ecommerce metric
  • Low-traffic sites can still improve through research-led fixes

Guides

  • CRO agency vs in-house team: The cost, speed and capability trade-offs between hiring a CRO agency, building an in-house experimentation team, or combining both.
  • Dedicated CRO agencies vs full-service agencies: When to hire a CRO-only specialist, a CRO-led agency or a multi-service agency with a CRO practice, and the trade-offs of each.
  • How much does a CRO agency cost?: What CRO and experimentation agencies charge, how retainers, guarantees and pay-for-lift models work, and what should be included in the fee.
  • How much traffic do you need for A/B testing?: How traffic, baseline conversion rate and the size of the change you want to detect decide whether A/B testing is practical, and what to do when it is not.
  • How to choose a CRO agency: A practical checklist for shortlisting conversion rate optimization agencies, with the questions to ask on the first call and the red flags to avoid.

Rankings

Tools

VWO

Experimentation and personalization suite
9.1

Mid-market teams wanting testing, behavior analytics and personalization in one suite.

Pricing
Plans published on vendor site; free starter tier historically offered
Vendor
Wingify

Convert

Privacy-focused A/B testing
8.3

Agencies and privacy-conscious teams wanting a flexible testing tool.

Pricing
Plans published on vendor site
Vendor
Convert Insights

GrowthBook

Open-source experimentation and feature flags
9.6

Teams wanting open-source, warehouse-native experimentation.

Pricing
Free and open source; paid cloud plans
Vendor
GrowthBook