Search This Blog

A/B Testing Guide

There are two main ways people approach testing:
1. Comprehensive Evaluation
2. Going for quick wins- testing elements most noticeable & easy to implement

Below is a list of useful leanings that I've gathered over time which work fairly well with institutionalizing a testing strategy/program in organizations.

10 Principles for A/B Testing:

1.       have 100 (or 250) conversions per variation
a.      if confidence is still not 95% (or higher) This means there is no significant difference between the variations
b.      Check the test results across segments to see if significance was achieved in any segment 

2.       Tests need to run for full weeks (7 days minimum)
a.       We need to rule out seasonality and test for full weeks
                                                               i.      Because your conversion rate can vary greatly depending on the day of the week

3.       Send test data into your Analytics tool & study segments of traffic
a.       While Optimizely has some built-in segmentation of results, it’s still no match to what you can do within a web Analytics tool
b.      If A beats B by 10%, that’s not the full picture. You need to segment the test data, that’s where the insights lie- Averages lie
                                                               i.      segment the heck out of result- by source of traffic, by user behavior(repeat visits), by outcomes (goals).

4.       Declare winner only when you’ve reached 95% or higher
a.       A/B split testing tools like Optimizely or VWO both tend to call tests too early: their minimum sample sizes are way too small.
b.      A sample size of 100 visitors per variation is not enough. Optimizley leads many people to call tests early and doesn’t have a setting where you may change the minimum sample size needed before declaring a winner
                                                               i.       tool for calculating/confirming sample sizes:

5.       Test Hypothesis, not random ideas
a.      A hypothesis is a proposed statement made on the basis of limited evidence that can be proved or disproved and is used as a starting point for further investigation.
b.      complete proper conversion research to discover where the problems lie, and then perform analysis to figure out what the problems might be, ultimately coming up with a hypothesis for overcoming the site’s problems.
                                                               i.      Use Web Analytics to identify pain points on the site
c.       Have clear action plans stated based on data results prior to launching tests

6.       Designing and Running best-of-class Experiments:
a.       Test one element (or section) of the page at a time between variations
b.      don’t test too many variations at once. It’s better to do simple A/B testing, you’ll get results faster
c.       test segments of visitors (target) as averages can lead to false positives
d.      Elements of Experiment:
                                                               i.      Existence: helps examine clutter and points of confusion & most important element on page
                                                             ii.      Real Estate: after most favorable element identified rearrange layout
                                                            iii.      Presentation: image, placement, color, background, borders, outline, style, font etc
                                                           iv.      Function: navigation, search, tools, flash, pop-ups, widgets, forms
                                                             v.      Copy: headline, paragraph vs. bullet, keywords, value prop, price, short vs. long
e.      Experiment Types:
                                                               i.      a/b: testing a single element with different versions
                                                             ii.      multivariate: testing multiple elements with different versions
                                                            iii.      multi-page: testing single element across multiple pages- usually for funnels

7.       Don’t overlap traffic with multiple tests
a.       Overlapping traffic will skew results
b.      If you want to test a new version of several layouts at once—for instance product page, cart and checkout—you should use multi-page experiments designed for this purpose.

8.       No Brainer Tests
a.       There is no best color, it’s always about visual hierarchy
b.      Use your traffic on high-impact stuff. Test data-driven hypotheses.
c.       Avoid testing in areas where there is low conversion/ traffic
                                                               i.      Don’t waste time waiting for a test result that takes many month
                                                             ii.      go for massive, radical changes – No testing, just switch

9.       Small gains are Acceptable
a.       If your site is pretty good, you’re not going to get massive lifts all the time
b.      Only 1 out of 8 A/B tests have driven significant change
                                                              i.      I have not failed 10,000 times. I have successfully found 10,000 ways that will not work.
c.       Most winning tests are going to give small gains—1%, 5%, 8%
                                                               i.      You’re going to do many, many tests. If you increase your conversion rate 5% each month, that’s going to be an 80% lift over 12 months. That’s compounding interest
d.      Be aware of conversion rate range of control vs. test
                                                               i.      You may beat control by a 2% at a given time but understand how it relates to controls past performance

10.    Pay attention to external factors
a.       winning test during the holidays might not be a winner in January (seasonality)
b.       You need to be aware of what your company is doing and conversion rate ranges in previous months

Beside the above, there are other factors that play into the mix such as Mobile testing, SEO considerations, prioritization of test ideas, path analysis between test page till conversion, qualitative findings (voice of customer), and organizational structure that should be considered.

here is a guide for getting started and managing expectations....

Tools for influencing Behavior: Expected Impact Time to Implement
  • CTA, headline, paragraph vs. bullet, keywords
  • Low
  • Easy
  • imagery, phone numbers
  • Low
  • Medium
  • rearrange layout
  • medium
  • Medium / High
  • Promotions (free delivery, gift card, % off, etc)
  • High
  • Medium / High
  • Price (test by area, segment, sensitivity, presentation)
  • High
  • High
  • New functionality / redesign (shopping cart, proactive call/chat)
  • High
  • High

Happy Testing!

Credits for my research go to:


  1. Thanks for sharing this article here about the guidance of electrical testing. After reading your article I got very much information about the tips of testing and it resolved many of my doubts. If anyone interested to know more about the electrical testing then visit

  2. You have shared a nice article about the Testing Guide. Your service is very interesting and useful. I liked it. If anyone looking to test and tag company at online, is the best choice.

  3. I recently came across your blog and have been reading along and I find here very much information about testing. test and tag sydney

  4. It's a nice article, Which you have shared here about the A/B testing. Your article is very useful for those who are looking to Tag And Test Melbourne, then visit

  5. Very well written article. It was an awesome article to read. Complete rich content and fully informative. If anyone looking for Test and tagging in Adelaide, Test and tagging 

  6. hanks a lot for sharing such a good source with all, i appreciate your efforts taken for the same. I found this worth sharing and must share this with all.
    Insight Perth are a unique design company with designers performing annual report, prospectus, sustainability report design.
    graphic designers perth