Experimentation and Testing: A Primer

This post is a primer on the delightful world of testing and experimentation (A/B, Multivariate, and a new term from me: Experience Testing). There is a lot on the web about A/B or Multivariate testing but my hope in this post is to give you some rationale around importance and then a point of view on each methodology along with some tips.

I covered Experimentation and Testing during my recent speech at the Emetrics Summit and here is the text from that slide:

Experiment or Go Home:
• Customers yell our problems (when they call or see us), they bitch, they rarely provide solutions
• Our bosses always think they represent site users and they want to do site design (which all of us promptly implement!!)
• The most phenomenal site experience today is stale tomorrow
• 80% of the time you/we are wrong about what a customer wants / expects from our site experience

That last one is hard to swallow because after all we are quite full of ourselves. But the reality is that we are not our site's customer, we are too close to the company, its products and websites. Experimentation and testing help us figure out we are wrong, quickly and repeatedly and if you think about it that is a great thing for our customers, and for our employers.

Experimentation and testing in the long run will replace most traditional ways of collecting qualitative data on our site experiences such as Lab Usability. Usability (in a lab or in a home or remotely) is great but if our customers like to surf our websites in their underwear then would it not be great if we could do usability on them when they are in their underwear?

It is important to realize that experimentation and testing might sound big and complex but it is not. We are lucky to live at at time when there are options available that allow us to get as deep and as broad as we want to be, and the cost is not prohibitive. There are three types of testing that are most prevalent (the first two are common).

A/B Testing: Usually this is a all encompassing category that seems to represent all kinds of testing. But A/B testing essentially represents testing of more than one version of a web page. Each version of the web page usually uniquely created and stand alone. The goal is to try, for example, three versions of the home page or product page or support FAQ page and see which version of the page works better. Almost always in A/B testing you are measuring one outcome (click thrus to next page or conversion etc). If you do nothing else you should do A/B testing.

How to do A/B Testing: You can simply have your designers/developers create versions of the page and depending on the complexity of your web platform you can put the pages up and measure. If you can't test them at the same time put them up one week after the other and try to control for external factors if you can.

Pro's of doing A/B Testing:
• This is perhaps the cheapest way of doing testing since you will use your existing resources and tools
• If you don't do any testing this is a great way to just get going and energize your organization and really have some fun
• My tip is first few times you do this have people place bets (where legal) and pick winners, you'll be surprised


Con's of doing A/B Testing:

• It is difficult to control all the external factors (campaigns, search traffic, press releases, seasonality) and so you won't be 100% confident of the results (put 70% confidence in the results and make decisions)
• It is limiting in the kinds of things you can test, just simple stuff and usually it is hard to discern correlations between elements you are testing

Multivariate Testing: Currently the cool kid on the block, lots of hype, lots of buzz. In A/B above you had to create three pages. Now imagine "modularizing"? your page (break it up into chunks) and being able to just have one page but change dynamically what modules show up on the page, where they show up and to which traffic. Then being able to put that into a complex mathematical engine that will tell you not only which version of the page worked but correlations as well.

For example for my blog I can create "modules" / "containers" of the core page content, the top header, and each element of the right navigation (pages, categories, links, search etc). In a multivariate test I could move each piece around and see which one worked best.

Pro's of doing Multivariate Testing:
• Doing Multivariate turbocharges your ability to do a lot very quickly for a couple of reasons
  • There are free tools like the Google Website Optimizer or paid tools like Offermatica, Optimost, and SiteSpect who can help you get going very quickly by hosting all the functionality remotely (think asp model) such as content, test attributes, analytics, statistics.
  • You don't have to rely on your IT/Development team. All they have to do is put a few lines of javascript on the page and they are done. This is a awesome benefit because most of the times that is a huge hurdle.
• It can be a continuous learning methodology

Con's of doing Multivariate Testing:
• The old computer adage applies, be careful of GIGO (garbage in, garbage out). You still need a clean pool of ideas that are sourced from known customer pain points or strategic business objectives. It is easy to optimize crap quickly.
• Website experiences for most sites are complex multi page affairs. For a e-commerce website it is typical for a entry to a successful purchase to be around 12 to 18 pages, for a support site even more pages (as we thrash around to find a answer!). With Multivariate you are only optimizing one page and no matter how optimized it cannot play a outsized role in final outcome, just the first step or two.

Most definitely do Multivariate but be aware of its limitations (and yes the vendors will tell you that they can change all kinds of things throughout the site experience, take it with a grain of salt and take time to understand what exactly that means).

Experience Testing: New term that I have coined to represent the kind of testing where you have the ability to change the entire site experience of the visitor using capability of your site platform (say ATG, Blue Martini etc). You can not only change things on one page or say the left navigation or a piece of text on each page, but rather you can change all things about the entire experience on your website.

For example lets say you sell computer hardware on your website. Then with this methodology you can create one experience of your website where your site is segmented by Windows and Macintosh versions of products, another experience where the site is segmented by Current customers and New customers and another where the site is purple with white font with no left navigation and smiling babies instead of product box shots.

With experience testing you don't actually have to create three or four websites, but rather using your site platform you can easily create two or three persistent experiences on your websites and see which one your customers react to best. Since any analytics tools you use collect data for all three the analysis is the same you do currently.

Pro's of doing Experience Testing:
• This is Nirvana. You have an ability to test on your customers in their native environment (think underwear) and collect data that is most closely reflective of their true thoughts.
• If your qualitative methods are integrated you can literally read their thoughts about each experience.
• You will get five to ten times more powerful results than any other methodology

Con's of doing Experience Testing:
• You need to have a website platform that supports experience testing, (for example ATG supports this)
• It takes longer than the other two methodology
• It most definitely takes more brain power

Experience testing is very aspirational but companies are getting into it and sooner rather than later the current crop of vendors will start to expand into that space as well.

Agree? Disagree? Counter claims? Please share your feedback via comments.