A look at Google Content Experiments

Guest contributor Nick Eubanks looks at Google’s newest foray into user-experience testing.

Although it is not a new practice, SEO as an industry seems to have only recently (within the past four years) realized the true potential impact of user-experience testing.

The demand for more sophisticated testing tools has lead to the creation of a number of applications focused on the 2 prevailing testing methodologies: split testing and multivariate testing.

When user-experience testing tools are used in conjunction with a well-integrated SEO campaign, you start down the path of conversion rate optimization.

The relationship between SEO and UX

The relationship between user experience (UX) and search engine optimization has always been there. SEO acquires the prospects and UX nurtures them through the conversion funnel, working symbiotically.

When you are driving hundreds, thousands, or hundreds of thousands of visitors to your web pages everyday, a five percent increase in one of your target conversions can mean significant revenue.

For this reason it is important as an organization to adopt user-experience testing at every level, so everyone is always asking “How can this be different?” as well as “How can this be better?”

Where Google fits in

Google got into the user-experience game in 2006. Their flagship product was named Google Website Optimizer, and was a very simple, very clean tool for running split and multivariate tests.

GWO quickly became a staple in user-experience testing. This was not a surprise since the tool was pretty easy to use and was able to integrate with most websites and content management systems.

One of the limitations was analyzing data at the cross-section of SEO and user experience. It was difficult without some advanced configuration to properly attribute how tests affected other on-site factors such as average time on site, pageviews, bounce rate, etc. Even then, with segments set up, it was still difficult to back out conversions to specific keywords.

In the wake of the growing demand (mostly from frustrations) for tighter integration between Google Analytics and Google Website Optimizer, Google merged the applications.

Introducing Google Content Experiments

Here is a preview of what the new user interface looks like for Google Content Experiments within Google Analytics:

As you can see, it is still nice and clean. Overall I think the reporting interface has gotten much better. However, I have seen a number of recent reports of some data reporting bugs.

What you need to know

This was not a 100 percent integration. With the move, some new things came out and some old things went away. Here is how they affect the tool:

The benefits

• Advanced segments
  If you have used Google Analytics in the past, you are already familiar with the power of advanced segments. If not, advanced segments allow you to create custom filters using almost any of the analytics heuristics, and apply them globally across a profile. This means any data you view, including your experiment data, will be filtered according to these settings.
• One control script
  Before you had to install one script for Google Analytics, and then another set of scripts for Website Optimizer, on the control page, all tracked variations, and the conversion page.
• Traffic allocation
  You can now set your experiments to dynamically receive a set percentage of your visitors, allowing you to throttle or constrain experiments as desired.
• Improved statistical engine
  Google has now added in the functionality to allow for your experiments to work toward statistical significance within a confidence interval.
• Regular expression filters
  This is an extension of Google Analytics functionality into GCE, allowing you to use regular expressions and URL parameters to filter out unwanted test data (for example, internal users).

The drawbacks

• Three-month maximum
  All experiments will expire after approximately 90 days. Google claims this is to avoid continuing to run experiments that will never achieve statistical significance. However, in my experience smaller gains (which can prove big in the long-run) tend to take more time to be representative of a large and diverse set of conversion prospects.
• Two-week minimum
  Google will not report to you on the statistical significance of your experiments before two weeks, regardless of how soon they become statistically significant. In my opinion this is just silly. If your website has the traffic to achieve significance at your confidence interval in even 13 days, that is when it should be reported. Any instance where a better-performing variation is not being shown is costing conversions. Google says this is for the betterment of the test as it will ensure people do not jump to conclusions. Personally, I would rather be given the data and left to make my own decisions.
• No pause button
  Even though Google claims it will attempt to shift traffic away from low-performing variations, you cannot physically pause them. Again, this just seems silly. If a variation is on a steady decline in terms of conversions I want to be able to stop that variation at any time. I see this as another way that using this tool could potentially cost you revenue.
• Maximum tests and variations
  You are limited to a maximum of 12 active tests, each with five variations. I understand it is a free tool and this is most likely in an attempt to regulate bandwidth, this is going to be a large impedance for larger websites. I have seen scenarios where there are more than 12 active tests on just one page.
• No multivariate testing
  Plain and simple, it’s gone. This is probably the most significant drawback of GCE. Multivariate testing enables you to test multiple elements on a page to see which combination of elements creates the highest conversion rate. I believe this was a move on Google’s part to specialize within their target user demographic. Most of the hardcore Google Analytics professionals I know prefer to work almost exclusively with split testing as it offers no guesswork and usually results can be attributed much faster.
• Dedicated URLs required
  GCE does not support testing variations on the same page. A entirely new page, at a new dedicated URL, must be created in order to test variations.This just creates more tedious work. It is not hard to duplicate a page and then cut out the pieces you are testing, but it is not as simple as being able to simply run both tests on the same URL.

The key takeaways

Google Content Experiments offers limited testing flexibility but greater visibility into related visitor attributes. It’s probably the best fit for companies or individuals who are just getting started with UX testing and plan to do so on a smaller scale.

Also, an ideal user will have either a very simple set of web pages to test or access to someone who is able to help them physically recreate their test variation pages.

Of course, I have always found it best to test things myself. If you want to go give GCE a test drive Theo Bennet has put together a nice tutorial on how to set up your first content experiment.