Efficient Frontier Insights

Multivariate Testing and Conversion Tweaking

This is a follow up post regarding my presentation from the Multivariate Testing and Conversion Tweaking panel at last week's PubCon in Las Vegas, where I presented alongside Vanessa Fox and Glenn Alsup. The deck from the presentation can be found on slideshare.

As a web designer, I wanted to share my thoughts on a shift in website design that I've seen referred to as "performance-based design" or "data-driven design." Essentially, this shift is about relying on user data and testing to drive design decisions because at the center of this movement is the idea that we can no longer assume that what we think as designers is best for users. This is an idea that I'm comfortable with as a designer who has become deeply entrenched in the online marketing world. The web is the most measurable and testable form of design in history and there are a number of reasons why now is the perfect time to take advantage of that fact.

The two biggest reasons are a down (although recovering) economy and free technology. As we slowly come out of the economic downturn, the lessons that businesses learned will guide their decisions for years to come. Getting "more for less" was how many companies survived and even excelled during these hard times. This involved finding and addressing inefficiencies, which is at the core of landing page testing and conversion tweaking. Free technologies such as Google Analytics and Website Optimizer, although not technically new, have created opportunities for even the smallest businesses to learn more about their visitors and test different approaches in their web design and content strategies.

I predict that in the coming year there will be noticeable effects of this shift towards performance-based design. We'll see a dramatic increase in investment in post-click optimization in order to drive greater ROI. With this will come a hunger for more data and with that, a need for new and meaningful ways to collect and analyze that data. I foresee the core focus of this new data collection and analysis to be social media.

Technology for Multivariate Testing

Multivariate testing is about finding the (sometimes very small) synergies that exist between numerous variables on your page that maximize customer throughput - those little things that when combined in the right way persuade visitors to convert. To do this, you need really complex statistical analysis. The good news is that technology has removed that complexity making it possible to focus on what to test, not how to test.

There are a variety of options and costs out there for managing multivariate testing. Many require some sort of JavaScript tagging to be embedded on your page, while others offer a full DNS-based solution that routes traffic to your site through a testing server first, allowing for fast and easy deployment of tests. Make sure to get your IT department involved. Often times the "yay" or "nay" decision on a technology comes down to them, so it’s best that they aren’t the last people you consult. And my advice is to go with a nimble solution, whatever that means for your specific situation. Success in testing is often the product of momentum and building on the successes of previous testing rounds. It is very difficult to gain that momentum when it takes 6 weeks for a page to be tagged or a new headline to be uploaded.

What should you test?

The simple answer to this is "everything." The problem with this is that if you were to test literally everything, you would find yourself in the middle of a giant mess. So, the challenge becomes knowing where to start.

To figure out where to start, I categorize a page into one of two categories:

• It's a disaster and needs help with broader design based concepts
• It's not bad and could benefit from some tweaking or fine-tuning

How I categorize pages largely relies on intuition and experience, so this is an area where the data might not help.

If a page is a disaster, then no amount of tweaking is ever going to give you worthwhile results. These pages first need help to make them more sticky, so you need to consider concepts like first impressions, user experience, and general layouts. These are the types of things that we consider when we think like designers. 

Since tweaking won’t work, I tend approach these projects by doing a radical redesign, or a completely new and different take on the subject.  For this I might test different page layouts, fonts, and colors while maintaining a consistency in the messaging. Since I might be testing radically different looks, I like to use A/B testing at this stage.

Once the stickiness of a page is maximized, then you have to start thinking like a marketer.  You’ve captured a visitor's attention and now they might actually pay attention to what you are saying! This is where "tweaking" comes into play. Now you have to test specific messaging to figure out what resonates with the user - what persuades them convert.  What are the different messages you could convey through your headline or your hero shot? This is the perfect scenario for multivariate testing where you have individual elements throughout the page that each have multiple options.

Specific Elements to Tweak

I wanted to provide at least a bare minimum of what everyone should be testing (tweaking) on their landing pages. The elements listed below are common to most landing pages, but they also can have a big impact on the conversion potential of your page.

• Brand Presentation – Visitors need to trust your brand. So brand presentation deals with how you present your brand on the page, whether that’s through a simple logo placement or a short introductory paragraph explaining your value proposition. As a general rule, the more well-known a brand is, the less effort you have to put into the brand presentation.
• Call to Action – Every landing page needs one - it’s the action you want your visitors to take. Important things to test about your call to action are placement, emphasis and wording.
• Product Image – How you present your product can influence a visitor's perception of its value. A perfect example comes from digital software downloads.  Pages that show a physical package (even if none exist and was just made up by the landing page designer) tend to convert better even though the user downloads the product and never receives a tangible product. The illusion of something tangible adds value in the user's mind.
• Hero Shot – The simple thought here is that people want to see themselves using the product or service they are considering purchasing.
• Buttons – Don't overlook or underestimate the influence of your buttons. If you have a button on your page, it is probably directly related to the call to action. I suggest never using standard browser buttons and instead opting to create visually stunning image buttons. Also, test the text that you include in your button (hint: it should never say "Submit"). Tell the user what they are going to get when they click on that button.
• Incentives – Offering the right incentive can make or break a page, especially in highly commoditized goods markets. Free shipping is almost always the best incentive you can offer.
• Headlines – If a user has made the decision to stick around on your page for longer than 2 seconds, then the headline is probably going to be the first thing that they read, so make sure that your headline is conveying an important message. To get an idea of how important a headline can be, check out this post about how 37signals improved conversions by 30% with simple headline testing.

Using Data to Determine The Variables for Your Tests

The last piece of this puzzle is figuring out what the variables should be for your test. Here, when I say "variables," I'm talking about the different options to test within the individual page elements, so what are the different headlines you should try or the different button texts.

This is where data on your site visitors and customers can be very useful. Data can fall into two categories: quantitative and qualitative. 
 


Quantitative data is the information that numbers tell us. This can be information from your site analytics about the number of users with a certain language setting or the number of referrals from a certain keyword, and you could use that data to influence whether you offer translations of your content or what keywords you use in your headlines. Quantitative paid search metrics on ad copy testing can help you determine what messaging resonates well with users.

Qualitative data tells you about your customers in a way that has emotion and attitude and a sort of raw honestly. This type of data can come from customer feedback directly to your customer service department or by researching what your customers (and maybe more importantly detractors) are saying about you in social media. Personas (which were covered by Vanessa Fox on our panel) are another great source of qualitative data.

Caveat: There is an important caveat as it relates to data, and I find this to be true anytime you are dealing with data, not just online. Beware of making decisions based on aggregate or average data. Tools like Google Analytics make it easy to focus in on the averages, because they are right there at the top of the data columns in larger, pretty type. But consider a test that you’ve design to reduce the bounce rate on one of your landing pages. After running the test long enough, you are able to reduce the bounce rate by 25%. That's a success right?  Well, yes and no. If you were to segment the data by the source of the traffic you may find that, overall, you reduced your bounce rate by 25%, but you increased the bounce rate on your display traffic by 50%! It's a smaller channel in terms of the overall traffic volume, so it gets lost in the aggregate data.


 
Likewise, you might run a multivariate test on one of your landing pages and find that conversions increased a modest 0.5%. That is probably not the amazing results you were looking for. However, if you take a closer look at the data, you might find that for a certain, small set of related niche keywords, the conversions skyrocketed, while conversions for the other keywords driving traffic to the landing page fell, resulting in the overall increase of only a half a percentage point. This gives you a valuable learning, which is: use the winning landing page for the small set of keywords it performed well on and then rerun your multivariate test without those keywords to pollute the testing process and see if a different combination of variables can increase conversions.

Segmenting Your Data for Meaningful Results

There are a myriad of ways to segment your data, but the 3 primary ones I focus on are:

• Demographic – This is the type of information that is related to personas (age, gender) and those segments that you probably never use on Google Analytics (language setting, country of origin, connection speed, etc.)
• Psychographic – This can be a tough one to nail down because it can deal with complicated measurements like what phase is the decision-making process someone is in regarding a purchasing decision. This is a focus of retargeting efforts in search and display. But you can also get certain psychographic segmented data from you analytics, like repeat (loyal) versus new customers.
• Source – This type of segmenting is very easy to do in your analytics program and can provide a lot of insight. Look for how visitors from different channels like paid search, natural search, affiliates and social media interact with your site and ultimately convert. Most likely, you’ll find it valuable to create different landing pages for different channels.

Something to keep in mind as you segment your data and recognize different conversion patterns, is that not everyone that comes to your site will convert. You need to be able (and willing) to accept that there is no value in some visitors, otherwise you'll waste time chasing unachievable conversion rates.

Take-Aways

These are some of the key takeaways that I summarized at the end of my presentation:

• Doubling your conversion rate is far easier (and cheaper) than doubling your traffic (directly relates to the down economy being a driving force behind testing and post-click optimization)
• Get to know all of your data intimately so you can uncover relationships
• Segment and combine your data for more meaningful interpretation
• Test everything until you know what matters

Ultimately the goal of all of this is to evolve your business through continuous testing and improvement cycles - to truly become data-driven.

Jason Cooper
Director, Value Added Services

How to Deal with Duplicate Content Within a Site

One of my favorite developments in recent years is the AJAX-driven dynamic search results filtering that you find on sites like Kayak.com and Endless.com. From a usability perspective, it’s a giant leap forward in product and results filtering that genuinely puts the user in the driver seat allowing real-time filtering driven by color swatches, slider bars and other similarly intuitive controls. It’s truly “fine tuning” of results.

However, like a lot of things on the web that are cool, fun and dreamt up by innovative web developers, this AJAX-powered filtering method can wreak havoc on your site’s “search engine friendliness” and ultimately negatively impact your natural search rankings.

AJAX is not a technology that search engines understand yet. To understand how search engine crawlers experience AJAX features, you have to turn off your browser’s JavaScript and then attempt to use the site as you normally would. You’ll find that some sites simply display a message telling you that you must have JavaScript enabled in order to view the site. In web programming parlance, a site like this is said to “not fail gracefully.” Those sites that do fail gracefully still allow the user (and search engine crawlers) to navigate the site and find the information they’re looking for. If you turn off your JavaScript and try this out for yourself, you’ll notice that the user-experience is no longer fluid because the pages have to reload in order to display updated content. You’ll also notice that instead of remaining on the same URL, you are now being taken deeper and deeper into SEO hell as more and more parameters are tacked on to the URL in order to facilitate the filtering process.

Example of URL from Endless.com for Men’s  Adidas Lace-up Fashion Sneakers, size 10 in white.
 

The URL for the same product search performed with javascript turned off:


OUCH! (and I should mention that I was not able to get the same set of product results with javascript turned off).

It’s not just AJAX that causes these URL-based problems for search engine crawlers. Other common features that have similar undesirable side effects are:

• Session tracking with URL session IDs
• Site search utilities for searching content within a site
• URL Tracking parameters parameters for visitor analysis or affiliate tracking
• Printer-friendly page versions

Why is this so bad for SEO? 

At the most basic level, it’s creating a huge duplicate content problem that just multiplies with each new option that you throw into the mix. I’m not talking about the type of cross-domain duplicate content associated with stealing another site’s content that can get your site penalized. Not all duplicate content problems are so nefarious. Duplicate content within a site is not that uncommon, especially with blogs, and doesn’t result in any sort of penalty. 

The problem with duplicate content within a site is that it dilutes the strength of your site and very often creates problems with indexation. Think of a well-optimized, duplicate content-free site like a well-brewed beer. Adding duplicate content is like adding water—if you add enough you lose the taste of the beer. From a more technical perspective, what’s happening is that the search engines are finding multiple pages that are optimized for a given keyword, each with a different number of external links pointing to it. So, instead of having one strong page the search engines would favor, a site ends up with many weaker pages that have much less potential to rank.

There are other undesirable side effects from this duplicate content problem. One is that the wrong page from your site might end up ranking. A perfect example of this is the “printer-friendly” version of page, which is duplicate content of the “regular” page, typically has no navigation and might lack the formatting and call to action of the regular page.  You would not want a visitor landing on the printer friendly version of the page directly from the search results.

Another side effect is decreased crawl efficiency. Ideally, you want the search engines crawlers to come back to your site regularly and download fresh versions of every page.  By drastically increasing your site’s page count with superfluous pages, you just create more work for the search engines crawlers and reduce the chances that they are crawling the most important pages regularly.

How to Deal with Duplicate Content

There are numerous ways to deal with duplicate content. I’m going to divide them into two categories and detail the best time to use each of the methods.

Category One: Search Engine Tools (Canonicalization Approaches)

The first category of methods for dealing with duplicate content on your site is tools and conventions that the search engines provide. All of these solutions refer to a process called “canonicalization,” which, by definition, is a process for converting data that has more than one possible representation (i.e. your duplicate content) into a "standard" canonical representation (i.e. a single version). All of the methods in this section are prescribed by the search engines (predominantly Google) for dealing with duplicate content.

www. vs non-www URLs – this is largely a problem of the past, but search engines used to treat http://www.yourdomain.com/somepage.html and http://yourdomain.com/somepage.html as different pages because technically they are different fully-qualified URLs. Google provides a way of setting your preferred domain in Google Webmaster Tools. Login to you Webmaster Tools account and look under “Site Configuration > Settings” and you’ll see a section for “Preferred Domain.”  Simply choose the presentation you prefer.

Since Google is the only search engine to provide a setting for this in their webmaster control panel, I also always recommend setting up a global 301 redirect to your preferred domain. The methods for doing this are different depending on your hosting platform and the solutions can be easily found with a simple search.

Meta "Canonical" tag – this is a convention that all 3 of the major search engines announced they were supporting early in 2009. Think of this as a "suggestion" of a 301 redirect. This meta tag goes in the <head> portion of your webpage and takes this form:
<link rel="canonical" href="http://www.yourdomain.com/somepage.html" />

This is a great solution for those nasty filtering URLs where the filter parameters are continuously tacked on to the end of the URL. You should have a clean, base URL that represents the unfiltered product or service, such as http://www.yourdomain.com/theproduct.html. On each of the subsequent filter pages, as you filter the available product options by color, size, features, etc. use the canonical URL tag to designate the base product URL and put all of your optimization and link building efforts into that URL.

The search engines have stated that they may not always pay attention to this tag, and I suspect that the biggest factor that would cause them to ignore the tag is a significant number of internal or external links pointing to one of the nasty filter URLs or substantially different content between the pages (especially different title tags), so be aware of how people are linking to your pages and how your filtering system alters key on-page elements.

For great coverage of this topic see Rand's post at: http://www.seomoz.org/blog/canonical-url-tag-the-most-important-advancement-in-seo-practices-since-sitemaps

Dynamic Parameter Handling – Google just recently announced this feature addition to their Webmaster Tools, joining Yahoo!’s Site Explorer which has supported dynamic parameter handling for some time.

This webmaster console feature allows users to designate which URL parameters the search engines should ignore. A URL parameter is any variable=value pair. Again, this is a great feature for dealing with those ugly URLs that are built using variables, such as http://www.yourdomain.com/theproduct.html?color=blue&size=5.

Another URL parameter that is notorious for creating duplicate content is the sessionID. If any of the URLs on your site have “sessionid” in them, this dynamic parameter filtering is a must.

You can specify up to 15 parameters to ignore, so this feature is limited. Also, like the canonical URL tag, the search engines look at your settings for this feature as a recommendation, not a directive. 

See Google’s recent post about the addition of this feature: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=147959

XML Sitemaps – XML sitemaps allow webmasters to suggest URLs to the search engines. Like the canonical URL tag and the dynamic parameter filtering, your sitemaps will not become law with the search engines. The search engines’ primary means of understanding your site and discovering new pages is the actual crawl process, because this most closely matches how an actual user experiences your site.

You can read more about using sitemaps here: http://www.google.com/support/webmasters/bin/topic.py?topic=8476

Best Solution: Use Them All – Since each of these methods for dealing with duplicate content on your site is looked at by the search engines as a “suggestion,” you should use as many of them in combination as you can. There’s no reason not to set your preferred domain (www vs. non-www) in Google because it takes less than a minute. Coding the canonical URL tag into your site’s content management system could be a monumental task, so that might be something to put on the roadmap for future site developments, but the dynamical URL parameter filtering through the webmaster consoles at Google and Yahoo! is another pretty easy feature to setup. Setting up XML sitemaps might also be a monumental task, especially since software and web-based services designed to automate this process rely on crawling technology similar to the search engine crawlers to map the site site and create the sitemap. This method of sitemap generation would capture the ugly, parameter URLs as well and would therefore could require a lot of work to clean them up before submitting to the search engines. Many content management systems come with XML sitemap generation features or have add-on modules that accomplish the task. These will likely require some configuring to make sure that only the clean, base URLs are included.  If the process must be done manually, an easy solution is to start with the most important pages of your site like the main category pages, the best selling products, etc.

Category Two: Server and Coding Methods (Non-Canonicalization Approaches)

The tactics in this section for dealing with duplicate content do not rely on canonicalization, but rather provide methods for guiding search engines crawlers, in essence, taking them by the hand and leading them through the maze of your site. One of my biggest problems with the canonicalization prescriptions provided by the search engines is that they are only suggestions and beyond that, the search engines don’t provide any additional color around why they listen to these suggestions sometimes and not others. I like more direct approaches that I know are being respected. Search engine crawlers have a beautiful feature built in to them: subservience. There are many different ways to tame a bot.

Meta Robots Tag—“noindex,follow” – the meta robots tag is a tag created specifically for providing instructions to the search engine crawlers at the page level.  Of the different commands that can be given to a crawler using this tag, the one most pertinent to this discussion on duplicate content is “noindex, follow” which takes this form:
<meta name=”robots” content=”noindex,follow” />

This command tells the search engines “don’t waste time downloading this page into your index, but please, follow all of the links on the page.” This can be a great solution for those filter pages with the nasty URLs that don’t introduce any new content with each additional filter parameter. It’s also well suited for the search results pages from your site’s search feature where the results pages themselves don’t provide any value and are constantly changing, but the links on the page are important to follow.

This tactic prevents the crawl inefficiencies by directing the search engines to not download pages and it prevents duplicate content from getting included in their index while still allowing an open door for the crawlers to freely crawl the site.

301 Redirects – There are times when you come across duplicate content on your site that just doesn’t make sense - a page on the main site duplicated in an archive or a page leftover from a redesign that got included somewhere else. Whatever the reason, when you find duplicate content that just plain isn’t necessary, you should use a server 301 redirect from one of the pages to the other. These pages may be splitting link juice (there could be links from external sites pointing to both pages, dividing the value each page should be getting) or giving to the search engines confusing signals. You should look at analytics or your server log files to see which of these pages is getting more referral traffic, because that is likely the page with the most back links and the one that should remain active. Sometimes the layout or design of the page will dictate which should stay regardless of link data. Either way, the 301 redirect will help consolidate the value of these duplicate pages into one page.

Robots.txt File – The robots.txt file, found at http://www.yourdomain.com/robots.txt is the first thing that any crawler is supposed to check when accessing your site. This is your first line of defense against the crawlers and can be used to provide directives about which pages or directories to avoid crawling. If your site is setup properly, the robots.txt file can be a good option for preventing the search engine crawlers from crawling and indexing endless URLs built from filter parameters or your site’s search results pages. The way to configure your site to take advantage of this approach is to have the filter feature and site search functionality tied to one directory or page and use that directory/page only for the filtering or search results content (i.e. don’t put any unique or important content in there that you want crawled). 

For instance, your main product page for widget X might be http://www.yourdomain.com/widget-x.php From this page, visitors might want a blue widget x with a wooden handle and pink fur. Choosing to filter this, or any product, should send the user to a special filter page or directory, such as http://www.yourdomain.com/filter/widget-x.php?color=blue&handle=wooden&fur=pink. 

In your robots.txt file, you would include a directive to disallow the /filter/ directory:

User-Agent: *
Disallow: /filter/

With this directive in place in your robots.txt file you can go crazy with your filtering as long as it all happens in the /filter/ directory. The search engines will avoid crawling this directory and keep them focused on the important, optimized pages of your site.

Be aware when implementing this method that you will get “errors” in your Google Webmaster Console report for “Pages Blocked by robots.txt.” I’ve never agreed with Google including this report under the “Crawl Errors” section because the robots.txt file is a legitimate means for controlling spider access to your site. The purpose of that report in GWT is to point out the pages that are being blocked in case you weren’t aware. You are likely to get a rather large number of these “errors” in GWT if you have had the pages live and crawlable for a while and are now blocking them after the fact. This is because Google has them in their index and will try accessing them directly versus finding them through a crawl. If you launch the pages and have them blocked from the start, you should experience relatively few error messages.

URL Rewriting – This option definitely requires some significant technical and programming know-how and should only be considered by advanced developers. URL rewriting is a server-side method for “prettying up” ugly URLs. Server-side means that it is performed by some method on the server and thus happens before the page is sent to the user or search engine crawler. 

URL rewriting mostly involves writing rules that your server uses to interpret and translate URL parameters. For you instance, you might have a rule that says “turn all instances of color=blue into /blue/ ” It’s kind of like the URL parameter handling feature the search engines offer, but instead of telling the server to ignore the parameters, it tells the server to translate them into something more friendly and readable. 

Personally, I find this approach to be MUCH easier on servers using Apache, which relies on the .htacess file for holding the translation commands. On servers using Windows running IIS, I’ve found ISAAPI Rewrite to be a decent alternative, but there is very little support for the product and its coding conventions in the wider developer community.

Conclusion
Conclusion

See how annoying and unproductive duplicate content can be. Keep in mind that we’ve just been looking at the problem of duplicate content on your site and not duplicate content from other people copying your site. Duplicate content within your own site can dilute the value of your content and hinder search engine rankings, not to mention that it can make for a horrible user experience.

This post laid out a number of different methods for handling the problem. Some were methods provided or supported by the search engines, while others were webmaster solutions with roots in server and bot management from a time long before Webmaster Tools were available from the search engines.

My personal opinion is that the methods provided by the search engines through webmaster consoles are more for the lay-people who don’t have system administrators or hardcore coders on their staff. When possible, I prefer the server and coding solutions, mainly because I’m old school and consider these solutions to be more definite, but that doesn’t mean I wouldn’t use or advocate for any of the solutions outlined above. Each has its own “right time” to be used.

Jason Cooper
Director, Value Added Services