Efficient Frontier Insights

How Important is Click Through Rate in Google’s Quality Score Formula?

Recently, I wrote an "Analyze This" column for SearchEngineLand about the importance of click through rate in determining your quality score on Google. A link to the full article is below but the bottom line is this. If you are looking to improve your quality score, first seek to improve your CTRs. But don’t waste your time trying to get the highest score. Once you have the CTR piece of puzzle solved, you can concentrate on the other factors such as landing page quality, ad copy relevance and campaign structure.
http://searchengineland.com/how-important-is-click-through-rate-in-googles-quality-score-formula-27296

Siddharth Shah

The Importance of Being Honest...

Trust and honesty; it’s the basis of any good relationship, business or otherwise, and implies openness, communication and accountability. Although the engines operate an opaque bidding auction which can bring about uncertainty regarding bid price, trust between you and your agency is crucial.
 
If you are running paid search at scale it is likely you will hand over the running of an account to a third party, which is where things get complicated; but too many advertisers  don’t know what their agency is doing. This is where the advertiser needs to insist on full visibility. It is important to have full access to all your SE accounts, ownership of the SE accounts and data, detailed performance reports and access to the specialist search engine teams. This will help you learn to have confidence in your agency; inevitably there will be periods where you lose track, and that’s where you have to trust your agency to keep you up to date with projects that are being undertaken.

It is also important to understand working with an agency is not a black and white model. Using a hybrid of outsourcing technology and strategic services can work well for advertisers with large and complex campaigns. Ask yourself, who is best to control tasks such as bid management, measurement, inventory, forecasting, ad copy, landing pages etc; can you realistically do it all yourself? Use pixel tracking to measure conversions, and use data feeds to send us information about rejections and cancellations - are you able to deliver this information to an agency, and are they able to take it? Clear communication and measurement is the key to a transparent relationship.

At Efficient Frontier we provide different levels of service to ensure that each client gets the correct level of support, from Full Service through to Technology Only - in fact our Technology Only clients have the same dashboard privileges that we do, meaning that they can steer our technology themselves (with guidance).

By following these guidelines, we aim to build on the trust that each client puts in us to run and optimise their search marketing campaigns.

Jimmy Felstead
UK Account Manager

UK Q3 Data Sees First Increase in Advertiser Confidence in Three Quarters

In addition to Monday's Efficient Frontier U.S. Search Engine Performance Report, we also released the Efficient Frontier UK Search Engine Performance report yesterday which takes a look at overall market trends from a fixed sample of our UK clients.
 
Performance on the whole is looking up for the UK and for the first time in three quarters we have seen an increase in advertiser confidence as advertisers from our UK Client Index start to spend more on Search Engine Marketing. In fact search spend was up 10% YoY and 11% QoQ which is a great sign for the imminent Christmas retail period.  
 
We expect that advertisers will jump at the chance to gain marketshare in the lead up to Christmas by capitalizing on lower CPCs and reduced competition. Incidentally, CPCs are starting to stabilise again after the yearlong tumble which should be music to the ears of the Search Engines. 
 
In the UK, we are continuing to see a strong demand for search and the news that was released by the IAB/ PwC (30 Sept 09) that online advertising has now bypassed TV to become the UK's single biggest advertising medium is an enormous symbol of confidence in the sector. We will have to wait and see whether online advertising continues to garner more media budget than TV when the economy improves. 
 
You can download our UK Search Engine Performance Report here.
 
Merinda Peppard
Global Marketing Manager

Q3 Data Provide Optimistic Signals for a Positive Q4

The last week at Efficient Frontier has been particularly interesting as we prepared and just released our Q3 2009 US Search Engine Performance Report. The U.S. Q3 data tells a variety of stories depending on the lens you apply. At the end of day, we believe search is on an upswing and, assuming our economy continues to stabilize, Q4 will produce growth that exceeds normal seasonal expansion. Let's take a quick look at how we arrived at our optimistic Q4 outlook.

On a Year on Year (YoY) basis, ad spending across our customer index fell by 5%. In contrast, and importantly for future trends, quarter on quarter revenue grew by 5%. So what is the net effect of 5% down YoY and 5% up QoQ positive? 
 


First, Q409 will be the first quarter where YoY comparisons will turn favorable. Q308 felt some of the effects of our economic turmoil. But remember, the Lehman Brothers failure that sparked the flames of decline hit in mid-September of 08. 
 
Next, a deeper look at trends by sector unveil an uneven pattern for spend. Travel continues to decline due to a number of factors including lower consumer demand and booking fee revenue. In contrast, spending in the autos category perked up as the Cash for Clunkers brought both consumer and advertiser demand to search. And most critical for the fourth quarter, retail showed positive spend growth QoQ and YoY as consumer search volume continues to expand in that sector.
 
Finally, advertiser sentiment is improving. The savvy marketers who make up our client base at Efficient Frontier see opportunity in the improved efficiency of search. Lower competition has helped each engine drive better ROI for our advertiser base. For example, Google is delivered 47% more ROI Q309 vs. Q308. Metrics may be loosening up with economic stability giving advertisers the power to capitalize on efficient sales volume in search.

We'll dive into more details in blog posts to come. And we welcome discussion on Q3. Are others seeing similar trends? How are advertisers approaching Q4? Leave a comment and join the conversation.

Justin Merickel
Vice President, Marketing and New Product Development

How to Deal with Duplicate Content Within a Site

One of my favorite developments in recent years is the AJAX-driven dynamic search results filtering that you find on sites like Kayak.com and Endless.com. From a usability perspective, it’s a giant leap forward in product and results filtering that genuinely puts the user in the driver seat allowing real-time filtering driven by color swatches, slider bars and other similarly intuitive controls. It’s truly “fine tuning” of results.

However, like a lot of things on the web that are cool, fun and dreamt up by innovative web developers, this AJAX-powered filtering method can wreak havoc on your site’s “search engine friendliness” and ultimately negatively impact your natural search rankings.

AJAX is not a technology that search engines understand yet. To understand how search engine crawlers experience AJAX features, you have to turn off your browser’s JavaScript and then attempt to use the site as you normally would. You’ll find that some sites simply display a message telling you that you must have JavaScript enabled in order to view the site. In web programming parlance, a site like this is said to “not fail gracefully.” Those sites that do fail gracefully still allow the user (and search engine crawlers) to navigate the site and find the information they’re looking for. If you turn off your JavaScript and try this out for yourself, you’ll notice that the user-experience is no longer fluid because the pages have to reload in order to display updated content. You’ll also notice that instead of remaining on the same URL, you are now being taken deeper and deeper into SEO hell as more and more parameters are tacked on to the URL in order to facilitate the filtering process.

Example of URL from Endless.com for Men’s  Adidas Lace-up Fashion Sneakers, size 10 in white.
 

The URL for the same product search performed with javascript turned off:


OUCH! (and I should mention that I was not able to get the same set of product results with javascript turned off).

It’s not just AJAX that causes these URL-based problems for search engine crawlers. Other common features that have similar undesirable side effects are:

• Session tracking with URL session IDs
• Site search utilities for searching content within a site
• URL Tracking parameters parameters for visitor analysis or affiliate tracking
• Printer-friendly page versions

Why is this so bad for SEO? 

At the most basic level, it’s creating a huge duplicate content problem that just multiplies with each new option that you throw into the mix. I’m not talking about the type of cross-domain duplicate content associated with stealing another site’s content that can get your site penalized. Not all duplicate content problems are so nefarious. Duplicate content within a site is not that uncommon, especially with blogs, and doesn’t result in any sort of penalty. 

The problem with duplicate content within a site is that it dilutes the strength of your site and very often creates problems with indexation. Think of a well-optimized, duplicate content-free site like a well-brewed beer. Adding duplicate content is like adding water—if you add enough you lose the taste of the beer. From a more technical perspective, what’s happening is that the search engines are finding multiple pages that are optimized for a given keyword, each with a different number of external links pointing to it. So, instead of having one strong page the search engines would favor, a site ends up with many weaker pages that have much less potential to rank.

There are other undesirable side effects from this duplicate content problem. One is that the wrong page from your site might end up ranking. A perfect example of this is the “printer-friendly” version of page, which is duplicate content of the “regular” page, typically has no navigation and might lack the formatting and call to action of the regular page.  You would not want a visitor landing on the printer friendly version of the page directly from the search results.

Another side effect is decreased crawl efficiency. Ideally, you want the search engines crawlers to come back to your site regularly and download fresh versions of every page.  By drastically increasing your site’s page count with superfluous pages, you just create more work for the search engines crawlers and reduce the chances that they are crawling the most important pages regularly.

How to Deal with Duplicate Content

There are numerous ways to deal with duplicate content. I’m going to divide them into two categories and detail the best time to use each of the methods.

Category One: Search Engine Tools (Canonicalization Approaches)

The first category of methods for dealing with duplicate content on your site is tools and conventions that the search engines provide. All of these solutions refer to a process called “canonicalization,” which, by definition, is a process for converting data that has more than one possible representation (i.e. your duplicate content) into a "standard" canonical representation (i.e. a single version). All of the methods in this section are prescribed by the search engines (predominantly Google) for dealing with duplicate content.

www. vs non-www URLs – this is largely a problem of the past, but search engines used to treat http://www.yourdomain.com/somepage.html and http://yourdomain.com/somepage.html as different pages because technically they are different fully-qualified URLs. Google provides a way of setting your preferred domain in Google Webmaster Tools. Login to you Webmaster Tools account and look under “Site Configuration > Settings” and you’ll see a section for “Preferred Domain.”  Simply choose the presentation you prefer.

Since Google is the only search engine to provide a setting for this in their webmaster control panel, I also always recommend setting up a global 301 redirect to your preferred domain. The methods for doing this are different depending on your hosting platform and the solutions can be easily found with a simple search.

Meta "Canonical" tag – this is a convention that all 3 of the major search engines announced they were supporting early in 2009. Think of this as a "suggestion" of a 301 redirect. This meta tag goes in the <head> portion of your webpage and takes this form:
<link rel="canonical" href="http://www.yourdomain.com/somepage.html" />

This is a great solution for those nasty filtering URLs where the filter parameters are continuously tacked on to the end of the URL. You should have a clean, base URL that represents the unfiltered product or service, such as http://www.yourdomain.com/theproduct.html. On each of the subsequent filter pages, as you filter the available product options by color, size, features, etc. use the canonical URL tag to designate the base product URL and put all of your optimization and link building efforts into that URL.

The search engines have stated that they may not always pay attention to this tag, and I suspect that the biggest factor that would cause them to ignore the tag is a significant number of internal or external links pointing to one of the nasty filter URLs or substantially different content between the pages (especially different title tags), so be aware of how people are linking to your pages and how your filtering system alters key on-page elements.

For great coverage of this topic see Rand's post at: http://www.seomoz.org/blog/canonical-url-tag-the-most-important-advancement-in-seo-practices-since-sitemaps

Dynamic Parameter Handling – Google just recently announced this feature addition to their Webmaster Tools, joining Yahoo!’s Site Explorer which has supported dynamic parameter handling for some time.

This webmaster console feature allows users to designate which URL parameters the search engines should ignore. A URL parameter is any variable=value pair. Again, this is a great feature for dealing with those ugly URLs that are built using variables, such as http://www.yourdomain.com/theproduct.html?color=blue&size=5.

Another URL parameter that is notorious for creating duplicate content is the sessionID. If any of the URLs on your site have “sessionid” in them, this dynamic parameter filtering is a must.

You can specify up to 15 parameters to ignore, so this feature is limited. Also, like the canonical URL tag, the search engines look at your settings for this feature as a recommendation, not a directive. 

See Google’s recent post about the addition of this feature: http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=147959

XML Sitemaps – XML sitemaps allow webmasters to suggest URLs to the search engines. Like the canonical URL tag and the dynamic parameter filtering, your sitemaps will not become law with the search engines. The search engines’ primary means of understanding your site and discovering new pages is the actual crawl process, because this most closely matches how an actual user experiences your site.

You can read more about using sitemaps here: http://www.google.com/support/webmasters/bin/topic.py?topic=8476

Best Solution: Use Them All – Since each of these methods for dealing with duplicate content on your site is looked at by the search engines as a “suggestion,” you should use as many of them in combination as you can. There’s no reason not to set your preferred domain (www vs. non-www) in Google because it takes less than a minute. Coding the canonical URL tag into your site’s content management system could be a monumental task, so that might be something to put on the roadmap for future site developments, but the dynamical URL parameter filtering through the webmaster consoles at Google and Yahoo! is another pretty easy feature to setup. Setting up XML sitemaps might also be a monumental task, especially since software and web-based services designed to automate this process rely on crawling technology similar to the search engine crawlers to map the site site and create the sitemap. This method of sitemap generation would capture the ugly, parameter URLs as well and would therefore could require a lot of work to clean them up before submitting to the search engines. Many content management systems come with XML sitemap generation features or have add-on modules that accomplish the task. These will likely require some configuring to make sure that only the clean, base URLs are included.  If the process must be done manually, an easy solution is to start with the most important pages of your site like the main category pages, the best selling products, etc.

Category Two: Server and Coding Methods (Non-Canonicalization Approaches)

The tactics in this section for dealing with duplicate content do not rely on canonicalization, but rather provide methods for guiding search engines crawlers, in essence, taking them by the hand and leading them through the maze of your site. One of my biggest problems with the canonicalization prescriptions provided by the search engines is that they are only suggestions and beyond that, the search engines don’t provide any additional color around why they listen to these suggestions sometimes and not others. I like more direct approaches that I know are being respected. Search engine crawlers have a beautiful feature built in to them: subservience. There are many different ways to tame a bot.

Meta Robots Tag—“noindex,follow” – the meta robots tag is a tag created specifically for providing instructions to the search engine crawlers at the page level.  Of the different commands that can be given to a crawler using this tag, the one most pertinent to this discussion on duplicate content is “noindex, follow” which takes this form:
<meta name=”robots” content=”noindex,follow” />

This command tells the search engines “don’t waste time downloading this page into your index, but please, follow all of the links on the page.” This can be a great solution for those filter pages with the nasty URLs that don’t introduce any new content with each additional filter parameter. It’s also well suited for the search results pages from your site’s search feature where the results pages themselves don’t provide any value and are constantly changing, but the links on the page are important to follow.

This tactic prevents the crawl inefficiencies by directing the search engines to not download pages and it prevents duplicate content from getting included in their index while still allowing an open door for the crawlers to freely crawl the site.

301 Redirects – There are times when you come across duplicate content on your site that just doesn’t make sense - a page on the main site duplicated in an archive or a page leftover from a redesign that got included somewhere else. Whatever the reason, when you find duplicate content that just plain isn’t necessary, you should use a server 301 redirect from one of the pages to the other. These pages may be splitting link juice (there could be links from external sites pointing to both pages, dividing the value each page should be getting) or giving to the search engines confusing signals. You should look at analytics or your server log files to see which of these pages is getting more referral traffic, because that is likely the page with the most back links and the one that should remain active. Sometimes the layout or design of the page will dictate which should stay regardless of link data. Either way, the 301 redirect will help consolidate the value of these duplicate pages into one page.

Robots.txt File – The robots.txt file, found at http://www.yourdomain.com/robots.txt is the first thing that any crawler is supposed to check when accessing your site. This is your first line of defense against the crawlers and can be used to provide directives about which pages or directories to avoid crawling. If your site is setup properly, the robots.txt file can be a good option for preventing the search engine crawlers from crawling and indexing endless URLs built from filter parameters or your site’s search results pages. The way to configure your site to take advantage of this approach is to have the filter feature and site search functionality tied to one directory or page and use that directory/page only for the filtering or search results content (i.e. don’t put any unique or important content in there that you want crawled). 

For instance, your main product page for widget X might be http://www.yourdomain.com/widget-x.php From this page, visitors might want a blue widget x with a wooden handle and pink fur. Choosing to filter this, or any product, should send the user to a special filter page or directory, such as http://www.yourdomain.com/filter/widget-x.php?color=blue&handle=wooden&fur=pink. 

In your robots.txt file, you would include a directive to disallow the /filter/ directory:

User-Agent: *
Disallow: /filter/

With this directive in place in your robots.txt file you can go crazy with your filtering as long as it all happens in the /filter/ directory. The search engines will avoid crawling this directory and keep them focused on the important, optimized pages of your site.

Be aware when implementing this method that you will get “errors” in your Google Webmaster Console report for “Pages Blocked by robots.txt.” I’ve never agreed with Google including this report under the “Crawl Errors” section because the robots.txt file is a legitimate means for controlling spider access to your site. The purpose of that report in GWT is to point out the pages that are being blocked in case you weren’t aware. You are likely to get a rather large number of these “errors” in GWT if you have had the pages live and crawlable for a while and are now blocking them after the fact. This is because Google has them in their index and will try accessing them directly versus finding them through a crawl. If you launch the pages and have them blocked from the start, you should experience relatively few error messages.

URL Rewriting – This option definitely requires some significant technical and programming know-how and should only be considered by advanced developers. URL rewriting is a server-side method for “prettying up” ugly URLs. Server-side means that it is performed by some method on the server and thus happens before the page is sent to the user or search engine crawler. 

URL rewriting mostly involves writing rules that your server uses to interpret and translate URL parameters. For you instance, you might have a rule that says “turn all instances of color=blue into /blue/ ” It’s kind of like the URL parameter handling feature the search engines offer, but instead of telling the server to ignore the parameters, it tells the server to translate them into something more friendly and readable. 

Personally, I find this approach to be MUCH easier on servers using Apache, which relies on the .htacess file for holding the translation commands. On servers using Windows running IIS, I’ve found ISAAPI Rewrite to be a decent alternative, but there is very little support for the product and its coding conventions in the wider developer community.

Conclusion
Conclusion

See how annoying and unproductive duplicate content can be. Keep in mind that we’ve just been looking at the problem of duplicate content on your site and not duplicate content from other people copying your site. Duplicate content within your own site can dilute the value of your content and hinder search engine rankings, not to mention that it can make for a horrible user experience.

This post laid out a number of different methods for handling the problem. Some were methods provided or supported by the search engines, while others were webmaster solutions with roots in server and bot management from a time long before Webmaster Tools were available from the search engines.

My personal opinion is that the methods provided by the search engines through webmaster consoles are more for the lay-people who don’t have system administrators or hardcore coders on their staff. When possible, I prefer the server and coding solutions, mainly because I’m old school and consider these solutions to be more definite, but that doesn’t mean I wouldn’t use or advocate for any of the solutions outlined above. Each has its own “right time” to be used.

Jason Cooper
Director, Value Added Services

Should you bid on Brand Terms?

Some advertisers believe that bidding on their own brand terms is foolish and translates to money down the drain. The rationale being that the user has decided to convert with the brand he searched for and it would make more sense if he did it by clicking on the natural listings. Brand terms almost always appear at the top of the natural listings. So why waste valuable budget on PPC?

But how much budget are you actually saving by cutting out brand terms? And what about customer fickleness?

By not bidding on your brand, you have a whole new set of issues to deal with :

1.    Affiliates: Affiliates love bidding on brand terms as this mean lots of commission for almost no hard work. And this means that the advertiser ends up paying a lot more for his brand than he would, by simply bidding on it. Affiliates can be difficult to control and often stealthy. They might bid out of office hours and geo target very cleverly not to get caught out. These are just some of the known tactics. Bidding on your brand can mitigate the effects  of "direct-to-site" affiliates.

2.    Google’s broad match algorithm: Bidding on your brand terms reduces the possibility of the more expensive generic ad appearing for brand + generic searches as a result of Google’s broad match algorithm.

3.    Time Sensitive Promotions: Paid search ads will allow you to tailor your messaging which you cannot do so easily on SEO.  Use SEO for broad company information and messages that will survive for a prolonged period  and use paid search ads for opportunities to promote key tactical campaign offers including time sensitive information, pricing and calls to action.

4.    Competitors: Competitors will not miss out an opportunity to bid on your brand and steal your traffic! Although you might have a customer searching for your brand, a hard-to-resist offer in a competitor’s ad can change the customer’s mind at the last minute. Make sure you protect your brand by bidding on it.

5.    PR: Due to the nature of social media, the possibility of negative stories and reviews rising up through natural search is very high. Bidding on your brand has the added bonus of pushing these stories down the page and hopefully below the fold.

6.    Universal Search: The increasing instance of universal search, such as map results or images, is also pushing natural search listings down the page. To ensure your brand is still visible it is important to bid on your brand.

7.    Last but not least: CTR should be so good on your own brand that what you pay should be minimal.

To conclude, an advertiser should consider very carefully whether they have the time and the resources to monitor their brand 24/7 before they stop bidding on their brand terms. There is also significant research that using PPC and SEO together can actually increase CTR, conversion rate and total revenues.

Rekha Patil, UK Account Manager