Handling Parameters to Prevent Duplicate Content

One of the challenges SEO managers face is to minimize the amount duplicate content on their sites.  Duplicate content can be problematic of the following reasons:

• PageRank Dilution: as the backlinks to your pages will be pointing to various version of your pages, rather than the canonical version of the pages getting all the benefits, your PageRank will be divided between various pages.
• Crawl Efficiency: search engines rather than focusing on crawling your unique content, they end up crawling the same content available on multiple URLs.  The larger your site is, the more sever the issue will be.

One of the causes of duplicate content is parameters in the URLs.  The parameters are added to the URLs for various reasons such as:

• Tracking purposes
• Sorting SRPs
• Page Layout
• Filtering search results (e.g., by price)
• Session Ids
• Product Ids
• Affiliate Ids

URL parameters usually fall into two categories: 1) parameters that do not change page content 2) parameters that change the content page.

Both Google and Bing Webmasters tools offer the option to handle your parameters to minimize duplicate content problems on your site. This is a different action from creating a robots.txt file, where you can also include specific instructions on how to handle parameters.

Google provides webmasters with a list of the parameters they’ve found in the URLs on their sites and provide them with suggestions on how to handle the parameters.  You can confirm or reject those suggestions and can add parameters that aren’t listed.  In Google Webmaster Tools, you can ask Google to ignore up to 15 parameters.  Please note that the tool is case sensitive.

Ignoring Parameters in Google Webmaster Tools

Interesting enough, even if you ask Google to ignore some parameters, they do not always honour your request – some times for good reasons!  You need to be careful which parameters you ignore as you could potentially ask Google to ignore important parameters that, if dropped from the index, could wipe out large number of your URLs from their index.

Recently Google made the interface more user-friendly to minimize the chances of making mistakes while providing Google with more details.  You can now choose from one of the following options:

• Let Googlegot decide – to be used when you are not sure how the parameter behaves
• Every URL – to help Google determine if a URL is unique
• Only URLs with value=x – to instruct Google which version of URLs to crawl
• No URLs – to explicitly ask Google not to crawl certain URLs

You can also canonicalize the URLs by adding the rel=”canonical”element to the source of your preferred URL.

Bing Webmaster Tools also offers a similar URL Normalization tool to help Bing manage crawl more efficiently and avoid unnecessary site bandwidth usage by Bing brawler.

Parameter names cannot exceed 100 characters and maximum number of parameters that can be ignored in Bing is 20. Like Google, Bing takes these settings as hints and does not guarantee to ignore the parameters.

Adding a Parameter to Bing URL Normalization Tool

Filed Under: Search Engine Optimization • SEO