SEO Information

How to Prevent Duplicate Content with Effective Use of the Robots.txt and Robots Meta Tag


Duplicate content is one of the problems that we regularly come across as part of the search engine optimization services we offer. If the search engines determine your site contains similar content, this may result in penalties and even exclusion from the search engines. Fortunately it's a problem that is easily rectified.

Your primary weapon of choice against duplicate content can be found within "The Robot Exclusion Protocol" which has now been adopted by all the major search engines.

There are two ways to control how the search engine spiders index your site.

1. The Robot Exclusion File or "robots.txt" and

2. The Robots < Meta > Tag

The Robots Exclusion File (Robots.txt)
This is a simple text file that can be created in Notepad. Once created you must upload the file into the root directory of your website e.g. www.yourwebsite.com/robots.txt. Before a search engine spider indexes your website they look for this file which tells them exactly how to index your site's content.

The use of the robots.txt file is most suited to static html sites or for excluding certain files in dynamic sites. If the majority of your site is dynamically created then consider using the Robots Tag.

Creating your robots.txt file

Example 1 Scenario
If you wanted to make the .txt file applicable to all search engine spiders and make the entire site available for indexing. The robots.txt file would look like this:

User-agent: *
Disallow:

Explanation
The use of the asterisk with the "User-agent" means this robots.txt file applies to all search engine spiders. By leaving the "Disallow" blank all parts of the site are suitable for indexing.

Example 2 Scenario
If you wanted to make the .txt file applicable to all search engine spiders and to stop the spiders from indexing the faq, cgi-bin the images directories and a specific page called faqs.html contained within the root directory, the robots.txt file would look like this:

User-agent: *
Disallow: /faq/
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /faqs.html

Explanation
The use of the asterisk with the "User-agent" means this robots.txt file applies to all search engine spiders. Preventing access to the directories is achieved by naming them, and the specific page is referenced directly. The named files & directories will now not be indexed by any search engine spiders.

Example 3 Scenario
If you wanted to make the .txt file applicable to the Google spider, googlebot and stop it from indexing the faq, cgi-bin, images directories and a specific html page called faqs.html contained within the root directory, the robots.txt file would look like this:

User-agent: googlebot
Disallow: /faq/
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /faqs.html

Explanation

By naming the particular search spider in the "User-agent" you prevent it from indexing the content you specify. Preventing access to the directories is achieved by simply naming them, and the specific page is referenced directly. The named files & directories will not be indexed by Google.

That's all there is to it!

As mentioned earlier the robots.txt file can be difficult to implement in the case of dynamic sites and in this case it's probably necessary to use a combination of the robots.txt and the robots tag.

The Robots Tag
This alternative way of telling the search engines what to do with site content appears in the section of a web page. A simple example would be as follows;

In this example we are telling all search engines not to index the page or to follow any of the links contained within the page.

In this second example I don't want Google to cache the page, because the site contains time sensitive information. This can be achieved simply by adding the "noarchive" directive.

What could be simpler!

Although there are other ways of preventing duplicate content from appearing in the Search Engines this is the simplest to implement and all websites should operate either a robots.txt file and or a Robot tag combination.

Should you require further information about our search engine marketing or optimization services please visit us at http://www.e-prominence.co.uk - The search marketing company


MORE RESOURCES:
This RSS feed URL is deprecated, please update. New URLs can be found in the footers at https://news.google.com/news


Forbes

15 Common SEO Mistakes To Avoid
Forbes
Understanding SEO is the ultimate marketing discipline, especially in today's highly digital world. If you want to drive traffic to your site and increase sales, you have to keep up with the ever-changing search engine algorithms. Of course, in the ...
The benefits of using SEO for your small businessAugusta Free Press
Hiring The Right North London SEO CompanyThe Frisky

all 8 news articles »


Forbes

SEO Is A Must For Small Businesses: Here's How To Get Started
Forbes
If this means you have to take a crash course on search engine optimization (SEO), so be it. Another option is to create your website and then solicit SEO services from a reputable company. Although you may not have a large budget as a small business ...
Move Past Beginner's Level Link-Building Practices to Improve SEOMarTech Advisor
SEO Ranking - What Factors You Need to Focus on for Your SitePerformanceIN

all 6 news articles »


Forbes

Three Simple Steps To Seasonal SEO Success
Forbes
'Tis the season for seasonal SEO. Now you're asking: “Wait, seasonal SEO. Is that even a thing? I thought SEO was a long-term strategy.” Yes, seasonal SEO is surely a thing -- a big thing -- that should work hand-in-hand with your long-term strategy.



Search Engine Land

5 ways ignoring SEO could affect your bottom line
Search Engine Land
There's a common misconception that the financial benefits of SEO are not as clear as say, social media or PPC. But anyone familiar with SEO knows that it is highly measurable and in most cases even better value for money than both social media and PPC.

and more »


Business 2 Community

How Much Should You Budget for SEO?
Business 2 Community
SEO is only one part of digital marketing. Knowing how much of your budget to dedicate to SEO lies in understanding what your goals are, and your current progress toward those goals. Answering these questions can help you determine how much of the ...



Forbes

Rich Snippets: A Valuable And Underutilized SEO Strategy
Forbes
In reviewing your latest digital marketing strategy, perhaps you feel like you've dotted all your i's and crossed all your t's, yet you're still not ranking as high as you'd like. You've optimized the title tags and meta descriptions and targeted all ...



CMSWire

Why SEO Efforts So Often Fall Short
CMSWire
The inability to get people to implement SEO recommendations is the most common SEO fail of them all. I know, this sound like a no-brainer — but it's one of the biggest issues that we deal with as an industry. Often SEOs can explain perfectly why ...



Vertical Measures (blog)

The Weekly Measure: Talk Triggers, Leaving Facebook & Small Business SEO
Vertical Measures (blog)
Every Friday, Vertical Measures gathers all the best content from across the web and compiles it into The Weekly Measure. We feature the entire spectrum of digital marketing, including content marketing, paid media, social media, search engine ...



CMSWire

How to Ensure Your SEO Plan Is Put Into Action
CMSWire
Businesses opt not to implement SEO recommendations for a variety of common reasons: lack of understanding, limited resources, bureaucracy and more. While this stonewalling will unfortunately sound all too familiar to SEO practitioners, there are ways ...



Forbes

The Truth About Launching An SEO Strategy
Forbes
I've never been a fan of the self-proclaimed "SEO guru." Search engine optimization (SEO) strategies can seem like such mythical solutions: Do XYZ for X amount of months and “they” will just come to your website. It sounds too good to be true. The fact ...


Google News

home | site map
© 2006