SEO Information

How to Prevent Duplicate Content with Effective Use of the Robots.txt and Robots Meta Tag


Duplicate content is one of the problems that we regularly come across as part of the search engine optimization services we offer. If the search engines determine your site contains similar content, this may result in penalties and even exclusion from the search engines. Fortunately it's a problem that is easily rectified.

Your primary weapon of choice against duplicate content can be found within "The Robot Exclusion Protocol" which has now been adopted by all the major search engines.

There are two ways to control how the search engine spiders index your site.

1. The Robot Exclusion File or "robots.txt" and

2. The Robots < Meta > Tag

The Robots Exclusion File (Robots.txt)
This is a simple text file that can be created in Notepad. Once created you must upload the file into the root directory of your website e.g. www.yourwebsite.com/robots.txt. Before a search engine spider indexes your website they look for this file which tells them exactly how to index your site's content.

The use of the robots.txt file is most suited to static html sites or for excluding certain files in dynamic sites. If the majority of your site is dynamically created then consider using the Robots Tag.

Creating your robots.txt file

Example 1 Scenario
If you wanted to make the .txt file applicable to all search engine spiders and make the entire site available for indexing. The robots.txt file would look like this:

User-agent: *
Disallow:

Explanation
The use of the asterisk with the "User-agent" means this robots.txt file applies to all search engine spiders. By leaving the "Disallow" blank all parts of the site are suitable for indexing.

Example 2 Scenario
If you wanted to make the .txt file applicable to all search engine spiders and to stop the spiders from indexing the faq, cgi-bin the images directories and a specific page called faqs.html contained within the root directory, the robots.txt file would look like this:

User-agent: *
Disallow: /faq/
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /faqs.html

Explanation
The use of the asterisk with the "User-agent" means this robots.txt file applies to all search engine spiders. Preventing access to the directories is achieved by naming them, and the specific page is referenced directly. The named files & directories will now not be indexed by any search engine spiders.

Example 3 Scenario
If you wanted to make the .txt file applicable to the Google spider, googlebot and stop it from indexing the faq, cgi-bin, images directories and a specific html page called faqs.html contained within the root directory, the robots.txt file would look like this:

User-agent: googlebot
Disallow: /faq/
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /faqs.html

Explanation

By naming the particular search spider in the "User-agent" you prevent it from indexing the content you specify. Preventing access to the directories is achieved by simply naming them, and the specific page is referenced directly. The named files & directories will not be indexed by Google.

That's all there is to it!

As mentioned earlier the robots.txt file can be difficult to implement in the case of dynamic sites and in this case it's probably necessary to use a combination of the robots.txt and the robots tag.

The Robots Tag
This alternative way of telling the search engines what to do with site content appears in the section of a web page. A simple example would be as follows;

In this example we are telling all search engines not to index the page or to follow any of the links contained within the page.

In this second example I don't want Google to cache the page, because the site contains time sensitive information. This can be achieved simply by adding the "noarchive" directive.

What could be simpler!

Although there are other ways of preventing duplicate content from appearing in the Search Engines this is the simplest to implement and all websites should operate either a robots.txt file and or a Robot tag combination.

Should you require further information about our search engine marketing or optimization services please visit us at http://www.e-prominence.co.uk - The search marketing company


MORE RESOURCES:
What Is Dwell Time & Why It Matters for SEO  Search Engine Journal

Here's what you really need to know about dwell time, how search engines use it, how it impacts rankings, and how to influence it.


Service workers and SEO: What developers should know  Search Engine Land

Developers working on websites, as compared to native apps, were long at a disadvantage because they were limited to using web technologies like HTML, ...


5 practical data blending examples for smarter SEO insights  Search Engine Land

Sometimes we face questions that are hard to answer with the information from isolated tools. One powerful technique we can use is to combine data from ...


SEO survival in 2019: Words of wisdom from Moz’s Dr. Pete Meyers  Search Engine Land

Search engine optimization is a never-ending process that is frequently subjected to rule changes and playbook adjustments – many of which tend to come ...


SEO Maintenance Checklist: Crucial Daily, Monthly, Quarterly & Yearly Tasks  Search Engine Journal

Learn the specific daily, monthly, quarterly, and yearly activities to implement for success in this SEO maintenance checklist.


Publishing Thousands of URLs at Once – Is It Risky for SEO?  Search Engine Journal

Find out whether publishing thousands, or hundreds of thousands, of new pages at the same time could have any SEO risks or unintended consequences.


What Is Rank Transition & What Does It Mean for SEO?  Search Engine Journal

Discover what rank transition is according to Google's Ranking Documents patent and what its implications are for search engine optimization.


7 Best SEO Tools to Help You Rank Higher in Google  Entrepreneur

The key to building and sustaining organic traffic is knowing which are the best SEO tools to use. Today, there are so many SEO checkers on the market that it be ...


SEO: 9 Tips for Writing Great Content  Practical Ecommerce

Textual *content* enables search engines to understand your site. The words you choose should engage shoppers and boost your site's relevance to the ...


What is HTML, cache and SEO? What are cookies and how do algorithms work? Digital language explained  Evening Standard

Tech speak can give us all a headache sometimes. Every day in the digital age, it seems we have a new phrase to contend with that describes the intricate ...


Exploring the Role of Content Groups & Search Intent in SEO  Search Engine Journal

Here's what SEO professionals need to know about a Google patent "Optimized Web Domains Classification Based on Progressive Crawling with Clustering."


Stay on Top of Your Site's SEO Ranking for Less Than $30  Entrepreneur

Keeping up with the Kardashians is one thing, but keeping up with Google's ever-changing search engine ranking algorithm is a whole 'nother issue.


The 20-Minute Technical SEO Workweek  Search Engine Journal

Have you checked your technical SEO lately? Use this workflow to quickly get a pulse of your technical SEO in just 20 minutes a week.


SEO Tidbits From Google Fights Disinformation White Paper  Search Engine Roundtable

On Friday, Google published on their blog about fighting disinformation and posted a 30 page white paper. Truth is, there is nothing really new in this white ...


SEO Guide 2019: Your Strategy for This Year  Search Engine Journal

Are you ready to become up-to-date with the world of search engine optimization? Read WebCEO's guide to learn the top SEO tactics you need in 2019.


Updated for 2019! ‘Enterprise SEO Platforms: A Marketer’s Guide’  Search Engine Land

MarTech Today's “ Enterprise SEO Platforms: A Marketer's Guide” has been updated for 2019. Compiled from the latest research, this 61-page report is your ...


SEO for Real Estate Pros: How to Grow Your Digital Footprint - RealtyBizNews: Real Estate News  RealtyBizNews

Using SEO for real estate means using tools and techniques to grow your business in lots of ways. Here's what you need to know.


5 SEO Career Trends for 2019 [New Research]  Search Engine Journal

While the idea of SEO being dead is nothing more than worn out, industry fodder, SEO as a job title and role have been legitimately questioned. Not so long ago, ...


6 Ways SEO Tools Can Help You Run Your Business  Search Engine Journal

Tools make a marketer's life more productive. This post will tell you why and how SEO tools can help you better run your business.


Crawl-First SEO: A 12-Step Guide to Follow Before Crawling  Search Engine Journal

Crawl-first SEO focuses on two of the main parts of the search engine infrastructure: crawling and indexing. If all the pages on a site aren't crawled, they can't be ...


Angular Universal: What you need to know for SEO  Search Engine Land

If you use Angular to power your website, then you have an extra hurdle to cross when it comes to SEO. Fortunately, Angular Universal makes it easy to leap ...


Sharpen your SEO & SEM skills. See the SMX Advanced agenda!  Marketing Land

For nearly 15 years, SMX® Advanced has been the event expert search marketers attend to learn performance-enhancing tactics, make game-changing ...


New Study Shows How Much SEO Professionals Are Earning Per Year  Search Engine Journal

A recent study on the local search industry reveals how much the average SEO professional earns in a year, and what they bill per hour. Data is based on ...


Context in Outbound Links for High Ranking SEO  Search Engine Journal

Linking out to other sites is important. The sites you choose to link to can cause your site to lose rankings. This article discusses outbound links for SEO.


SEO: Free Keyword Research in 3 Steps  Practical Ecommerce

There are many expensive tools that claim to simplify and speed up the keyword research process. If you don't have thousands of dollars in your budget, ...


How to keep your SEO team (and yourself) relevant  Search Engine Land

Weekly rank tracking. Optimization meetings with the media, copy, UX and dev teams. Monthly analytics reporting with the client. And decks. So many decks.


How to Set & Achieve Realistic SEO Goals for 2019  Search Engine Journal

What milestones does your company need to reach in 2019? As marketers and SEO professionals, charting the path toward our goals is one of the exercises we ...


Tips for Selecting the Right SEO Provider for Your Accounting Firm  CPAPracticeAdvisor.com

Taking the first steps into the world of search engine optimization (SEO) can be intimidating. If you have spent anytime researching the topic online then you ...


SearchCap: Google updates, SEO education & A/B testing  Search Engine Land

Mueller previews Google image search changes, old Search Console sunsetting at NYC meetup. Jan 25, 2019 by Barry Schwartz. Here are the highlights from a ...


Vietnam coach Park Hang-seo confirms desire to give up coaching multiple teams  FOX Sports Asia

Vietnam coach Park Hang-seo claims that he still plans to concentrate on only one of the two national teams for the country.


What Is an SEO Professional in 2019? An Identity Crisis Averted  Search Engine Journal

Picture this: You're an SEO professional, and someone asks you what exactly it is that you do. What do you say? I'm a: Search analytics guru; *Content* strategist ...


SEO Evolution: What’s next for SEO?  Fourth Source

The year is almost at an end and with the dawn of a new one comes the prospect of Google's raft of SEO changes that the company is ushering for 2019.


SEO: Monitoring Traffic from Domain Mergers, with Google Data Studio  Practical Ecommerce

A common practice among retailers is creating microsites to target different audiences. The strategy can produce strong conversions because the user ...


Wix ‘SEO Wiz’ Stars in Big Super Bowl Ad, Fumbles with SEO Community  Search Engine Journal

Did you hear that sound last night during the Super Bowl? It was the collective SEO community hitting their heads against a proverbial desk. Wix's foray into the ...


MarTech Today Research: A marketer’s guide to enterprise SEO platforms  Search Engine Land

Enterprise SEO continues to grow more complex as the technology and algorithms powering search engines evolve. Comprehensive enterprise SEO platforms ...


How to Activate a Sponsorship to Achieve Content Marketing & SEO Goals  Search Engine Journal

Want high-quality coverage and links? Here's how to maximize sponsorships to help you to achieve your *content* marketing and SEO goals.


Is It Time To Outsource SEO? Here's What You Need To Consider  Forbes

Everyone and their mother is chasing after the holy grail of internet marketing: organic search traffic.


An SEO Guide to URL Parameter Handling  Search Engine Journal

URL parameters create duplicate *content*, waste crawl budget, and dilute ranking signals. Learn six ways to avoid potential SEO issues with URL parameters.


9 Things to Do When Your SEO Pro Quits  Practical Ecommerce

An ecommerce company's search-engine-optimization pro will eventually leave. Here's what to do next. First, ask her why. Make it clear that you want her ...


10 Ways to Think Like Googlebot & Boost Your Technical SEO  Search Engine Journal

Have you found yourself hitting a plateau with your organic growth? High-quality *content* and links will take you far in SEO, but you should never overlook the ...


SEO vs SEM – What’s the Difference and Why it Matters to Your Business  G2 Crowd

Learn the key difference between SEO and SEM and how they greatly increase the visibility of your site on search engines like Google, Yahoo, Bing, and more.


SearchCap: Quora keyword targets, Angular SEO & old stock SEO  Search Engine Land

Learn how to manage product unavailability without hurting your SEO Feb 1, 2019 by Max Cyrek. When your e-commerce site has out-of-stock products, here ...


How to Use Python to Analyze SEO Data: A Reference Guide  Search Engine Journal

Do you find yourself doing the same repetitive SEO tasks each day or facing challenges where there are not tools that can help you? If so, it might be time for you ...


60+ SEO Stats to Help You Rank Better in 2019 [Infographic]  Social Media Today

Are you looking to maximize your search performance in 2019? Need to know the stats and facts to help you understand the SEO process? The team from Red ...


Targeted SEO: Here’s how to grow a page from 200 visits per month to 30,000  Search Engine Land

Spoiler alert: SEO works. Strategic, targeted SEO can drive huge results for the right pages. Of course, the catch here is that you need the “right” pages to see the ...


SearchCap: Bing Maps API, Google and YouTube ads and targeted SEO  Search Engine Land

Now your Google text ads can show on YouTube search results, too. Jan 30, 2019 by Ginny Marvin. Google says early tests have shown similar performance ...


Vietnam coach Park Hang-seo begins planning for AFC U-23 Championship  FOX Sports Asia

South Korean Park Hang-seo joined as the head coach of the Vietnam national team back in 2017. His reign saw the Golden Dragons go on one of their more ...


Super Bowl Sunday SEO: The importance of search during the Big Game  ClickZ

SEO is crucial during the Super Bowl, both for $5 million-dollar advertisers and companies hoping to appear in local search results.


Selecting an SEO Company in 2019  The Realtime Report

Selecting an SEO Company in 2019. SEO is one of the fast-changing concepts in the world of technology. Each year (in the last five years), different concepts ...


Linkpushing, an Innovative SEO Service, Launches a New Process to Rank Websites - Press Release  Digital Journal

Linkpushing Recently Revamped its Services to Better Reflect the Search Engine Environment.


SEO Keywords Best Practices for 2019 (for the Win!)  Business 2 Community

Come the new year, you might be wondering if the keyword SEO whose rules you've been adhering to until now will still make sense with the frequent updates ...


Stirling 2019: Soon-Seok Seo seeks revenge  International Paralympic Committee

South Korea's Soon-Seok Seo wants to take care of some unfinished business from a year ago at the PyeongChang 2018 Paralympic Winter Games. His team ...


Battle of the SERP: Combining SEO & PPC Forces to Win  Business 2 Community

Legend has it there's a castle deep in the heart of the second page of Google. Guarding a castle is a monster named Bowser. Every so often, Bowser will creep ...


4 Link Building Tips for International SEO  Search Engine Journal

Discover four link building tips for global websites that can be easily implemented even by companies with limited resources.


The Amazon SEO game plan: SEO refinements that increase visibility  DigitalCommerce360

Effective SEO can help you stand above the noise on Amazon. Crafting the perfect keywords without misleading your customers and making your descriptions ...


SEO vs PPC: How to Choose What’s Best For Your Business  Business 2 Community

As a business owner are you often confused about choosing the right online marketing channel? Most businesses struggle with scanty resources. Marketing ...


SearchCap: Local SEO survey, Moz domain authority and GOOG earnings  Search Engine Land

5 takeaways from Google's Q4 2018 earnings for search marketers. Feb 5, 2019 by Ginny Marvin. Google executives discussed search experience, campaign, ...


Why SEO Should Be A Focus Of Any Content Strategy  Business 2 Community

As a small business owner, you are juggling too many balls to waste time on fruitless marketing efforts. Not only is it frustrating; it's a waste of money. Whether ...


[SEO Roundup] 30+ Experts share their top SEO strategies  MintTwist Blog

Link building? Keyword research? PageSpeed? There are plenty of aspects within SEO, which can make it difficult to prioritse your efforts. We asked 31 SEO ...


YouTube SEO: 5 Hacks for Quicker, Better YouTube Keyword Research  Business 2 Community

Proper YouTube keyword research is the backbone of a good YouTube SEO strategy. Why? Because finding the right YouTube keywords could mean the ...


Why Prioritize Local SEO?  Vertical Measures

Local SEO consists of many different local citation components, but is absolutely vital to the success of any organization aiming to reach to a nearby audience.


Loyalty Marketing: The Proven Complement to Your SEO Strategy  Search Engine Journal

Here's how loyalty marketing can complement your SEO strategy and why the two disciplines need to be conducted seamlessly.


'Very concerning': Senate watchdog hasn't finished an ethics probe since 2017  CBC News

The Senate Ethics Officer didn't publish a single inquiry report last year, prompting some senators to wonder exactly what the office (which has a $1.2 million ...


Top 10 mistakes companies are making with local SEO  ZDNet

Almost two out of five business leaders do not know if their website is optimised for local SEO. So, what should they look out for?


15 Local SEO Steps You Should Take Right Now  Business 2 Community

Local search plays an incredibly important part in how customers discover local businesses. An incredibly 82 percent of smartphone shoppers conduct “near ...


Rise of Mobile Users Alters SEO Tactics  Practical Ecommerce

The percentage of U.S. searchers clicking on advertisements in Google search results is growing. Likewise, the percentage of visitors that click nothing is also ...


Virginia SEO Agency Launches Tree Service Contractor Digital Marketing Services  Associated Press

Culpeper, Virginia based digital marketing agency Big Bamboo Marketing updated its digital marketing solutions to provide media relations and reputation ...


Sportswear brand ASICS selects Re:signal as European SEO agency partner  The Drum

Sportswear brand Asics has appointed Re:signal following a competitive pitch for its European SEO account.


The Mammoth SEO Checklist to Rank Any Blog [Infographic]  Social Media Today

Are you looking for ways to increase traffic to your blog? Want to know how to better optimize your on-site *content* in order to boost your Google rankings?


Google Search Console sending alerts for big ranking, traffic drops  Search Engine Land

Google is now sending alerts and notifications from Google Search Console when it detects a “substantial drop” in clicks to your website from the Google search ...


40 Major SEO Mistakes You Must Avoid in 2019 [Infographic]  Social Media Today

Are you struggling to get your website to rank on the first page of Google? Want to see if you're making any SEO mistakes that could be holding your website ...


How Artificial Intelligence & Machine Learning Can Affect Our Future?  Times of India

International Business News: Hi, I am TuhinBanik; born in North East India and raised in Kolkata. Throughout my childhood days, I had a keen interest in ...


4 Reasons Why Regular SEO Audits of Your Website Undeniable  Business 2 Community

When it comes to SEO, you cannot question if your online business needs it or not. SEO has become an undeniable force in the online marketing industry.


How to Optimize a Website for Local SEO  Business 2 Community

An astounding 46 percent of all Google searches are local. Is your website strategically optimized for local search? If not, you've come to the right place!


5 Best SEO Tips for 2019  Thrive Global

These are great SEO tips and tactics for building big traffic in 2019.


Yoast SEO to add “live indexing” feature for Bing  OnMSFT

Following Bing's announcement that they are hoping to change the way search engines discover new and updated *content*, one of WordPress's largest SEO ...


Responsive Web Design Questions About Mobile SEO  Business 2 Community

Users are bouncing, conversions are sparse and sales are way down. Or, *insert in your problem here.*. Whatever the concern, let's be honest… your site needs ...


6 Simple Local SEO Tactics Financial Marketers Can Use Today  The Financial Brand

Community banks and credit unions can turn their limited geographical reach to their advantage without big budgets or SEO expertise.


jeonghwa seo crafts brutalist concrete interior for etcetera cafe in seoul  Designboom

in the design of etcetera cafe seoul, jeonghwa seo emphasizes the material contrast between concrete and glass, and a visual continuity toward the street.


How to Use the Power of Local SEO for Your eCommerce Site  Business 2 Community

Local SEO is a necessity for eCommerce businesses that want to thrive, not just survive. eCommerce and local SEO can seem mutually exclusive – the very ...


Why Budapest Is The 'Go To' City For Entrepreneurs  Forbes

In recent years cities like London and Berlin have been the destinations of choice for European entrepreneurs looking to make their mark on the global stage.


Local SEO Begins with Google My Business  Business 2 Community

A free tool that allows you to control how your business appears on Google Maps and Google search results? Yes, this is Google My Business (GMB).


SEO: 6 Things to Do When You Have Nothing to Do  Practical Ecommerce

Search engine optimization is self-directed. Unless you're being managed by another practitioner who tells you what to do and when to do it, SEO can be difficult ...


Important SEO Trends To Watch In 2019  Forbes

Search engine optimization changes constantly, and businesses that want to stay ahead of the competition need to be aware of them. Keeping a pulse on these ...


Google doc rekindles myth that click-through rate affects rankings  Search Engine Land

The debate around whether Google uses click data, CTR, for core ranking will not die. No matter how many times Google will outright say they do not use it, ...


15 Common SEO Mistakes To Avoid  Forbes

Understanding SEO is the ultimate marketing discipline, especially in today's highly digital world. If you want to drive traffic to your site and increase sales, you ...


Introducing SEO for developers. It’s time.  Search Engine Land

Computer programming is making enormous innovative leaps on the web, evolving from crusty old-school AJAX, to jQuery, to reactive views that synchronize ...


These Are The Biggest SEO Trends Of 2019  Forbes

SEO strategy with components for successful marketing as icons on cubes on wooden backgroundGetty. How will the world of search engine optimization ...


5 SEO trends that will matter most in 2019  Search Engine Land

To be atop the waves, think about your SEO strategy in advance. A shortcut to success: get to know the upcoming trends and work out an action plan for each.


Park Eun Bin, Seo Ji Hoon, And BTOB's Yook Sungjae In Talks For New tvN Drama  soompi

More actors are in talks to star in tvN's upcoming drama “Man Who Bakes Bread” (literal title). “Man Who Bakes Bread” is a romance drama based on a Naver ...


SEO Is A Must For Small Businesses: Here's How To Get Started  Forbes

You are starting a business and have a checklist of the things you need to get your business going. You have the idea for a product or *service* -- check.


How SEO Has Evolved (And What's Next)  Forbes

Since the early days of SEO, marketers have deployed online strategies to ensure that their websites would gain greater visibility on search engine results pages ...


A checklist: Important SEO points to cover in a content campaign  Search Engine Land

After you've chosen the right execution for your link-building campaign and you're getting ready to launch it, you can't forget about search engine optimization ...


The 12 Areas of SEO Knowledge You Must Master to be Successful  Search Engine Journal

The role of an SEO requires mastery of certain skills and knowledge. Here's what SEOs need to master for career advancement and campaign success.


6 Reasons Why SEO Might Not Be Right For Your Business  Forbes

Contrary to popular belief, SEO is not something every business needs, especially small businesses with modest marketing budgets. Here's how to determine ...


Best Ways To Improve Your SEO And Why You Need To Do It  Forbes Now

The more quality *content* you produce, the better results you will see.


SEO Industry Approaching $80 Billion But All You Want Is More Web Traffic  Forbes

Borrell Associates predicted in 2016 that by the year 2018 businesses in the United States would spend an estimated $72 billion in search engine optimization ...


Why Prioritization Is Critical to SEO Success  Search Engine Journal

Here's why prioritizing on-site and technical optimization, *content* creation and repurposing, and link building will set your SEO campaign up for success.


12 SEO Goals for 2019: Ideas That Will Inspire You  Search Engine Journal

It's time to set your SEO goals for 2019. Need some inspiration? Read this roundup of New Year's goals from the SEO community.


The importance of understanding intent for SEO  Search Engine Land

Search is an exciting, ever-changing channel. Algorithm updates from Google, innovations in the way we search (mobile, voice search, etc.), and evolving user ...


home | site map
© 2006