SEO Information

Search Engine Spiders Lost Without Guidance - Post This Sign!


The robots.txt file is an exclusion standard required by all web crawlers/robots to tell them what files and directories that you want them to stay OUT of on your site. Not all crawlers/bots follow the exclusion standard and will continue crawling your site anyway. I like to call them "Bad Bots" or trespassers. We block them by IP exclusion which is another story entirely.

This is a very simple overview of robots.txt basics for webmasters. For a complete and thorough lesson, visit http://www.robotstxt.org/

To see the proper format for a somewhat standard robots.txt file look directly below. That file should be at the root of the domain because that is where the crawlers expect it to be, not in some secondary directory.

Below is the proper format for a robots.txt file ----->

User-agent: *
Disallow: /cgi-bin/
Disallow: /images/
Disallow: /group/

User-agent: msnbot
Crawl-delay: 10

User-agent: Teoma
Crawl-delay: 10

User-agent: Slurp
Crawl-delay: 10

User-agent: aipbot
Disallow: /

User-agent: BecomeBot
Disallow: /

User-agent: psbot
Disallow: /

--------> End of robots.txt file

This tiny text file is saved as a plain text document and ALWAYS with the name "robots.txt" in the root of your domain.

A quick review of the listed information from the robots.txt file above follows. The "User Agent: MSNbot" is from MSN, Slurp is from Yahoo and Teoma is from AskJeeves. The others listed are "Bad" bots that crawl very fast and to nobody's benefit but their own, so we ask them to stay out entirely. The * asterisk is a wild card that means "All" crawlers/spiders/bots should stay out of that group of files or directories listed.

The bots given the instruction "Disallow: /" means they should stay out entirely and those with "Crawl-delay: 10" are those that crawled our site too quickly and caused it to bog down and overuse the server resources. Google crawls more slowly than the others and doesn't require that instruction, so is not specifically listed in the above robots.txt file. Crawl-delay instruction is only needed on very large sites with hundreds or thousands of pages. The wildcard asterisk * applies to all crawlers, bots and spiders, including Googlebot.

Those we provided that "Crawl-delay: 10" instruction to were requesting as many as 7 pages every second and so we asked them to slow down. The number you see is seconds and you can change it to suit your server capacity, based on their crawling rate. Ten seconds between page requests is far more leisurely and stops them from asking for more pages than your server can dish up.

(You can discover how fast robots and spiders are crawling by looking at your raw server logs - which show pages requested by precise times to within a hundredth of a second - available from your web host or ask your web or IT person. Your server logs can be found in the root directory if you have server access, you can usually download compressed server log files by calendar day right off your server. You'll need a utility that can expand compressed files to open and read those plain text raw server log files.)

To see the contents of any robots.txt file just type robots.txt after any domain name. If they have that file up, you will see it displayed as a text file in your web browser. Click on the link below to see that file for Amazon.com

http://www.Amazon.com/robots.txt

You can see the contents of any website robots.txt file that way.

The robots.txt shown above is what we currently use at Publish101 Web Content Distributor, just launched in May of 2005. We did an extensive case study and published a series of articles on crawler behavior and indexing delays known as the Google Sandbox. That Google Sandbox Case Study is highly instructive on many levels for webmasters everywhere about the importance of this often ignored little text file.

One thing we didn't expect to glean from the research involved in indexing delays (known as the Google Sandbox) was the importance of robots.txt files to quick and efficient crawling by the spiders from the major search engines and the number of heavy crawls from bots that will do no earthly good to the site owner, yet crawl most sites extensively and heavily, straining servers to the breaking point with requests for pages coming as fast as 7 pages per second.

We discovered in our launch of the new site that Google and Yahoo will crawl the site whether or not you use a robots.txt file, but MSN seems to REQUIRE it before they will begin crawling at all. All of the search engine robots seem to request the file on a regular basis to verify that it hasn't changed.

Then when you DO change it, they will stop crawling for brief periods and repeatedly ask for that robots.txt file during that time without crawling any additional pages. (Perhaps they had a list of pages to visit that included the directory or files you have instructed them to stay out of and must now adjust their crawling schedule to eliminate those files from their list.)

Most webmasters instruct the bots to stay out of "image" directories and the "cgi-bin" directory as well as any directories containing private or proprietary files intended only for users of an intranet or password protected sections of your site. Clearly, you should direct the bots to stay out of any private areas that you don't want indexed by the search engines.

The importance of robots.txt is rarely discussed by average webmasters and I've even had some of my client business' webmasters ask me what it is and how to implement it when I tell them how important it is to both site security and efficient crawling by the search engines. This should be standard knowledge by webmasters at substantial companies, but this illustrates how little attention is paid to use of robots.txt.

The search engine spiders really do want your guidance and this tiny text file is the best way to provide crawlers and bots a clear signpost to warn off trespassers and protect private property - and to warmly welcome invited guests, such as the big three search engines while asking them nicely to stay out of private areas.

Copyright © August 17, 2005 by Mike Banks Valentine

Google Sandbox Case Study http://publish101.com/Sandbox2 Mike Banks Valentine operates http://Publish101.com Free Web Content Distribution for Article Marketers and Provides content aggregation, press release optimization and custom web content for Search Engine Positioning http://www.seoptimism.com/SEO_Contact.htm


MORE RESOURCES:

Building and scaling global SEO Centers of Excellence  Search Engine Land

Wider spread internet adoption and mobile use worldwide are opening up lucrative new markets in Latin America, China and the rest of Asia-Pacific and more.


SEO: 9 Content Optimization Tools  Practical Ecommerce

Contextual relevance in search engine optimization refers to how the *content* on a website aligns with searchers' words and phrases. These nine *content* ...


8 Ways You’re Wasting Money on SEO Without Knowing It  Search Engine Journal

Not getting the ROI you expect from SEO? Here are eight ways you might be wasting time and money – and how to increase your chances of success.


SEO Explained to a 5 Year Old  Business 2 Community

Which got me thinking, when he wants to know about the world around him, the simplest explanation is always the best. I realised that this approach can also be ...


Seznam SEO: An Interview with Seznam’s Search Division Director  Search Engine Journal

Discover how Czech search engine Seznam approaches modern day search, how it processes JavaScript-powered websites, and more.


7 Reasons Why SEO Matters for Every Startup  Entrepreneur

No matter how lean your startup, you can at least afford do-it-yourself SEO.


Why You Must Stop Obsessing About These 5 Things in SEO  Search Engine Journal

Are you obsessed with the wrong things in SEO? Here are five things that are overrated or badly misused and a couple trends that still hold weight.


4 Most Important SEO Stats In 2019  Tech Digest

Search Engine Optimization is the practice of optimizing a website's *content*. Here we look at some of the latest SEO statistics in 2019.


Majority of SMBs doing SEO are in-housing  Search Engine Land

A majority of small businesses (SMBs) surveyed understand they should be doing SEO but only 36 percent have an SEO strategy in place and are actively ...


Boost Your Website's Visibility With This SEO Course  Entrepreneur

Turn your website into a profit-producing machine with this back-linking course.


5 Software Programs to Help You With SEO Strategies  Thrive Global

If you're a web designer or simply working in any aspect of the tech industry, then you should have SEO as one of your top priorities in terms of helping your ...


Two Ways To Avoid SEO 'Perfection' (And Why That's A Good Thing)  Forbes

Stop trying to serve Google and start trying to genuinely serve your target audience.


SEO: Using Lighthouse to Improve Site Speed  Practical Ecommerce

Site speed is tricky for search engine optimization practitioners. It's a ranking signal for Google and a key factor in a customer's experience. But few SEO pros ...


Skip the SEO and SEM basics  Marketing Land

Come to SMX Advanced June 3-5 in Seattle for expert-led search marketing sessions.


How to Create Content for SEO  Search Engine Journal

Creating valuable *content* is a crucial component of SEO. Follow these six steps to help your *content* rank higher on search engines.


The 4 Pillars of Enterprise SEO Success  Search Engine Journal

Mitigation, reporting, project management, and relationship building are all important to the success of enterprise SEO. Here's why.


5 additional data blending examples for smarter SEO insights  Search Engine Land

As I covered in my previous article, data blending can uncover really powerful insights that you would not be able to see otherwise. When you start shifting your ...


What killing rel=prev/next means for SEO  Search Engine Land

By now, you've probably heard that Google is no longer supporting rel=prev/next markup. In fact, Google hasn't supported it for years. That's unfortunate ...


The anatomy of an SEO robocall scheme  Search Engine Land

On Oct. 11, 2018, I received a phone call and discovered that Classy Brain (my company) was in existential crisis. The end was near – Google was after me.


SEO Strategy vs. Tactics: What’s the Difference? A Lot!  Search Engine Journal

Understanding the difference between strategy and tactics also means making sure everyone on your team is using the terminology in the same way.


6 Effective Onboarding Tips for New SEO Customers  Search Engine Journal

Offering SEO services? Here are six client onboarding tips that will improve your new business retention and customer lifetime values.


Four excellent case studies of structured data for SEO – Econsultancy  Econsultancy

The importance of structured data, a type of code that makes the *content* of webpages easier for search engines to interpret, is frequently emphasised in advice ...


Rel=Next/Prev & SEO: Insights from Google Crawling Behavior  Search Engine Journal

In an ecosystem where conjectures and theories dominate debates, Google's announcement on March 21 that rel next/prev tags are no longer used by the ...


Five Pro Small Business SEO Tips from a CEO  MarTech Advisor

Every CEO that one talks to, always want to see their website rank higher in the Google search and that's what each of them will stress in their meetings.


Dominate With SEO Meets #GoldenMic  AgWired

One of the folks who stopped by the ZimmComm booth during the Agri-Marketing Conference Connection Point was Sam Romain, Dominate With SEO.


301 vs. 302 Redirects & SEO: The What, Why & How  Search Engine Journal

301 vs. 302 redirects. Confused about which one to use? You're not the only one. Even in 2019, website owners are unclear about which type of redirect is best ...


Five SEO Strategies Financial Marketers Must Ditch ASAP  The Financial Brand

Some search engine optimization tactics, like matching *content* to user intent and building quality backlinks, have stood the test of time. But many SEO tasks that ...


SEO: Lessons from Google's 'rel=prev/next' Pagination Identifier  Practical Ecommerce

Search engine optimization professionals were caught off guard this week by Google's admission that it hasn't used its pagination identifier in “a number.


10 Quick Tips on SEO for a Restaurant Website  Customer Think

People explore different restaurant options on the search engines before they pick one. Boosting your restaurants' visibility online can be tricky. Though there ...


How to Build Authority in Local SEO  Search Engine Journal

Site authority is crucial to the success of any local SEO efforts. Here are some of the best ways to build the authority of a local business.


The Ultimate Guide to Press Release SEO: 2019 Edition  IR Magazine

It's critical to use search engine optimization (SEO) to create better-performing *content*. This guide will help you create more effective press releases and investor ...


Basic SEO Techniques to Improve Website Search Visibility  MarketingProfs.com

Businesses that rely on organic search know the importance of SEO, but ranking for keywords has become more difficult. Complete these 3 basic SEO tasks to ...


Evangelizing SEO: The Can’t-Lose Case for Your Content Team  Search Engine Journal

Here's how to build a convincing case to your *content* team that SEO isn't a checklist or a chore, but a superpower that will boost their success.


SERPstash can help boost your site's SEO power for only $24  PCWorld

SERPstash analyzes your site and automatically identifies SEO improvements.


Improve Your Marketing Strategy with These 4 SEO Tools  Business 2 Community

As Google continues to update its algorithm it's more important than ever to set up your SEO strategy for the best visibility online. Thankfully there are helpful ...


Things to look for in an SEO agency  Times Square Chronicles

Introduction. With many disreputable companies advertising themselves online, it can be hard to find the right SEO agency. SEO is an investment which is ...


Voice Search Is the Most Common and Biggest SEO Trend of 2019 to Follow  Influencive

It is said that you should think out of the box to stand apart from the crowd, but when it comes to SEO you must also follow the current trends to get the best ...


79% of Digital Advertisers are Also Using SEO in Their Marketing Strategies  Search Engine Journal

New data reveals that 79% of digital advertisers are also using SEO as part of their marketing strategy in 2019.


Demystifying SEO: A Simple Explanation for Jewelers  INSTORE MAG

YOU'RE NOT ALONE. A lot of people don't really understand what SEO is, why it's helpful, and what people are doing when they do it. But if it's as important as ...


Google Uploads Last JavaScript SEO Video: Dynamic Rendering For JavaScript Sites  Search Engine Roundtable

Google's Martin Splitt has uploaded his last video in the series of videos around JavaScript and SEO yesterday, this one was on the topic of dynamic rendering.


Are Disjointed and Misaligned SEO Benchmarks Impacting Content Management Practices?  MarTech Series

Google may have changed its algorithm but SEO practices are still making a huge impact on the *Content* Management strategies, especially for small and ...


SERPStash's Low Price Makes It a Must Have SEO Tool  Inverse

SERPStash is a tool that streamlines the process of search engine optimization for non-experts. It breaks everything down into 3 parts: identifying keywords, ...


2019 Innovation & Technology Awards winner: Helium SEO - Cincinnati Business Courier  Cincinnati Business Courier

The roughly 30-person staff of Helium SEO consists of only two creatives. That's because Helium's approach to search engine optimization and search ...


SEO National Welcomes Pet Minerals to Their Search Engine Optimization Family  PRUnderground

Utah SEO company beats out the competition to sign on cutting-edge pet wellness company.


Submit Questions Around JavaScript SEO For Google  Search Engine Roundtable

As you know, Google posted their last video on their JavaScript videos for SEO yesterday. But now that the first part of the series is over, Martin Splitt from Google ...


SMX Overtime: Learn the difference between mobile-first indexing and mobile-friendly SEO  Search Engine Land

General Manager of Perficient, Eric Enge, explained how to construct a mobile-friendly SEO site with good UX during the “Mobile First Indexing & Mobile ...


My Boss Got an Email Saying Our SEO Sucks  Search Engine Journal

Attention all site owners: Stop trusting spammers who email you misleading SEO reports designed to scare you. Trust in us, and the results we can provide.


Seo: The Artist's Dance  ArtfixDaily

NEW YORK, NY – Chelsea's Agora Gallery will feature the original work of Seo in A Rich Spring Profusion. The exhibition opens April 18 and runs through May ...


White Label SEO Platform by WebCEO Has Become a Turnkey SEO Business for Digital Agencies and Software Resellers  PRNewswire

LONDON, April 9, 2019 /PRNewswire/ -- WebCEO has relaunched its white-label SEO platform with powerful agency-level features, now allowing digital ...


Designing a New Website: Choosing the Best Tech for Your Dev  Search Engine Journal

Designing a new website? Here are the pros and cons of using a CMS vs. a website builder, as well as some SEO considerations when choosing a platform.


Is link building dead? Depends on who you talk to  Search Engine Land

Sure, link building used to be plagued by a morass of black-hat tactics like link farms, paid link schemes, hidden links and spammy *content*. But after Google ...


New Data Shows How Much SEOs are Charging for Services in 2019  Search Engine Journal

A study on digital marketing pricing shows how much SEO consultants and agencies are charging for their services in 2019.


Fake Amazon reviews giving unknown brands an SEO bump, says Which?  The Drum

Consumer watchdog Which? has warned of a glut of 'fake' Amazon reviews are propagating to such an extent that they risk undermining the integrity of the ...


What an SEO Consultant Should Deliver  Business 2 Community

What should an SEO consultant deliver? There are many opinions when it comes to search engine optimization, but for businesses it comes down to real results.


Why Local Pack SEO is Essential to Boost Your Business  Business 2 Community

Have you ever gone out and wandered the streets looking for a place to eat? Found one and wondered if it was worth your time and money? Now, when was the ...


Easy 7-Point SEO Checklist to Boost eCommerce Product Page Rankings  Pacifc54 Blog

Here are 7 proven methods of SEO for eCommerce product pages to drive organic traffic and increase product sales.


5 Essentials for SEO Content Writing  Business 2 Community

SEO, or search engine optimization, is the practice of tailoring *content* to appear at the top of search results. Optimizing your *content* for search best positions ...


Tip: Refresh your SEO knowledge | Tip of the day  Journalism.co.uk

These 18 actionable tips will help your *content* compete online in 2019.


LinkedIn Gives SEO Tips for Boosting Visibility of Company Pages  Search Engine Journal

LinkedIn has published a series of SEO tips for getting company pages found in search engines.


What is the impact of Google's March 2019 algorithm update?  Econsultancy

On 12th March, Google confirmed that it had once again updated its core algorithm. What do we know about the change so far, and how has it impacted SEO?


SEO Basics: Is Translation an SEO Duplicate Content Issue for Google?  TAPinto.net

Though people have tried myth-busting the idea of a penalty associated with duplicate *content*, blatantly using *content* that has been copied and pasted from ...


How A Low-Budget SEO Campaign Can Still Propel Your Digital Brand to Success  Influencive

Many digital brands or brick-and-mortar stores wishing to make the push into e-commerce face tough decisions on how to allocate small budgets. For small and ...


Google Help Video On Angular SEO  Search Engine Roundtable

Here is the next video in the series of JavaScript SEO videos from Google. This one is on Make your Angular web apps discoverable in search and SEO friendly.


Great Minds Link Alike: Basic Link-Building Strategies for Your SEO Plan  Business 2 Community

You have probably heard the saying, “It's not what you know, it's who you know.” Although I would argue that both are important, sometimes the latter can help ...


Singapore Online Marketing Agency Announces Comprehensive Multi-level SEO Solutions  High Tech Deck

Singapore search engine marketing agency IMSimplify has announced the expansion of its SEO services. The full-cycle online marketing agency in Singapore ...


20 Local SEO Dos and Don'ts You Must Follow for Success on Google [Infographic]  Social Media Today

Are you looking for ways to get your website to rank higher in local Google rankings? Want to know the best local SEO techniques small business website ...


Optimizing SEO with Schema Markup  Legal Talk Network

Martha van Berkel talks about all things schema markup, a.k.a. structured data.


Fife Marketing and SEO Services Launched  Newswire

Fife Web Services, based in Fife, Scotland, have launched a new website showcasing their services for local clients. They can help clients to establish a strong ...


The Art of Choosing the Right SEO Keywords  Business 2 Community

Search engine optimization (SEO) is more of an art than a science. Yes, there are best practices and several clear standards for SEO that make it seem like a ...


There’s no shortcut to authority: Why you need to take E-A-T seriously  Search Engine Land

Recently, I was talking to a client about a new project when he raised an interesting question. He was curious to know how I measured E-A-T, or “expertise, ...


The Essential 12-Point SEO Content Writing Checklist  Pacifc54 Blog

Looking to improve your *content* for SEO? We've outlined a 12 point SEO *Content* Writing checklist to help you make the most of your *content*.


3 Top Black Hat SEO Techniques: Ideal for Destroying Your Site Faster!  Business 2 Community

No matter how much the internet evolves, the mentality of some people stays the same. They want their cake and to eat it too. black hat seo techniques.


Using a SEO & Digital Marketing Company in London UK - What to look out for  Baltimore Post-Examiner

1. Save you time and resources—let the experts handle all your SEO needs and use your time and money on other important tasks. However, you need to ...


Does SEO Conflict with User Experience?  Business 2 Community

If you work with an SEO expert, you have encounter times when her recommendations or priorities conflict with the overall goal of improving the website. This will ...


Top 4 Technical SEO Talks at Brighton SEO April 2019  QueryClick

As a member of the Technical SEO Team at QueryClick here's a roundup of my most interesting learnings and takeaways from Brighton SEO 2019.


10 Keyword Density Checkers (And How They Still Can Help Your SEO)  Business 2 Community

First, let's clarify the idea. Today, semantic search reigns, as well as high-quality *content*, and great user experience. Search engines focus on topical relevance ...


SEO Trends in 2019  Business 2 Community

SEO Trends in 2019. Search engine optimization (SEO) has come a long way since the days of stuffing whatever keyword you wanted to rank for in your code ...


The secrets of meta descriptions and SEO  The Drum

Meta descriptions are a much misunderstood weapon in the search arsenal that can have a serious impact on page performance. The meta description field is ...


Mastering SEO for your Design Business  Business of Home

As a savvy design business owner, you've undoubtedly invested time and effort into a beautiful website, but if your site is buried in the crowd, does it even exist?


Content Strategy and the Google March 2019 Algorithm Update  Search Engine Journal

MarketMuse shares surprising *content* strategy trends associated with the March 2019 Update.


SEOs show mixed results following Google March 2019 core update  Search Engine Land

We asked you to help us analyze the March 12 core update and after over 500 results we wanted to share that overall the results came out pretty inconclusive.


SEO Testing Service Market Size, Future Forecast, Key Manufacturers and Industrial Opportunities to 2024  Market Research Updates

Industry Overview Of SEO Testing *Service* Market. The research report focuses on target groups of customers to help players to effectively market their products ...


Denver SEO Expert Forms South Denver SEO Alliance / MeetUp  castlerockco.com

Are you passionate about SEO (Search Engine Optimization)? The South Denver SEO MeetUp was designed to connect like-minded SEOs.


West Yorkshire agency handed five-figure SEO brief by furniture maker  Prolific North

Handmade furniture company, Parker Howley & Co, has appointed Vizulate Digital to deliver a five-figure digital marketing campaign that will support its ...


11 SEO strategy trends that are set to dominate digital marketing  YourStory

Search engine optimisation (SEO) is the backbone of brands when it comes to online presence. Ranking higher on SERP takes effort and time – and gets even ...


Why ABT Principal Hee Seo Loves the Art of Giving Back  pointemagazine.com

What are you proudest of in your career? That I learned how to work in the studio. I always loved being onstage, but now I love the process of getting there.


Local SEO: How to Optimise and Generate Leads  Business 2 Community

Local SEO is fast becoming a high priority for many businesses. Last year, Google published a study showing a 500% increase in “near me” searches, including ...


Avoid These 5 Blog Strategy & Creation Mistakes  Search Engine Journal

A blog is a powerful marketing tool for businesses. Avoid these big mistakes when creating a strategy and your blog posts.


Ahrefs, SEO toolset provider, to build their own search engine to compete with Google  Search Engine Land

Ahrefs, a popular SEO toolset provider platform, announced this morning on Twitter that they want to compete with Google by building their own search engine.


Mordy Oberstein, Author at Search Engine Land  Search Engine Land

Mordy is the head of marketing for Rank Ranger, an industry leading all-in-one SEO reporting suite. Outside of helping to build the Rank Ranger brand, Mordy ...


Press Release: The Australian SEO Industry Mined With Dodgy Practitioners – How To Avoid Them  PRWire

A common reason why small businesses delay the decision of using SEO services is represented by the negative stories they have heard from other people in ...


Marian M. Seo March 29, 2019 - Obits  Standard Speaker

Marian M. Seo, 86, of Lincoln Street, Hazleton, died Friday morning at her home. Born in Columbia, S.C., on Oct. 24, 1932, she was a member of the Christ ...


Singapore Local SEM Agency and SEO Services Expanded  Newswire

Singapore search engine marketing agency IMSimplify has announced the expansion of its SEO services.


PSB Academy picks Happy Marketer to bolster SEO and UX  Marketing Interactive

Private education institution, PSB Academy has appointed Happy Marketer, a Merkle Company, as its search agency. The length of appointment is for a year, ...


Head coach Park Hang-seo reveals Vietnam national football team’s secret to success  FOX Sports Asia

Vietnam have enjoyed unprecedented success in international football ever since South Korean tactician Park Hang-seo took charge as the head coach of their ...


7 Expert Tips to Protect Your Online Reputation  Search Engine Journal

Having a good reputation often means more sales for businnesses. Here is some detailed advice to help protect your reputation online.


How to Improve Ecommerce SEO with User Generated Content  Search Engine Journal

To drive more sales, ecommerce businesses first need to rise to the top of both paid and organic search results and earn lots of high-quality traffic. Between ...


SEO Nagi Reddy reviews local bodies’ polls with all officials  The Siasat Daily

Hyderabad: In connection with local bodies electios, State Election Commissioner Nagi Reddy today held a meeting with Chief Secretary SK Joshi, DGP.


AMP Stories Now Have a Dedicated Section in Google Search Results  Search Engine Journal

Google is adding a new section to search results which is dedicated to AMP Stories.


home | site map
© 2006