SEO Information

Playing in Googlebots Sandbox with Slurp, Teoma, & MSNbot - Spiders Display Differing Personalities


There has been endless webmaster speculation and worry about the so-called "Google Sandbox" - the indexing time delay for new domain names - rumored to last for at least 45 days from the date of first "discovery" by Googlebot. This recognized listing delay came to be called the "Google Sandbox effect."

Ruminations on the algorithmic elements of this sandbox time delay have ranged widely since the indexing delay was first noticed in spring of 2004. Some believe it to be an issue of one single element of good search engine optimization such as linking campaigns. Link building has been the focus of most discussion, but others have focused on the possibility of size of a new site or internal linking structure or just specific time delays as most relevant algorithmic elements.

Rather than contribute to this speculation and further muddy the Sandbox, we'll be looking at a case study of a site on a new domain name, established May 11, 2005 and the specific site structure, submissions activity, external and internal linking. We'll see how this plays out in search engine spider activity vs. indexing dates at the top four search engines.

Ready? We'll give dates and crawler action in daily lists and see how this all plays out on this single new site over time.

* May 11, 2005 Basic text on large site posted on newly purchased domain name and going live by days end. Search friendly structure implemented with text linking making full discovery of all content possible by robots. Home page updated with 10 new text content pages added daily. Submitted site at Google's "Add URL" submission page.

* May 12 - 14 - No visits by Slurp, MSNbot, Teoma or Google. (Slurp is Yahoo's spider and Teoma is from Ask Jeeves) Posted link on WebSite101 to new domain at Publish101.com

* May 15 - Googlebot arrives and eagerly crawls 245 pages on new domain after looking for, but not finding the robots.txt file. Oooops! Gotta add that robots.txt file!

* May 16 - Googlebot returns for 5 more pages and stops. Slurp greedily gobbles 1480 pages and 1892 bad links! Those bad links were caused by our email masking meant to keep out bad bots. How ironic slurp likes these.

* May 17 - Slurp finds 1409 more masking links & only 209 new content pages. MSNbot visits for the first time and asks for robots.txt 75 times during the day, but leaves when it finds that file missing! Finally get around to add robots.txt by days end & stop slurp crawling email masking links and let MSNbot know it's safe to come in!

* May 23 - Teoma spider shows up for the first time and crawls 93 pages. Site gets slammed by BecomeBot, a spider that hits a page every 5 to 7 seconds and strains our resources with 2409 rapid fire requests for pages. Added BecomeBot to robots.txt exclusion list to keep 'em out.

* May 24 - MSNbot has stopped showing up for a week since finding the robots.txt file missing. Slurp is showing up every few hours looking at robots.txt and leaving again without crawling anything now that it is excluded from the email masking links. BecomeBot appears to be honoring the robots.txt exclusion but asks for that file 109 times during the day. Teoma crawls 139 more pages.

* May 25 - We realize that we need to re-allocate server resources and database design and this requires changes to URL's, which means all previously crawled pages are now bad links! Implement subdomains and wonder what now? Slurp shows up and finds thousands of new email masking links as the robots.txt was not moved to new directory structures. Spiders are getting errors pages upon new visits. Scampering to put out fires after wide-ranging changes to site, we miss this for a week. Spider action is spotty for 10 days until we fix robots.txt

* June 4 - Teoma returns and crawls 590 pages! No others.

* June 5 - Teoma returns and crawls 1902 pages! No others.

* June 6 - Teoma returns and crawls 290 pages. No others.

* June 7 - Teoma returns and crawls 471 pages. No others.

* June 8-14 Odd spider behavior, looking at robots.txt only.

* June 15 - Slurp gets thirsty, gulps 1396 pages! No others.

* June 16 - Slurp still thirsty, gulps 1379 pages! No others.

So we'll take a break here at the 5 weeks point and take note of the very different behavior of the top crawlers. Googlebot visits once and looks at a substantial number of pages but doesn't return for over a month. Slurp finds bad links and seems addicted to them as it stops crawling good pages until it is told to lay off the bad liquor, er that is links by getting robots.txt to slap slurp to its senses. MSNbot visits looking for that robots.txt and won't crawl any pages until told what NOT to do by the robots.txt file. Teoma just crawls like crazy, takes breaks, then comes back for more.

This behavior may imitate the differing personalities of the software engineers who designed them. Teoma is tenacious and hard working. MSNbot is timid and needs instruction and some reassurance it is doing the right thing, picks up pages slowly and carefully. Slurp has addictive personality and performs erratically on a random schedule. Googlebot takes a good long look and leaves. Who knows whether it will be back and when.

Now let's look at indexing by each engine. As of this writing on July 7, each engine also shows differing indexing behavior as well. Google shows no pages indexed although it crawled 250 pages nearly two months ago. Yahoo has three pages indexed in a clear aging routine that doesn't list any of the nearly 8,000 pages it has crawled to date (not all itemized above.) MSN has 187 pages indexed while crawling fewer pages than any of the others. Ask Jeeves has crawled more pages to date than any search engine, yet has not indexed a single page.

Each of the engines will show the number of pages indexed if you use the query operator "site:publish101.com" without the quotes. MSN 187 pages, Ask none, Yahoo 3 pages, Google none.

The daily activity not listed in the three weeks since June 16 above has not varied dramatically, with Teoma crawling a bit more than other engines, Slurp erratically up and down and MSN slowly gathering 30 to 50 pages daily. Google is absent.

Linking campaign has been minimal with posts to discussion lists, a couple of articles and some blog activity. Looking back over this time it is apparent that a listing delay is actually quite sensible from the view of the search engines. Our site restructuring and bobbled robots.txt implementation seems to have abruptly stalled crawling but the indexing behavior of each engine displays distinctly differing policy by each major player.

The sandbox is apparently not just Google's playground, but it is certainly tiresome after nearly two months. I think I'd like to leave for home, have some lunch and take a nap now.

Back to class before we leave for the day kiddies. What did we learn today? Watch early crawler activity and be certain to implement robots.txt early and adjust often for bad bots. Oh yes, and the sandbox belongs to all search engines.

Mike Banks Valentine is a search engine optimization specialist who operates http://WebSite101.com and will continue reports of case study chronicling search indexing of http://Publish101.com


MORE RESOURCES:
SEO: 9 Tips for Writing Great Content  Practical Ecommerce

Textual *content* enables search engines to understand your site. The words you choose should engage shoppers and boost your site's relevance to the ...


7 Best SEO Tools to Help You Rank Higher in Google  Entrepreneur

The key to building and sustaining organic traffic is knowing which are the best SEO tools to use. Today, there are so many SEO checkers on the market that it be ...


How to Activate a Sponsorship to Achieve Content Marketing & SEO Goals  Search Engine Journal

Want high-quality coverage and links? Here's how to maximize sponsorships to help you to achieve your *content* marketing and SEO goals.


Exploring the Role of Content Groups & Search Intent in SEO  Search Engine Journal

Patents show us what strategies Google chooses to research and invest in, which in turn provides insight into how to best prepare a website for indexation by ...


5 Tips for Choosing Your Next SEO Firm  Entrepreneur

Making the right decision when choosing your next SEO firm has major consequences for your business and your bank account. The difference between a boom ...


The 20-Minute Technical SEO Workweek  Search Engine Journal

Have you checked your technical SEO lately? Use this workflow to quickly get a pulse of your technical SEO in just 20 minutes a week.


SEO: 6 Things to Do When You Have Nothing to Do  Practical Ecommerce

Search engine optimization is self-directed. Unless you're being managed by another practitioner who tells you what to do and when to do it, SEO can be difficult ...


5 SEO Career Trends for 2019 [New Research]  Search Engine Journal

What can data from LinkedIn, Indeed, and Census.gov tell us about SEO jobs? Discover five interesting trends from an analysis of 75 large U.S. metros.


6 Ways SEO Tools Can Help You Run Your Business  Search Engine Journal

Tools make a marketer's life more productive. This post will tell you why and how SEO tools can help you better run your business.


SEO vs SEM – What’s the Difference and Why it Matters to Your Business  G2 Crowd

Learn the key difference between SEO and SEM and how they greatly increase the visibility of your site on search engines like Google, Yahoo, Bing, and more.


Is It Time To Outsource SEO? Here's What You Need To Consider  Forbes

Everyone and their mother is chasing after the holy grail of internet marketing: organic search traffic.


Tangle Teezer appoints Avenue Digital for international SEO and paid search  The Drum

British hairbrush brand Tangle Teezer has appointed London search agency Avenue Digital to handle its SEO and PPC activity following a competitive pitch.


Lucid Leverage Fuels SEO Services In Phoenix, Celebrates 4 Years - Press Release  Digital Journal

Lucid Leverage are Search Engine Optimisation (SEO), social media management and online reputation marketing specialists providing their services in ...


Introducing SEO for developers. It’s time.  Search Engine Land

Computer programming is making enormous innovative leaps on the web, evolving from crusty old-school AJAX, to jQuery, to reactive views that synchronize ...


Angular Universal: What you need to know for SEO  Search Engine Land

If you use Angular to power your website, then you have an extra hurdle to cross when it comes to SEO. Fortunately, Angular Universal makes it easy to leap ...


BVM Now Providing SEO Services to Businesses in El Paso, TX  Markets Insider

HOUSTON, Feb. 15, 2019 /PRNewswire-PRWeb/ -- BVM (https://www.brazosvalleymarketing.com), an UpCity certified partner, is now offering SEO services in El ...


SEO Manager - Upwork  Built In Chicago

Upwork is hiring for a SEO Manager in Chicago. Find more details about the job and how to apply at Built In Chicago.


Why Prioritize Local SEO?  Vertical Measures

Local SEO consists of many different local citation components, but is absolutely vital to the success of any organization aiming to reach to a nearby audience.


SEO: Reclaiming Lost Links from Site Migrations  Practical Ecommerce

There is a little-used opportunity for longtime retailers to gain links. If your ecommerce site has changed platforms in its lifetime, you might have editorial links ...


Tips for Selecting the Right SEO Provider for Your Accounting Firm  CPAPracticeAdvisor.com

Taking the first steps into the world of search engine optimization (SEO) can be intimidating. If you have spent anytime researching the topic online then you ...


Wix ‘SEO Wiz’ Stars in Big Super Bowl Ad, Fumbles with SEO Community  Search Engine Journal

Did you hear that sound last night during the Super Bowl? It was the collective SEO community hitting their heads against a proverbial desk. Wix's foray into the ...


SearchCap: Quora keyword targets, Angular SEO & old stock SEO  Search Engine Land

Learn how to manage product unavailability without hurting your SEO Feb 1, 2019 by Max Cyrek. When your e-commerce site has out-of-stock products, here ...


Updated for 2019! ‘Enterprise SEO Platforms: A Marketer’s Guide’  Search Engine Land

MarTech Today's “ Enterprise SEO Platforms: A Marketer's Guide” has been updated for 2019. Compiled from the latest research, this 61-page report is your ...


What Is an SEO Professional in 2019? An Identity Crisis Averted  Search Engine Journal

Picture this: You're an SEO professional, and someone asks you what exactly it is that you do. What do you say? I'm a: Search analytics guru; *Content* strategist ...


25 tools to streamline YouTube SEO optimization  Search Engine Land

With video streaming services and social media platforms reporting large amounts of traffic, videos are slowly but surely moving front and center as the most ...


How reputation became a major ranking factor in SEO  Search Engine Land

Over 10 years ago, I predicted that Google could use quality scoring for organic rankings, and I also proposed a number of ways they could quantify the quality of ...


Learn the secrets of SEO with this certified training  Popular Science

While social media gets a lot of attention, most traffic still comes from search engines. If you want to promote your start-up or build a career in digital marketing, ...


MarTech Today Research: A marketer’s guide to enterprise SEO platforms  Search Engine Land

Enterprise SEO continues to grow more complex as the technology and algorithms powering search engines evolve. Comprehensive enterprise SEO platforms ...


SEO: Monitoring Traffic from Domain Mergers, with Google Data Studio  Practical Ecommerce

A common practice among retailers is creating microsites to target different audiences. The strategy can produce strong conversions because the user ...


Super Bowl Sunday SEO: The importance of search during the Big Game  ClickZ

SEO is crucial during the Super Bowl, both for $5 million-dollar advertisers and companies hoping to appear in local search results.


10 Ways to Think Like Googlebot & Boost Your Technical SEO  Search Engine Journal

Have you found yourself hitting a plateau with your organic growth? High-quality *content* and links will take you far in SEO, but you should never overlook the ...


How to Use Python to Analyze SEO Data: A Reference Guide  Search Engine Journal

Do you find yourself doing the same repetitive SEO tasks each day or facing challenges where there are not tools that can help you? If so, it might be time for you ...


60+ SEO Stats to Help You Rank Better in 2019 [Infographic]  Social Media Today

Are you looking to maximize your search performance in 2019? Need to know the stats and facts to help you understand the SEO process? The team from Red ...


How to redesign your website without effective your SEO  Customer Think

When you are planning to have a website redesign, it is vital that you take your time doing it and be careful, as if it is not done right, it could bring chaos to your ...


SEO for B2B Businesses: Is It Really a Necessity?  Thrive Global

Effective SEO (search engine optimization) is a large portion of the entire mission of a B2B *service* company, as if you don't have stellar SEO, people aren't ...


SEO Testing Service Market Will Generate New Growth Opportunities By 2025| Key Players: SEO SiteCheckup, WebDepend, TestMatick  TheTokenClock

HTF MI recently introduced Global SEO Testing *Service* Market study with in-depth overview, describing about the Product / Industry Scope and elaborates ...


11 SEO Tips & Tricks to Improve Indexation  Search Engine Journal

Here are 11 tips and tricks to ensure that your website gets crawled and indexed properly – and help your webpages rank higher in search.


Loyalty Marketing: The Proven Complement to Your SEO Strategy  Search Engine Journal

Here's how loyalty marketing can complement your SEO strategy and why the two disciplines need to be conducted seamlessly.


Five foolproof SEO strategies to supercharge your webpage | Articles | Strategy  Innovation Enterprise

*Content*, age, and authority are the three pillars of trust that equate to Google's elusive relevancy ranking. If you fail to provide these three, you'll likely flounder in ...


Fusion SEO Accelerates Growth with Expanding to Central New Jersey - Press Release  Digital Journal

OLD BRIDGE, New Jersey - February 15, 2019 - Fusion SEO has merged with SOCii Media Marketing, a Digital Marketing Agency located in Central New ...


Senior SEO Editor | Media jobs  Journalism.co.uk

Editorial jobs - 64 journalism, communications and media vacancies to view and apply for now.


Search Engine Land adds technical, in-house SEO experts to team  Search Engine Land

Search Engine Land is thrilled to announce that the team of experts who help us bring daily search marketing news and insights to thousands just got a bit ...


SEO Services | Search Engine Optimisation  QueryClick

Our data-led approach enables us to drive measurable improvements in revenue and profit from organic search. Each strategy is customised to match increased ...


4 Link Building Tips for International SEO  Search Engine Journal

Discover four link building tips for global websites that can be easily implemented even by companies with limited resources.


The Amazon SEO game plan: SEO refinements that increase visibility  DigitalCommerce360

Effective SEO can help you stand above the noise on Amazon. Crafting the perfect keywords without misleading your customers and making your descriptions ...


SEO vs PPC: How to Choose What’s Best For Your Business  Business 2 Community

As a business owner are you often confused about choosing the right online marketing channel? Most businesses struggle with scanty resources. Marketing ...


40 Major SEO Mistakes You Must Avoid in 2019 [Infographic]  Social Media Today

Are you struggling to get your website to rank on the first page of Google? Want to see if you're making any SEO mistakes that could be holding your website ...


YouTube SEO: 5 Hacks for Quicker, Better YouTube Keyword Research  Business 2 Community

Proper YouTube keyword research is the backbone of a good YouTube SEO strategy. Why? Because finding the right YouTube keywords could mean the ...


Top 10 mistakes companies are making with local SEO  ZDNet

Almost two out of five business leaders do not know if their website is optimised for local SEO. So, what should they look out for?


How to Create a Content Strategy for Both Users & Search Engines  Search Engine Journal

Learn actionable tips on how to create a *content* strategy for both users and search engines that will drive traffic to your website.


Former Starbucks SEO Howard Schultz Considers Presidential Bid  HuffPost

Schultz is considering running as an independent. He could unintentionally split the vote and serve Trump a second term.


Rise of Mobile Users Alters SEO Tactics  Practical Ecommerce

The percentage of U.S. searchers clicking on advertisements in Google search results is growing. Likewise, the percentage of visitors that click nothing is also ...


What is Topic Modeling? How Do You Use Topic Modeling for SEO  Business 2 Community

What is Topic Modeling? Topic modeling itself is a complex system that ranks *content* based not just on the keywords, but also on the context. We are long past ...


Yoast SEO to add “live indexing” feature for Bing OnMSFT.com  OnMSFT

Following Bing's announcement that they are hoping to change the way search engines discover new and updated *content*, one of WordPress's largest SEO ...


The Latest Google Search Console Updates: What You Need to Know  Search Engine Journal

Google Search Console is evolving at a rapid pace. Find out everything you need to know about all of the latest feature changes here.


"Super wolf blood moon” is a media and SEO invention  Quartz

There is no such thing as a “blood moon.” It's mostly a media invention, driven largely by efforts to get readers to visit news websites—through a tactic known as ...


ILS at forefront of monitoring for climate change impacts: John Seo  Artemis.bm

The impacts of climate change present challenges for all industries and markets, but over the last two decades, the insurance-linked securities (ILS)


Responsive Web Design Questions About Mobile SEO  Business 2 Community

Users are bouncing, conversions are sparse and sales are way down. Or, *insert in your problem here.*. Whatever the concern, let's be honest… your site needs ...


6 Simple Local SEO Tactics Financial Marketers Can Use Today  The Financial Brand

Community banks and credit unions can turn their limited geographical reach to their advantage without big budgets or SEO expertise.


Uee Thanks Former Co-Star Lee Seo Jin For Sending Her A Coffee Truck On Set  soompi

Uee recently took to social media to thank her former co-star Lee Seo Jin for his good deed! On February 15, Uee posted a photo of herself standing in front of ...


Google: Don't Break The Rules Even If Competitors Are  Search Engine Roundtable

Google's John Mueller at least twice this week had to respond on Twitter suggesting that SEOs do not break the rules, go against the Google webmaster ...


How to Use the Power of Local SEO for Your eCommerce Site  Business 2 Community

Local SEO is a necessity for eCommerce businesses that want to thrive, not just survive. eCommerce and local SEO can seem mutually exclusive – the very ...


IPG's Daniel Picken Joins The Wired Agency As SEO Manager  B&T

Continuing on their impressive growth trajectory, Sydney-based independent digital marketing agency, The Wired Agency has expanded their SEO team with a ...


Watch: Jin Goo And Seo Eun Soo Are Headstrong Lawyers In Highlight Video For “Legal High”  soompi

Legal High” has released a new highlight video showing a preview of the upcoming drama! JTBC's upcoming Friday-Saturday drama “Legal High” is about two ...


A Global Press Release Distribution Service Can Have a Massive Impact on SEO  Life Pulse Health

A press release is a piece of writing that is used for mostly announcements of new products, services, offers or sharing any valuable information that is beneficial ...


15 Common SEO Mistakes To Avoid  Forbes

Understanding SEO is the ultimate marketing discipline, especially in today's highly digital world. If you want to drive traffic to your site and increase sales, you ...


Best Ways To Improve Your SEO And Why You Need To Do It  Forbes Now

The more quality *content* you produce, the better results you will see.


These Are The Biggest SEO Trends Of 2019  Forbes

SEO strategy with components for successful marketing as icons on cubes on wooden backgroundGetty. How will the world of search engine optimization ...


SEO Firm Los Angeles Is A Must Have For Businesses To Stand Out Against Competitors - Press Release  Digital Journal

SEO firm Los Angeles is important to have behind you when seeking a business venture. Whether you are new to a business or have been at it for a while, SEO ...


Lee Pil Mo And Seo Soo Yeon Overflow With Love While Preparing To Be Newlyweds + Sing To Each Other At Wedding  soompi

Lee Pil Mo and Seo Soo Yeon have shown off their passionate love for each other once again! On the February 14 broadcast of TV Chosun's “Taste of Love,” ...


SEO Is A Must For Small Businesses: Here's How To Get Started  Forbes

You are starting a business and have a checklist of the things you need to get your business going. You have the idea for a product or *service* -- check.


5 SEO trends that will matter most in 2019  Search Engine Land

To be atop the waves, think about your SEO strategy in advance. A shortcut to success: get to know the upcoming trends and work out an action plan for each.


How SEO And Content Marketing Work Together To Fuel Your Online Success  Forbes Now

SEO and *content* marketing are not disparate departments. In fact, both need each other.


A checklist: Important SEO points to cover in a content campaign  Search Engine Land

After you've chosen the right execution for your link-building campaign and you're getting ready to launch it, you can't forget about search engine optimization ...


'Romance is a Bonus Book' preview: Eun-ho is great, but we definitely have Second Lead Syndrome for Seo-joon  MEAWW

We love rooting for Dan-yi and Eun-ho to get together on 'Romance is a Bonus Book', but we can't help but fall in love with Seo- joon.


How SEO Has Evolved (And What's Next)  Forbes

Since the early days of SEO, marketers have deployed online strategies to ensure that their websites would gain greater visibility on search engine results pages ...


SEO Industry Approaching $80 Billion But All You Want Is More Web Traffic  Forbes

Borrell Associates predicted in 2016 that by the year 2018 businesses in the United States would spend an estimated $72 billion in search engine optimization ...


The 12 Areas of SEO Knowledge You Must Master to be Successful  Search Engine Journal

The role of an SEO requires mastery of certain skills and knowledge. Here's what SEOs need to master for career advancement and campaign success.


The importance of understanding intent for SEO  Search Engine Land

Search is an exciting, ever-changing channel. Algorithm updates from Google, innovations in the way we search (mobile, voice search, etc.), and evolving user ...


12 SEO Goals for 2019: Ideas That Will Inspire You  Search Engine Journal

It's time to set your SEO goals for 2019. Need some inspiration? Read this roundup of New Year's goals from the SEO community.


Why Prioritization Is Critical to SEO Success  Search Engine Journal

Here's why prioritizing on-site and technical optimization, *content* creation and repurposing, and link building will set your SEO campaign up for success.


The Vicious Circle Of SEO (And How To Beat It)  Forbes

What's the secret to overtaking older *content* that already has a deep reservoir of backlinks? How can we break out of the vicious circle of SEO?


An SEO’s Letter to Santa: All I Want for Christmas This Year  Search Engine Journal

Dear Santa,. I've been a good boy this year. I haven't cloaked one page or bought one link. I have followed Google and Bing's terms of *service* the best I can.


6 Reasons Why SEO Might Not Be Right For Your Business  Forbes

Contrary to popular belief, SEO is not something every business needs, especially small businesses with modest marketing budgets. Here's how to determine ...


5 SEO Trends to Help Marketing Leaders Prepare for 2019  Forbes

As the year comes to a close, marketers have begun to wrap up their current campaigns and set their sights on the new year. And if 2018 has shown us anything, ...


How to Build and Protect Your Business' Reputation  Promotion World

The business world is full of challenges. In order to surpass them, a company needs to have certain measures and strategies in place that will help.


A Small Business Owner’s Guide to SEO  Search Engine Journal

Are you a small business owner? Then make sure you know and understand these 5 SEO principles and how they impact search and your SEO strategies.


Long Island SEO Company, fishbat, Shares 4 Tips For Promoting Holidays on Social Media to Stand Apart from the Crowd  PRNewswire

NEW YORK, Feb. 13, 2019 /PRNewswire/ -- As part of their continuing effort to teach companies how to make the most of their social media marketing, Long ...


SEO: Converting Keywords to Link Authority  Practical Ecommerce

Link authority is one of the most critical influencers of how a site ranks. How do keywords increase link authority? They indicate the *content* you need to create to ...


SEO Year In Review: 2018 saw Google updates and a focus on mobile  Search Engine Land

So, 2018 was an exciting year in SEO, with a number of large Google search algorithm updates, big changes within Google's management team, mobile first ...


Three Simple Steps To Seasonal SEO Success  Forbes

Seasonal SEO should work hand-in-hand with your long-term strategy.


5 Bad SEO Content Tactics You Should Have Abandoned Already  Search Engine Journal

Follow these tips to rank higher in the search results and avoid getting left behind with yesterday's outdated SEO *content* trends.


Why following SEO trends guarantees SEO budget waste.  Search Engine Land

Search engine optimization (SEO) can have a strong impact on a business, driving revenue and profits. Other times, it's frustrating and fruitless and feels like ...


home | site map
© 2006