What Is Googlebot And Why Should You Care?

This post is about Other Search Engines, SEO / 4 minutes

This is a guest post by Jon from LocalSEO.org where he explains how Googlebot and why it matters to rank your website. I accepted this contribution in 2011, when IAWSEO was still called N0TSEO and still covered plenty of regular SEO topics. This post is helpful to readers, so I’m leaving it as is — with the addition of some funny images because I like to make readers smile while they read.

Googlebot is Google’s site crawling robot: it finds pages on the web and passes them off to the Google Algorithm for processing.

In reality Googlebot doesn’t crawl the web at all. It works like a web browser, it sends a request to a server for a web page, downloads the whole page, and then moves on to the next page.

Googlebot consists of numerous computers requesting and fetching pages faster than you can with your web browser.

In fact, Googlebot can request thousands of different pages in chorus. To evade overpowering web servers, or crowding out requests from human users, Googlebot intentionally makes requests of each individual web server slower than its capacity.

How Does Googlebot Work?

While Googlebot fetches a page, it weeds out all the links appearing on the page and adds them only to what it considers are high-quality pages. By harvesting links from every page it encounters, Googlebot can rapidly make a list of links that can cover the web.

This method, known as deep crawling, also allows Googlebot to search deep within individual sites. Due to their massive size, deep crawls can reach more or less every page online.

You may also want to read  SEO for Businesses. The Essentials in Less than 1,000 Words

Because the web is enormous this process can take some time, so some pages may be crawled only once a month.

Googlebot was intended to be distributed on several (thousands) of machines to improve performance and scale as the web grows.

Also, to cut down on bandwidth usage, they run numerous crawlers on machines located near the sites they’re indexing in the network.

As a result, your logs may show visits from quite a few machines at google.com, all with the user-agent Googlebot. The goal is to crawl as many pages from a site as possible on each visit without overwhelming the user’s server bandwidth.

Even though its purpose is simple, Googlebot must be programmed to handle several challenges.

First, since Googlebot sends out real-time requests for thousands of pages, the queue of “visit soon” URLs must be constantly examined and compared with URLs already in Google’s index. Duplicates in the queue must be purged to avoid Googlebot fetching the same page again. Googlebot must decide how frequently to re-examine a page.

On one hand, it’s a waste of resources to re-index an untouched page. On the other hand, Google wants to re-index altered pages to deliver up-to-date results.

Googlebot shouldn’t access a site more than once every few seconds. However, due to network delays, it’s possible that the rate will appear to be slightly higher over short periods.

In general, Googlebot ought to download only one duplicate of each page at a time. If you see that Googlebot is downloading a page multiple times, it’s probably because the crawler was stopped and restarted.

You may also want to read  3 Essential Steps To Free Your Blog Of Google's Influence

Versions of Google Bot:

Googlebot has two versions namely Deepbot and Freshbot:

Freshbot:

Freshbot crawls the website and looks for fresh content. Freshbot visit the website that frequently change, depending on how often.

Deepbot:

Deepbot tries to follow every link on a website and download as many pages as possible of your website. This process will be completed about once a month.  But, may be slower if your site is stagnant and not often updated.

Conclusion

Now that you have a great basic understanding of what Googlebot is and how it works it is important to make the most of this situation.

For example, knowing that Freshbot exists it would be a very good idea to update your site frequently so that Google sends Freshbot your way.

All things equal a site that is updated more often will be ranked higher than one that isn’t touched by its owner.

So, start updating your site with high quality fresh content and Google will reward you.

Jon is a local SEO expert at LocalSEO.org.  He works with small to medium-sized business owners to provide them high rankings and quality traffic via Google Places and Organic SEO.

Originally posted on: Written on September 22, 2011, Thursday

Luana Spinetti is a B2B blogger and artist for hire based in Italy. She has been in business since 2009. When she's not busy writing or drawing for herself or her clients, you may find her reading a Sci-Fi book or scientific articles on robotics and Computer Science. Get in touch with Luana on Twitter (@LuanaSpinetti), Instagram (@luanaspinetti) or Goodreads.

Leave a Reply

Your email address will not be published.

The following GDPR rules must be read and accepted:
This form collects your name and content so that we can keep track of the comments placed on the website. However, IP addresses are anonymized and emails are NOT required. This is done to maintain your privacy and ensure that no personal data is stored in our systems. For more info check our privacy policy where you will get more info on where, how and why we store your data.

Looking for older posts? Check the Articles archive!