Quick Notes on Crawler_Googlebot_robots, Feedfetcher and Meta tags


Quick points and a handy Notes to manage Google SEO best practices with Crawler, Googlebot, robots, Feedfetcher, and Meta tags, 

  1. "Crawler" is a universal term for any robotic program used to automatically discover and scan websites by following links from one webpage to another.
  2. "Googlebot" is Google's web crawling bot it is also called a "spider". IT is nothing but a very well written and advanced program by Google.
  3. Crawling is the process by which Googlebot discovers new and updated pages to be added to the Google index.
  4. Googlebot uses an algorithmic process to determine which sites to crawl, how often, and how many pages to fetch from each site.
  5. Crawlers usually are located near to the sites they're indexing in the network.
  6. Crawling rates depend on multiple factors, If you see Crawling is more frequent you can request a change in crawl rate with Google.
  7. We can prevent Googlebot from crawling content on your site with many options as per Google's documentation.
  8. Verify in regular intervals if web crawler accessing your server really is Googlebot.
  9. Googlebot identifies itself with a user-agent string, but this can be spoofed; the best way to identify accesses by Googlebot is to use a reverse DNS lookup.
  10. If you find information in Google's search results as spam, paid links, or malware, Please report to Google.
  11. Google grabs RSS or Atom feeds for Google Play Newsstand and PubSubHubbub using Feedfetcher. Feedfetcher collects and periodically refreshes these user-initiated feeds, but does not index them in Blog.
  12. There are many Google crawlers we may see in referrer logs, and how they should be specified in robots.txt, the robots meta tags, and the X-Robots-Tag HTTP directives. (More Read on Common Google crawlers).
  13. The robots.txt file is how search engine crawlers crawl pages or files the crawler can or can't request from your site.
  14. Meta tags are the best way for a webmaster to provide search engines with relevant information about sites and Contents.


I will try to enhance this article at frequent intervals when I learn something new about this topic.

Leave a comment

Make sure you enter all the required information, indicated by an asterisk (*). HTML code is not allowed.