Crawl and Search Bots Optimisation

Ways you could control Search Bots

  1. robots.txt
  2. Header Meta Tag
  3. Using X-Robots-Tag
  4. Change HTTP Status code ( Eg:  403 – forbidden ..etc )

Optimise Google Crawl Efficiency

X-Robots-Tag HTTP Header


Robots Meta Tag

<meta name=“robots” content=“noindex”>

Specify a Search Bot

<meta name=“googlebot” content=“noindex”>

Few Example Search Bots

Googlebot Mobile
Yandex - Russia
baiduspider - China
Sougou+web+spider - China
YoudaoBot - China
Slurp - Yahoo
Exabot - France

Robots Meta Tag Values and Search Engine Support

Robots value Google Yahoo! MSN / Live Ask
index Yes Yes Yes Yes
noindex Yes Yes Yes Yes
follow Yes Not Sure Not Sure Yes
nofollow Yes Yes Yes Yes
none Yes Not Sure Not Sure Yes
noarchive Yes Yes Yes Yes
nosnippet Yes No No No
noodp Yes Yes Yes No
noydir No Yes No No


Differerence between Robots.txt and X-Robots-Tag Header and Robots Meta Tag

  • Robots.txt is Crawler Directive ( Sets Direction for the crawler )
  • X-Robots-Tag is Indexer Directive ( Set on a per page and/or per element basis )
  • X-Robots-Tag and Meta Robots Tag are similar but X-Robots-Tag has more flexibility compared to Robots Meta Tag


  • Robots.txt – Powerful
  • Robots Meta Tag – More Powerful
  • X-Robots-Tag HTTP Header- Most Powerful

Example Situation

  • Robots.txt – Basic Site level and Page Level
  • Robots Meta Tag – HTML Page Level
  • X-Robots-Tag HTTP Header – Includes Various File types also ( Eg: PDF, image ..etc )

Block a certain Data Type