WordPress Robots.txt Sample

WordPress Robots.txt
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/
Adding Sitemaps to WordPress Robots.txt
User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

Sitemap: http://www.example.com/post-sitemap.xml
Explanation
Allowing all Bots

  • Allowing any Bots to Crawl

User-agent: *
Disallow:
Not Allowing any Bots

  • Not Allowing any Bots to Crawl

User-agent: *
Disallow: /

Block a Folder
User-agent: *
Disallow: /Folder/
Block a File
User-agent: *
Disallow: /file.html
 Block a page and/or a directory named private
User-agent: *
Disallow: /private
Block All Sub Folders starting with private
User-agent: *
Disallow: /private*/
Block URL’s end with
User-agent: *
Disallow: /*.asp$
Block URL’s which includes Question Mark (?)
User-agent: *
Disallow: /*?*
Block a File Type
User-agent: *
Disallow: /*.jpeg$
Block all Paginated pages which don’t have “?”  » Read more about: WordPress Robots.txt Sample  »