IMPORTANT: be very careful when playing about with your robots.txt, its very easy to block off your entire site without realising!…
Covered in this post:
- What is a robots.txt file?
- When should i use Robots.txt?
- How does it all work?
- Googlebot and My Robots.txt
- The X-Robots-Tag HTTP header
A Robots.txt is a file used by webmaster to advise spiders and bots (e.g. googlebot) where in the website they are allowed to crawl. The robots.txt is stored in a website’s root folder, for example: http://www.domain.com/robots.txt