What is robots.txt and when to use it?

Robots.txt is a file containing directives to disallow crawling for well-known bots (search engine & others). Here's how and when you …