What is robots.txt and when to use it?

Robots.txt is a file placed on a website’s home directory that contains directives or instructions for well-known bots (search engine or other crawlers).