There are times when you may want to block search engines (or other robots) from including certain directories or files in search engine results. One way to do this is to create a text file called a robots file. You can use Notepad or any other text editor to create the file.
If you wanted to block robots (like Google, Yahoo, Bing, etc) from everything on your website you can use:
User-agent: *
Disallow: /
Is that it? Yes. Simply copy and past that into a blank text file and upload the file to your top-level directory. You can test it by accessing your robots.txt file @ www.yourdomain.com/robots.txt
When the bots visit your page they will read this file and get instruction.
Below, I’ve included some other examples of usage:
