Control search engine indexing with robots.txt

If you wish to restrict all or part of your website from being indexed by various search engine robots you can use a robots.txt file. For it to work properly it should be a simple ASCII text file named exactly “robots.txt” and it should be placed in the domain root directory. The well behaved robot [...]