SEO Tip #7 - Robots.txt File

robots.txt file, is an instruction file for the SE Crawlers.

To create robots.txt file, open a notepad or any other text editor and save it as robots.txt - lower case only! Upload the file to the root directory. (www.yourdomain.com/robots.txt)

There are only 2 rules you can assign to the SE crawlers:
User-agent: define specific search engine robot
Disallow: the pages you want to block

User-agent:
If you want your robots.txt file will be for all SE spiders just type:

User-agent: *

asterisks means all search engines.


Disallow:
Use this to tell the crawlers/spiders where they are not allow to crawl.
it looks like that:
Disallow: /file.html [Don't visit the file file.html]
Disallow: /abc/ [Don't visit any file under the folder abc]
Disallow: [leave blank - spiders allowed to crawl to all pages
Disallow: / [spiders disallowed to crawl to all pages

Best place to read more about robot.txt file
is Here