Welcome to "SEOCrazy"!!! Subscribe the Blog Feed for Latest SEO Tips and Techniques.
Enjoy your stay... :-)

Saturday, April 5, 2008

Robots.txt - Stop Search Engines To Access Your Private Files

The robots.txt file is a text file containing commands to the engine crawlers research to clarify their pages who may or may not be indexed. Thus any search engine began its exploration of a website seeking robots.txt at the root of the site.

Format robots.txt

The robots.txt (written in lower case and plural) is an ASCII file that are at the root of the site and may contain the following commands:

* User-Agent: allows you to specify the robot affected by the following guidelines.
* The value means "all search engines".
* Disallow: allows you to specify the pages to exclude from indexing. Each page or path to exclude must be on a line at hand and must begin with. The value / sole means "all pages."

The robots.txt file should contain no blank line!

Examples of robots.txt:

* Exclusion of all pages:

User-Agent: *
Disallow: /

* Exclusion of any page (equivalent to the absence of robots.txt, all pages are visited):

User-Agent: *
Disallow:

* Authorization of a single robot:

User-Agent: nomDuRobot
Disallow:
User-Agent: *
Disallow: /

* Exclusion of a robot:

User-Agent: NomDuRobot
Disallow: /
User-Agent: *
Disallow:

* Excluding one-page:

User-Agent: *
Disallow: / directory / path / page.html

* Exclusion of several page:

User-Agent: *
Disallow: / directory / path / page.html
Disallow: / repertoire/chemin/page2.html
Disallow: / repertoire/chemin/page3.html

* Exclusion of all pages of a directory and its subfolders:

User-Agent: *
Disallow: / directory /

4 comments:

muzamaml said...

Really informative post this was for me as I am learning SEO basics. Came here from DP and learned a lot from your blog. I will keep coming. Bookmarked it. thanks

Manish Chauhan said...

Thanks muzamaml...

I hope this blog could serve you some more eo information..!!

Thanks again for posting your views here.

batman_2004 said...

I am a very new user to blogs-extremely new-and my blog now seems to be hacked by a robots.txt something or other? Every time I go to my blog page, it forwards me on to the Dell homepage, which is extremely annoying. How do I stop this? Please keep in mind I have a poor understanding of blog technology. Thanks!

internet marketing services said...

i appreciate your post. It is very informative and very helpful on my research regarding seo techniques. thanks a lot.