The Robots.txt file is used to restrict the search engine crawlers from indexing the selected (Private, confidential and admin pages)areas of your website.
Showing posts with label crawlers. Show all posts
Showing posts with label crawlers. Show all posts
Sunday, October 31, 2010
what is the usage of Robots.txt file?
The Robots.txt file is used to restrict the search engine crawlers from indexing the selected (Private, confidential and admin pages)areas of your website.
Subscribe to:
Posts (Atom)
