SEO Tutorial - robots.txt

robots.txt file tells search engines whether they can access and therefore crawl parts of your site. This file, which must be named robots.txt, is placed in the root directory of your site.

You may not want several pages of your site crawled because they might not be useful to users if found in a search engine's search results(Mostly any pages behind a login page). If you do want to prevent search engines from crawling your pages, Google Webmaster Tools has a friendly robots.txt generator to help you create this file. Note that if your site uses subdomains and you wish to have certain pages not crawled on a particular subdomain, you'll have to create a separate robots.txt file for that subdomain.

Add comment

  • Comment
  • Preview

Recent Posts