- This topic has 20 replies, 17 voices, and was last updated 3 years, 9 months ago by
.
-
Posts
-
The robots.txt file is primarily used to specify which parts of your website should be crawled by spiders or web crawlers. It can specify different rules for different spiders.
Googlebot is an example of a spider. It’s deployed by Google to crawl the Internet and record information about websites so it knows how high to rank different websites in search results.
Using a robots.txt file with your website is a web standard. Spiders look for the robots.txt file in the host directory (or main folder) of your website.
A robots.txt file notifies search engine crawlers which URLs on your site they can access. This is mostly intended to prevent your site from becoming overburdened with requests; it is not a strategy for keeping a web page out of Google. You should use noindex or password-protect a web page to keep it out of Google’s index. A robots.txt file is used to regulate crawler traffic to your site and, depending on the file type, to keep a file off Google.
- You must be logged in to reply to this topic.