use of robots.txt

This topic contains 1 reply, has 2 voices, and was last updated by  Virsen 1 month ago.

Viewing 2 posts - 1 through 2 (of 2 total)
  • Author
    Posts
  • #9917 Reply

    John
    Participant

    what is the use of robots.txt?

    #9931 Reply

    Virsen
    Participant

    Robot.txt consist of terminology such as user agent, disallow. The Website Owners inform crawlers that which pages of website they would not visit. The actual path of crawler traversing the website is initiated through robots.txt

Viewing 2 posts - 1 through 2 (of 2 total)
Reply To: use of robots.txt
Your information: