What is robots.txt? [Archive] - Internet Life Forum

View Full Version : What is robots.txt?

eagleeye dental

09-11-2014, 05:49 AM

What is robots.txt?

comnetsolutions

09-15-2014, 10:21 AM

robots.txt file instruct web crawler how to crawl and index web page in website

Fli

09-16-2014, 09:57 AM

example of a crawler is "GoogleBot" (bot of Google search engine)

Fli

09-16-2014, 09:17 PM

Example robots.txt file contents can be found via search engine like google: robots.txt example