PDA

View Full Version : What is robots.txt?


eagleeye dental
09-11-2014, 04:49 AM
What is robots.txt?
comnetsolutions
09-15-2014, 09:21 AM
robots.txt file instruct web crawler how to crawl and index web page in website
Fli
09-16-2014, 08:57 AM
example of a crawler is "GoogleBot" (bot of Google search engine)
Fli
09-16-2014, 08:17 PM
Example robots.txt file contents can be found via search engine like google: robots.txt example