PDA

View Full Version : What is robots.txt?


himanshu
06-18-2019, 07:39 AM
What is robots.txt?
Mihirthakur
06-18-2019, 08:33 AM
Robots.txt is used to instruct bots How to crawl the web pages
sohailkhan
06-18-2019, 09:46 AM
Robot.txt is mainly used to instruct the web bots how to crawl the web pages
ajay_11
06-19-2019, 10:38 AM
Robots.txt is a text file webmasters use to instruct google bots to crawl the website
suhel
06-20-2019, 07:40 AM
Robots.txt is used to instruct bots How to crawl the website
Raju
10-03-2019, 12:31 PM
Robots.txt is a text file webmasters create to instruct web robots how to crawl pages on their website.