Robots.txt is common name of a text filethat is uploaded to a Web site's root directoryand linked in the html code of the Web site. The robots.txt file is used to provide instructions about the Web site to Web robotsand spiders. Web authors can use robots.txt to keep cooperating Web robots from accessing all or parts of a Web site that you want to keep private
No comments:
Post a Comment