Information on the robots.txt Robots Exclusion Standard and other articles about writing well-behaved Web robots.
robots.txt（统一小写）是一种存放于网站根目录下的ASCII编码的文本文件，它通常告诉网络搜索引擎的漫游器（又称网络蜘蛛），此网站中的哪些内容是不能被搜索引擎的 ...
The robots.txt standard was developed in 1994, when large-scale web indexing became popular; indexers such as Lycos and AltaVista used it. ...
User-agent: * Crawl-delay: 10 Sitemap: http://www.whitehouse.gov/feed/media/
Learn about the robots.txt, and how it can be used to control how search engines and crawlers do on your site.
robots.txt 文件限制抓取网络的搜索引擎漫游器对您的网站的访问。这些漫游器是自动的，它们在访问任意网站的网页之前，都会查看是否存在阻止它们访问特定网页 ...
A robots.txt file restricts access to your site by search engine robots that crawl the web. These bots are automated, and before they access pages of a site ...
Robots.txt is a text (not html) file you put on your site to tell search robots which pages you would like them not to visit. Robots.txt is by no means ...
robots.txt for http://www.w3.org/ # # $Id: robots.txt,v 1.59 2010/01/29 15:52:50 ted Exp $ # # For use by search.w3.org User-agent: W3C-gsa Disallow: ...
