Commit message (Collapse) | Author | Age | Files | Lines | |
---|---|---|---|---|---|
* | - Update to 6.02 | Sunpoet Po-Chuan Hsieh | 2012-02-19 | 1 | -2/+2 |
| | | | | | | | Changes: http://search.cpan.org/dist/WWW-RobotRules/Changes Notes: svn path=/head/; revision=291809 | ||||
* | WWW::RobotRules parses /robots.txt files which are used to forbid conforming | Frederic Culot | 2011-04-14 | 1 | -0/+2 |
robots from accessing parts of a web site. The parsed files are kept in a WWW::RobotRules object, and this object provides methods to check if access to a given URL is prohibited. WWW: http://search.cpan.org/dist/WWW-RobotRules/ This new port is needed to update www/p5-libwww. Notes: svn path=/head/; revision=272722 |