rubygem-webrobots helps writing robots.txt compliant web robots in Ruby.