GitHub - Sunama/simply-robotstxt: Parse a robots.txt file in ruby

Branches Tags

Name		Name	Last commit message	Last commit date
Latest commit History 16 Commits
lib		lib
spec		spec
test/data		test/data
.rspec		.rspec
Gemfile		Gemfile
Gemfile.lock		Gemfile.lock
Guardfile		Guardfile
README		README
Rakefile		Rakefile
robotstxt.gemspec		robotstxt.gemspec

Repository files navigation

robots.txt is an extremely simply format, but it still needs some parser loving. This class will normalise robots.txt entries for you.

USAGE
-----

With the following robots.txt:

User-agent: *
Disallow: /logs

User-agent: Google
Disallow: /admin

Sitemap: http://something.com/sitemap.xml

Use it like this:

require 'robotstxtparser'

# Also accepts a local file
rp = RobotsTxtParser.new()
rp.read("http://something.com/robots.txt")

rp.user_agent('Google') # returns ["/logs", "/admin"]
rp.user_agent('Autobot') # returns ["/logs"]

rp.sitemaps # returns ["http://something.com/sitemap.xml"]