robots.js 0.9.4

A Node.js library for parsing robots.txt files

  Add it to your Download Basket!

 Add it to your Watch List!

0/5

Rate it!

What's new in robots.js 0.9.2:

  • Handles http(s) request errors.
Read full changelog
send us
an update
LICENSE TYPE:

MIT License

USER RATING:
UNRATED
  0.0/5
DEVELOPED BY:
Eugene Kalinin
HOMEPAGE:
github.com
LANGUAGE:
JavaScript
CATEGORY:
C: \ Modules \ Other Modules
Can be used to pull out data from the robots.txt file and feed it to other applications or statistics tools.

Will extract target URL, reason for access and status data.

Last updated on June 4th, 2014

Runs on: Windows / Linux / Mac OS / BSD / Solaris

requirements

#robots.txt parser #user agent #file parser #parse #robots.txt #file #read

Add your review!

SUBMIT