robots.js 0.9.4

A Node.js library for parsing robots.txt files
Can be used to pull out data from the robots.txt file and feed it to other applications or statistics tools.

Will extract target URL, reason for access and status data.

last updated on:
June 4th, 2014, 4:54 GMT
price:
FREE!
developed by:
Eugene Kalinin
homepage:
github.com
license type:
MIT License
operating system(s):
Windows / Linux / Mac OS / BSD / Solaris
category:
C: \ Modules \ Other Modules

FREE!

In a hurry? Add it to your Download Basket!

user rating

UNRATED
0.0/5
 

0/5

Rate it!
What's New in version 0.9.2
  • Handles http(s) request errors.
read full changelog

Add your review!

SUBMIT