ReferrerCop 1.1.0 review

ReferrerCop project parses Apache log files or AWStats data files and removes entries for referring URLs that match a list of known r

License: GPL (GNU General Public License) File size: 77K Developer: Ryan Grove

0

ReferrerCop project parses Apache log files or AWStats data files and removes entries for referring URLs that match a list of known referrer spammers.

It then returns the filtered files via standard output. Run your logs through ReferrerCop either before or after they’re processed by AWStats (or before they’re processed by any other log analyzer) to eliminate annoying referrer spam from your web statistics.

Filtering is performed using a blacklist and an optional whitelist. The lists are simply text files containing regular expressions (one pattern per line). ReferrerCop comes with a large, heavily optimized and frequently updated blacklist.

ReferrerCop is extremely fast and uses very little memory; it is capable of filtering files of virtually any size. On my Athlon XP 1800 running FreeBSD 5.4, ReferrerCop is capable of processing about 14,000 Apache log entries per second (your mileage may vary).

Requirements:
Ruby 1.8.2 or higher

Usage

referrercop [-f | -i | -n | -s] [options] [< file > ...]
referrercop -u < url > [options]
referrercop {-h | -V}

Modes:

-f, --filter Filter the specified files (or standard input if no files are specified), sending the results to standard output. This is the default mode.
-i, --in-place Filter the specified files in place, replacing each file with the filtered version. A backup of the original file will be created with a .bak extension.
-n, --extract-ham Extract ham (nonspam) URLs from the input data and send them to standard output. Duplicates will be suppressed.
-s, --extract-spam Extract spam URLs from the input data and send them to standard output. Duplicates will be suppressed.
-u, --url < url > Test the specified URL.

Options:

-b, --blacklist < file > Blacklist to use instead of the default list.
-v, --verbose Print verbose status and statistical info to stderr.
-w, --whitelist < file > Whitelist to use instead of the default list.

Information:

-h, --help Display usage information (this message).
-V, --version Display version information.

What's New in This Release:
The format of blacklists and whitelists has changed slightly.
Performance with large lists has been improved.
The blacklist update feature is now more reliable.