robots.txt 2.0 review

Download

robots.txt is a script in PHP that acts like a normal robots.txt file, but with a few differences. When a Spider attempts to access

License:	GPL (GNU General Public License)
File size:	0K
Developer:	Andrew Smith

robots.txt is a script in PHP that acts like a normal robots.txt file, but with a few differences.

When a Spider attempts to access robots.txt, the script will "disallow" access to a list of pre-defined directories. When a normal user attempts to access robots.txt, it will "disallow" access to all directories.

This is a security precuation and an attempt to prevent users from finding "hidden" directories by looking at the robots.txt file.

What's New in This Release:
The two serious bugs that meant robots.txt did not function properly were fixed.
A bug that meant the data printed back would not be parseable by Web crawlers was fixed.
A new function to clean up data from the files was added.

robots.txt 2.0 screenshot
Zoom

robots.txt 2.0 search tags

robots.txt 2.0: robots.txt is a script in PHP that acts like a normal robots.txt file, but with a few differences. When a Spider attempts to access
Robocode Robots 1.0: Robocode Robots project is a collection of robocode robots. This project is a collection of robots for the robocode framework
bot-trap 0.92: bot-trap allows your Web site to automatically ban bad Web robots (a.k.a
Riding With Robots 0.1: Riding With Robots is a SuperKaramba theme to ride along with the armada of robotic space probes now exploring the Solar System. I
Ultratron 2.2: Ultratron is an arcade game where you have to destroy the four evil robots of the Apocalypse! The last human has been slain by evi
RealTimeBattle 1.0.8: RealTimeBattle project is a a robot programming game for Unix. RealTimeBattle is a programming game, in which robots controlled by
Template::Tutorial 2.15: Template::Tutorial are template toolkit tutorials. This section includes tutorials on using the Template Toolkit

robots.txt 2.0 review

Alternative/similar