Something for IT - Linux, security and programming

Recently I noticed that a new spider was going thru all of our sites, but there was something special about this one:

1. I couldn’t find any information for a web search engine, directory or anything similar that would explain a sort of legitimate usage of the recovered information.
2. It goes thru the site at an incredible page rate (about 120 files per minute…)

3. The spider seems to be poorly programmed. The logs show the spider going thru CSS, GIF, JPG and MPG files.

4. Some sites on the Internet mention a raise in the amount of spam after being visited by the psycheclone spider.

With an IP address lookup, I found that the requests come from McColo Corporation, with is assigned the range 208.66.192.0 to 208.66.195.255, so before taking any actions regarding psycheclone’s requests, I contacted McColo’s support service. Let’s see if they respond to my request. (At the time of this writing, there wasn’t an answer)

In the meantime, if you think psycheclone is a threat or problem for your server, you could block their requests with the following two methods:

Block McColo’s complete IP range using iptables

You could do this for port 80 (HTTP) or for all ports. Consider that this solution might keep you from providing services to legitimate users on that IP range.

To block all traffic:

iptables -I INPUT -s 208.66.192.0/255.255.252.0 \\
-j REJECT --reject-with icmp-port-unreachable

To block only port 80:

iptables -I INPUT -s 208.66.192.0/255.255.252.0 \\
-p tcp -m tcp --dport 80 -j REJECT \\
--reject-with icmp-port-unreachable

If you decide to keep the iptables rules, don’t forget to make them permanent!

Modify your PHP scripts to stop serving content to the spider

With this approach, you’ll have to modify your PHP scripts so psycheclone’s request is redirected to another server (in this case McColo’s):

[php]
/* Insert at the beginning of the PHP page */
$reqAgent = $_SERVER['HTTP_USER_AGENT'];
if (stristr($reqAgent, “psycheclone”) == TRUE) {
header(”Location: http://mccolo.com/english/contact.html”);
exit();
}
[/php]