Spiders txt bug fix

When the code in includes/application_top.php was written, it was assumed that spiders and bots do not accept cookies. Today, this is no longer true and more and more spiders and bots are accepting cookies in the search for more web content.

When a bot accepts cookies it gets a session and next thing you know it is adding products to your shopping cart. In the worst case the spider or bots can get lost in your shopping cart going round and round in circles. To see this happening I recommend you install the Who's Online Enhancement (homepage The fix to this is simple :-

in includes/application_top.php find the following code :-

$session_started = false;
if (SESSION_FORCE_COOKIE_USE == 'True') {
tep_setcookie('cookie_test', 'please_accept_for_session', time() 60*60*24*30, $cookie_path, $cookie_domain);
if (isset($HTTP_COOKIE_VARS['cookie_test'])) {
tep_session_start();
$session_started = true;
}
} elseif (SESSION_BLOCK_SPIDERS == 'True') {
$user_agent = strtolower(getenv('HTTP_USER_AGENT'));
$spider_flag = false;

if (tep_not_null($user_agent)) {
$spiders = file(DIR_WS_INCLUDES . 'spiders.txt');

for ($i=0, $n=sizeof($spiders); $i<$n; $i ) {
if (tep_not_null($spiders[$i])) {
if (is_integer(strpos($user_agent, trim($spiders[$i])))) {
$spider_flag = true;
break;
}
}
}
}
if ($spider_flag == false) {
tep_session_start();
$session_started = true;
}
} else {
tep_session_start();
$session_started = true;
}

and replace it with the following code :-

$spider_flag = false;
if (SESSION_BLOCK_SPIDERS == 'True') {
$user_agent = strtolower(getenv('HTTP_USER_AGENT'));

if (tep_not_null($user_agent)) {
$spiders = file(DIR_WS_INCLUDES . 'spiders.txt');

for ($i=0, $n=sizeof($spiders); $i<$n; $i ) {
if (tep_not_null($spiders[$i])) {
if (is_integer(strpos($user_agent, trim($spiders[$i])))) {
$spider_flag = true;
break;
}
}
}
}
}

$session_started = false;
if ($spider_flag == false) {
if (SESSION_FORCE_COOKIE_USE == 'True') {
tep_setcookie('cookie_test', 'please_accept_for_session', time() 60*60*24*30, $cookie_path, $cookie_domain);
if (isset($HTTP_COOKIE_VARS['cookie_test'])) {
tep_session_start();
$session_started = true;
}
} else {
tep_session_start();
$session_started = true;
}
}

Thats it!

If they ever change the spiders.txt file to a php array, then it would improve the effiency in a huge way, but until that happens this is the best way to catch all spiders!

Please note: the above code does not ignore comment lines in spiders.txt (even the original code didn't). I would recommend deleting the comments out of the spiders.txt file to speed things up a touch.