Project Detail

Description

Hello,

I´m currently running a crawler that uses the phpcrawler php class which works fine with almost all pages/urls, but with certain pages there occurs a problem which basically is that the script stops working without giving any error message or anything like that.

Now an example of a page on such a problem occurs would be:
http://www.ejk-kajak.de

Now here would be a demo code, which would simply stop - always at the same page (teamer.htm):


include("classes/phpcrawler.class.php");

class MyCrawler extends PHPCrawler
{
function handlePageData($page_data)
{
echo "Page requested: ".$page_data["url"]."<br>";
echo "Status: ".strtok($page_data["header"], "\n")."<br>";
echo "Referer-page: ".$page_data["referer_url"]."<br>";
if ($page_data["received"]==true)
echo "Content received: ".$page_data["bytes_received"]." bytes";
else
echo "Content not received";

echo str_pad(" ", 5000); // "Force flush", workaround
flush();
}
}



// Klasse laden
$crawler = new MyCrawler();

$crawler->setURL("http://www.ejs-kajak.de");
$crawler->setFollowMode(1);
$crawler->addReceiveContentType("/text\/html/");
$crawler->addNonFollowMatch("/.(jpg|gif|png)$/ i");
$crawler->setCookieHandling(true);
$crawler->setTrafficLimit(0);
$crawler->setPageLimit(0);
$crawler->go();
$report = $crawler->getReport();
print_r($report);


Now the job would be to fix the class, so that this doesn´t happen any more.