PHP web crawler - Codewalkers

Codewalkers Forums Sponsor:

Warn humeroushubris · View humeroushubris's Warnings · View your Warnings · #1

January 8th, 2013, 04:46 PM

humeroushubris

Registered User

Join Date: Jan 2013

Posts: 1

Time spent in forums: 1 h 48 m 3 sec
Reputation Power: 0

noob - PHP web crawler

I have a basic PHP web crawler script and I need to expand its functionality, the problem is I'm a total noob at PHP and my knowledge is very basic so I'm coming here for some help.

My goal is to have a basic user input (text box) and when the user types in a phrase; let's say "Red Apples" and hits the enter button the script should start crawling the web for the phrase "Red Apples" and store the plain text results along with the URL they originated from in a database.

Here is what I've got so far:

Code:

error_reporting( E_ERROR );
 
define( "CRAWL_LIMIT_PER_DOMAIN", 50 );
 

$domains = array();

$urls = array();
 
function crawl( $url )
{
  global $domains, $urls;
 
  echo "Crawling $url... ";
 
  $parse = parse_url( $url );

  $domains[ $parse['host'] ]++;
  $urls[] = $url;
 
  $content = file_get_contents( $url );
  if ( $content === FALSE )
  {
    echo "Error.\n";
    return;
  }
 
 
  $content = stristr( $content, "body" );
  preg_match_all( '/http:\/\/[^ "\']+/', $content, $matches );
 
  echo 'Found ' . count( $matches[0] ) . " urls.\n";
 
  foreach( $matches[0] as $crawled_url )
  {
    $parse = parse_url( $crawled_url );
 
    if ( count( $domains[ $parse['host'] ] ) < CRAWL_LIMIT_PER_DOMAIN
        && !in_array( $crawled_url, $urls ) )
    {
      sleep( 1 );
      crawl( $crawled_url );
    }
  }
}

If anybody could point me in the right direction that would be awesome.

Warn DavidMR · View DavidMR's Warnings · View your Warnings · #2

January 9th, 2013, 06:33 AM

DavidMR

Contributing User

Codewalkers Beginner (1000 - 1499 posts)

Join Date: Apr 2007

Location: Galway

Posts: 1,369

Time spent in forums: 1 Month 4 h 23 sec
Reputation Power: 8

with that code sample I dont see how you can achieve a search facility. how about parsing a google search instead?

__________________
When I die, I want to go peacefully like my Grandfather did, in his sleep -- not screaming, like the passengers in his car.

Warn IAmALlama · View IAmALlama's Warnings · View your Warnings · #3

January 9th, 2013, 01:07 PM

IAmALlama

Join Date: Apr 2007

Location: San Diego, CA

Posts: 2,267

Time spent in forums: 2 Weeks 1 Day 6 h 30 m 38 sec
Reputation Power: 9

do you have a specific list of pages you want to crawl, or is the objective to crawl the entire web? sounds like what you are creating is essentially google.