Building A Web Spider – Part 1

Whether you want to gather data for a database web site, or determine which links are broken in your site a web spider is a great tool for getting repetitive tasks done. Scraping web pages from the internet is a great way to gather content and ideas for web sites that you want to build. You can build a web spider to check the current prices of your favorite stocks, or even to get the balance of your bank account.

I use web spiders all the time to build databases for my database sites… I have even created them to get my email for me and respond if certain conditions are met. The secret to understanding how to build a great web spider is to first understand the environment that it must work in.

What do I mean by this? Well first you must understand that a web spider is simply a computer program. And computer programs are based upon rules (wait didn’t I hear this in the MATRIX) and these programs will blindly follow these rules until we tell them to stop. So for us to have our internet spider complete the task that we ask of it, we first must create the rules that it will follow. And of course for us to create the rules we must first understand the environment.

To illustrate this point, let’s say that we want to create a web spider to download our favorite stock quote from Yahoo (substitute MSN, or Etrade if you like). Let’s write some pseudo-code for the way we would do this manually:

1. Goto the main page of Etrade.com
2. Fill-in form on front page with our stock symbol – MSFT
3. Hit Enter
4. Look to see the stock price which is in large font and red

Now let’s see what kind of question may arise when we start creating a web spider from this simple process.

1. Can I bypass the form by entering a dynamic url (say finance.yahoo.com/stockquote.asp?stocksym=MSFT)?
2. If I have to use the form, what are the form variables that I need to send and how (GET vs POST)?
3. Do I need to store cookies or session variables?
4. How do I get the stock price from the web page after downloading?
5. the list goes on and on.

You notice that this simple example of getting a stock quote has some questions that need to be answered before creating the rules of your web spider. And the best way to get these answers is by going directly to the source… the web page you are trying to scrape. So let’s get started!

To really figure out how a web site works we are going to need to look at the HTML code as well as the urls that appear in your navigation bar of your browser. So that is where we are going to start.

Navigate to Yahoo finance by going HERE

You will notice the base url is finance.yahoo.com, you can find this at the top of the browser. Next fill in the form for the stock that you want to look up (MSFT) and hit the enter button. You will notice now that the url has change to:

http://finance.yahoo.com/q?s=msft

Apparently the portion of the url that changes the stock to viewed is contained in the “q?m=” portion of the url. Therefore, I can look up any stock that I choose by putting a different value behind the “q?m=” portion. In this case it is very simple to create a spider to look up a lot of stocks by feeding different stock symbols into this url and the downloading the page.

So how do you download a page from the internet? Well PHP has a lot of really cool ways of doing this. I am only going to show you one quick way now. That is to use PHP’s file_get_contents() function.

Let’s say we want to get Microsoft’s stock page. Use the following code:

<?

$url = “http://finance.yahoo.com/q?m=MSFT”; // This is the url for Microsoft

///// The next line will download the HTML and put it into a variable called $page

$page = file_get_contents($url);

echo $page; /// will print the html onto your page

?>

The file_get_contents function might be disabled on certain servers. Hostgato.com is where I have my hosting accounts and I know that it works there. I used to use Godaddy.com and I know that their cheaper shared hosting won’t allow for using the file_get_contents() function.

In my next post, I am going to show you how to extract the values that you want from the page that you just downloaded. So until then play around with the above code by changing the value of the stock symbol.
Stumble it!

**************************************************************************

* Looking for a comprehensive course on Web Page Scraping?

* Let me know your interest by commenting on the SpyderSchool Post

* ************************************************************************

Advertisement