Spyder Web Tech's SEO Journey

Announcing the Future Launch of the “SpyderSchool”

spyderwebtech — Fri, 30 Jan 2009 17:10:36 +0000

Coming Soon - SpyderSchool Launch

I want to thank everyone who left a comment, emailed me, and even took the time to track me down and give me a phone call about starting this project. It seems as if there is a real need for this subject and to my knowledge there is yet to be a definitive source of web automation information on the net.

I am very happy to announce that I am going forward with the SpyderSchool, an online school on how to Automate the Internet. The scheduled release is tentatively planned for June 1st, 2009.

This school will teach its members how to spider, scrape, mine data, and automate nearly any site or process on the net. I will teach you how to do this in step-by-step videos, tutorials, and live seminars.

I am limiting the number of students that I will be accepting to the SpyderSchool for the first year to 250. So if you are interested, please fill the form linked below as soon as possible.

Put me on the Waiting List for the SpyderSchool

The first 250 people who sign up will be contacted first, and the if there are still openings then I will continue down the list until all spots are filled. Only then will I open up the enrollment, so don’t wait! Even if you are only remotely interested, you had better sign-up.

In the SpyderSchool you will learn-by-doing with hands on examples, data-mining challenges, and competitions to test yours skills and hone your newly found techniques. As you complete each challenge, you will be building a library of web scraping code that you can use for your future career in web automation as well as a portfolio to impress your future clients.

Don’t have any experience programming for the web? No worries, the SpyderSchool is being built for the beginner.

In the SpyderSchool, you will learn:

How the Internet Works
What Types of Scraping Technologies Exist
How to Analyze a Web Site for Scraping
Basic/Advanced Web Spider Programming
What Tools to Use
How to Automate Your Web Hosting Accounts (Linux Servers)
How to Automate Forms
How to Automate AJAX Dynamic Content Sites
How to Beat Captcha
and Much, Much, More.

This course we be taught using only open-source technologies, sorry Microsoft, so you won’t have to pay for one single solitary thing besides your tuition to the course. There will be no up-sells, down-sells, cross-sells, or any other marketing pressure… guaranteed. I hate that crap, nothing but learning here.

Be looking forward to more posts on the progress of the SpyderSchool. And be sure to sign-up for more information at the following location:

Put me on the Waiting List for the SpyderSchool

You can also follow the progress of this school at my Twitter:

http://www.twitter.com/spyderwebtech

Thanks again for the huge amount of interest, and for those who took the time to encourage me to start this project.

–Spyderwebtech

Scraping Web Pages with cURL Tutorial – Part 2

spyderwebtech — Wed, 13 Aug 2008 13:29:24 +0000

In Scraping Web Pages with cURL Tutorial – Part 1, I demonstrated how to create a web spider class that uses the cURL library to transfer any type of data from the web direct to your server.

In this tutorial we are going to talk about how to parse that data into some sort of usable form by extending our wSpider class functionality.

The key to scraping web pages is to first understand how a web page is laid out and it’s resulting HTML structure. It is this HTML structure that allows our spider to identify and scrape the part of the web page that you are interested in. So let’s take a look at some example HTML and review what kind of HTML tags we may encounter with our spider.

Below is an example of the HTML that you might see on a typical web page:
My Web Page Title

`Header 1`



















Goto Next Page

As you can see that each part of the web page is encompassed between and opening tags ( <..> ) and closing tags ( ) . Every web page will have these two primary sections:

section – The content between the opening and closing tags provide information about the website/web page including title, keywords, doctype, ect.
section – This section contains all of the visual items of a web page including text, images, links, tables, and a container type item called a DIV.

Most of our attention will be focused on the section as this is where a lot of the “juicy” content that we might what to scrape. In a later tutorial, I will show you how to create a competition analysis spider working mostly in the section.

Within the tags of our example website we find the

,
, , and tags which we will pass onto our spider to scrape either the content between these tags OR a attribute of these tags. For example, the link tag has information between the opening and closing tag as well as a HREF property which will tell us the URL that will be navigated when the link is clicked.

So let’s extend our wSpider class that we began to build in part 1 of this tutorial, by creating a function that will take the HTML stored in our $this->html string and strip out these different tags. I am going to call this function parse_array because I want to take all the occurrences of the supplied tag and then store it into an array.

Below is the code for our parse_array() function:

`function parse_array($beg_tag, $close_tag) { preg_match_all("($beg_tag.$close_tag)siU", $this->html, $matching_data); return $matching_data[0]; }`

The above function takes two parameters which are the beginning tag ($beg_tag) and the ending tag($close_tag) and then will capture everything in between these tags. This will happen for each and every time the program finds this particular configuration.

The real workhorse of this function the preg_match_all function that is part of PHP. The preg_match_all function takes a regular expression and searches a string for all of the occurrences and then extracts them into an array for us (in this case an array called $matching_data).

The regular expression that we use is the “($beg_tag.$close_tag)siU” portion of the function above which starts by looking at the beginning tag supplied by the user. The next part of the expression is the .* which is sometimes called the “greedy” because the . will match anything and the * will grab as much as it can. The last part of the expression is the closing tags, once again supplied by the user, and then some flags that will ignore case which we will talk about in a later tutorial on regular expressions.

Once all this work is done the function will return the array back to where we called it from.

Let’s take a quick look on how we can uses our new parse_array() function to scrape all of the links from our example HTML above.

The first thing we need to know is how is a link structured in HTML so we can send our function the beginning tag and the end tag. A link’s structure looks like this from our example:

`Goto Next Page`

I can see that the link begins with a “”. If I uses these as my start and end tags in my function then the function should return an array with one element and inside that element will be the entire HTML for the link above.

Putting it all together, below is the code that I would use to download all the links from my example web page. I am also going to put in a foreach loop that will be used to do whatever I want to with each link. This loop will be important to us when we create a spider that can crawl multiple web pages and website without further user interaction.

$myspider = new wSpider();
$myspider->fetchPage(“http://wwww.examplesite.com/example.html”);
$linkarray = $myspider->parse_array(“”);

foreach ($linkarray as $result) {

/// store each $result in database or create a new spider to spider next page

}

As you can see from the code above we create a wSpider instance by using the new keyword. Next we get the HTML by using the fetchPage() function and we set the target URL in the constructor of the function (see part 1 of this tutorial if you don’t understand). From this function we store the resultant HTML in the $this->html variable within the wSpider class instance.

The next line of code assigns the result of our parse_array() function to another variable called $linkarray which we use in a foreach loop at the bottom of this code. Remember, what has been returned into the $linkarray array is the entire HTML of the link including the tags, so depending on which type of information you want you are going to have to strip out that information.

PHP has a lot of great functions that can help us grab what information we want. For instance, if we wanted to grab the anchor text of these links we could use the strip_tags() function in PHP to grab the text between the opening and closing tags (this can be used for anything that has an opening and closing tag).

Once we have the information we want then we can do a number of things such as save the information to a database, spawn a new spider to scrape the next page, or to record what the anchor text is for each link (for SEO purposes) as explained above.

In the next tutorial, we will continue to develop our spider class to include a number of functions that will allow us to make repetitive tasks very simple, and how to use these new functions to create a simple spider that can crawl an entire website.

Other Web Spider Tutorials:

Build a Web Spider – Part 1

Building a Web Spider – Part 2

Stumble it!

**************************************************************************

* Looking for a comprehensive course on Web Page Scraping?

* Let me know your interest by commenting on the SpyderSchool Post

* ************************************************************************

Scraping Web Pages with cURL Tutorial- Part 1

spyderwebtech — Fri, 08 Aug 2008 14:23:31 +0000

In my last post, Scraping Web Pages with cURL, I talked about what the cURL library can bring to the table and how we can use this library to create our own web spider class in PHP.

What I want to do in this tutorial is to show you how to use the cURL library to download nearly anything off of the web. In upcoming tutorials I will show you how to manipulate what you downloaded and extract whatever information that you want and either store that data in a database or save it on your server.

Creating a PHP Class -

Before we start talking about the cURL library, I first want to show you how to create a class in PHP. Classes are very useful in that they can hold a number of properties and functions and will allow use to easily reuse the code for better productivity. Classes are PHP’s way of creating what are called “Objects”.

An Object in PHP is very similar to an Object in the physical world. Everything that we can see and touch are objects. For instance, a Book is an Object. The Book also has a number of properties that describe that book such as an author, title, number of pages, line spacing, publisher, ect. We can use this same concept in our PHP code to represent a book by using the following code:

class Book { var $author = ""; var $title = ""; var $nopages = 0; var $publisher = "";

function Book($author, $title, $pages, $publisher)
{
$this->author = $author;
$this->title = $title;
$this->nopages = $pages;
$this->publisher = $publisher;

}
}

The above code is an example of class that I built used to describe a book.

After declaring the class, I have listed a number of properties that I want the class to have including author, title, number of pages, and publisher.

Next comes a function called the “constructor” which must be the same name as the class. All this function is used for is to set the properties that are passed into the object (i will show you how to do this later).

The $this keyword is used to reference the current instance of the object. So in this case, we are talking about the Book that we are describing, and not all the rest of the books on planet earth. $this is a very useful keyword that makes object oriented programming possible.

Now comes the good stuff, creating instances of the object we just created. Let’s say I wanted to create an Object from the class book (let’s say that I am creating an online library for people to look up their favorite romance novel). All we need to do is write the following code:

$firstbook = new Book("John Doe", "In Love", 200, "ACME Publishing");

$secondbook = new Book("John Doe", "In Love 2", 230, "ACME Publishing");

We have now created an instance of the Book in PHP with an author of “John Doe” and a title of “In Love”. John Doe’s second book is created in exactly the same way by declaring a “new Book” and passing different values into the class constructor. As you can see, we can reuse this code as many times as we want very easily.

Having reusable code structured this way, we can create hundreds of web spiders very quickly with very little effort. So now let’s create of web spider class…

Creating a Web Spider Class in PHP -

Now lets use the same thinking in creating a web page scraping spider class that we can use to download virtually anything off of the web. Let’s start our class by giving it a name of “wSpider” and let’s create the constructor.

class wSpider {

var $ch; /// going to used to hold our cURL instance var $html; /// used to hold resultant html data var $binary; /// used for binary transfers var $url; /// used to hold the url to be downloaded

function wSpider() { $this->html = ""; $this->binary = 0;

$this->url = “”;
}
}

In the above code, we create some properties which we are going to need for our class and then in the constructor we initialize all the properties. Right now our class does a whole lot of nothing, so to add functionality we are going to have to add functions.

A function, or sometimes called a method, is a list of instructions of what to do on that object. In the book example, I could have a method called Read(), which could cause someone to begin reading that book.

So for our wSpider class, let’s create a function underneath the constructor called fetchPage():
function fetchPage($url) { $this->url = $url; if (isset($this->url)) {

$this->ch = curl_init (); /// open a cURL instance

curl_setopt ($this->ch, CURLOPT_RETURNTRANSFER, 1); // tell cURL to return the data

curl_setopt ($this->ch, CURLOPT_URL, $this->url); /// set the URL to download

curl_setopt($this->ch, CURLOPT_FOLLOWLOCATION, true); /// Follow any redirects

curl_setopt($this->ch, CURLOPT_BINARYTRANSFER, $this->binary); /// tells cURL if the data is binary data or not

$this->html = curl_exec($this->ch); // pulls the webpage from the internet

curl_close ($this->ch); /// closes the connection

}
}

The above function does the following:

Checks to see if the url was passed through the function
Sets the options for the web page pull (see code for what each option does)
Pulls the web page from the Internet
Closes the cURL connection

The resultant html from the web spider is held in the $this->html property. Below is the finished code used to download an HTML web page and print it out to the screen.

$mySpider = new wSpider(); //// creates a new instance of the wSpider $mySpider->fetchPage("http://www.msn.com"); /// fetches the home page of msn.com

echo $mySpider->html; /// prints out the html to the screen

If you wanted to download a picture instead you would have to set the $this->binary equal to true(1). Pictures, Images, and Videos are made up of data that is binary (1s and 0s). So the following code will download a picture and store the data into the $this->html variable.

$mySpider = new wSpider(); //// creates a new instance of the wSpider $mySpider->binary = 1; /// turns on the binary transfer mode $mySpider->fetchPage("http://www.msn.com/pic123.jpg"); /// fetches a picture off of the msn home page

You can then use some PHP code to save it to a database or to save a picture file to your hard drive. The sky are the limit as to what you want to do with this data. Actually, this is the same technique that we can use to by-pass Captcha so make sure that you know how to use it.

So we are off to a good start. What you have learned here is crucial to understand if you are planning on doing some large scale scraping projects. Creating reusable code can sometimes take longer in the beginning, but if you do it right… then you can create spiders very quickly and you can use it on all your future project.

In the next tutorial we will talk about how to manipulate the data that you just have pulled as well as how to extend our class to be more functional. Happy Scraping!!

Other Web Spider Tutorials:

Scraping Web Pages with cURL Tutorial- Part 2

Build a Web Spider – Part 1

Building a Web Spider – Part 2

Stumble it!

**************************************************************************

* Looking for a comprehensive course on Web Page Scraping?

* Let me know your interest by commenting on the SpyderSchool Post

* ************************************************************************

Scraping Websites With cURL

spyderwebtech — Thu, 07 Aug 2008 13:50:00 +0000

Web Page Scraping is a hot topic of discussion around the Internet as more and more people are looking to create applications that pull data in from many different data sources and websites.

In my other tutorials, I talked about using PHP’s file_get_contents function to pull a web page and download the information into a variable string for later manipulation. This method of pulling data off of the web is very good when you are dealing with only text and html.

But what if you wanted to download pictures, graphics, or video off a number of websites and store them on your server? This is were PHP’s file_get_contents can not help us.

Introducing cURL!

cURL is command line tool for transferring files with URL syntax, which means that we can transfer most any type of file using this tool. Most, not all, web servers have the cURL library module installed already so you won’t have to do anything to begin using this powerful library.

cURL has the ability to transfer files using an extensive list of protocols, including:

FTP
FTPS
HTTP
HTTPS
TFTP
SCP
SFTP
Telnet
DICT
FILE
LDAP

As you can see cURL can not only use the HTTP protocol (which is what PHP’s file_get_contents function uses), but also the FTP protocol which can prove very useful if you want to create a web spider to upload files to server automatically or FTP videos to video sharing sites.

The good news is that cURL is so powerful that it can do most everything that you will ever need to do when it comes to web page scraping. The down-side is that cURL can be very tricky to deal with because there are a tremendous number of options to set and pit-falls to side step.

What I hope to do in this series of tutorials is show you how to work with cURL and how to create you own web scraping class in PHP so you can reuse the code time and time again. So let’s begin…

cURL and Your Web Server

Like I had mentioned that most of the time cURL is already set-up on your web server if you are using a hosted plan. (Sometimes on the “cheaper” plans, cURL is disabled so contact your administrator to see if they will enable it for you)

I personally do most of my web page scraping using my local web server. That’s right, you don’t even need to pay for a hosted server to scrape web pages. All you need is a computer and a web server like Xampp!

If you are using Xampp, like I recommended in my tutorial Creating a Local Development Environment, you will need to enable the cURL module in PHP.

To do this goto the PHP.ini file in your Xampp/php folder and the Xamp/apache/bin folder and uncomment the “php_curl.dll” line by removing the semi-colon.

; Windows Extensions ; Note that ODBC support is built in, so no dll is needed for it. ; Note that many DLL files are located in the extensions/ (PHP 4) ext/ (PHP 5) ; extension folders as well as the separate PECL DLL download (PHP 5). ; Be sure to appropriately set the extension_dir directive.

;extension=php_apc.dll
;extension=php_apd.dll
;extension=php_bcompiler.dll
;extension=php_bitset.dll
;extension=php_blenc.dll
;extension=php_bz2.dll
;extension=php_bz2_filter.dll
;extension=php_classkit.dll
;extension=php_cpdf.dll
;extension=php_crack.dll
extension=php_curl.dll
;extension=php_cvsclient.dll
;extension=php_db.dll
;extension=php_dba.dll
;extension=php_dbase.dll
;extension=php_dbx.dll

Save the changes and restart your web server.

You are now ready to start scraping the web. In the next tutorial, I will show you how you can create your own web scraping class in PHP using cURL.

Next tutorial:

Scraping Web Pages Using cURL Tutorial – Part 1

**************************************************************************

* Looking for a comprehensive course on Web Page Scraping?

* Let me know your interest by commenting on the SpyderSchool Post

* ************************************************************************

The WebSpyder School….

spyderwebtech — Tue, 22 Jul 2008 16:24:44 +0000

A Course on How to Scrape the Internet…Who is Interested??

This blog has really changed since the time I started it earlier this year. I am still working on my SEO empire and it is starting to pay off, but the majority of the traffic coming to this blog are people who are interested in learning the mechanics of “how to scrape data off the web”.

In addition to this, I get emails every single week from people who need help scraping a particular site or aren’t sure how to store data in a database and retrieve it. Some questions even go into more advanced topics like how to automate desktop applications and post the data to the web.

I am a person who believes that it is more important to “teach a person to fish” rather than give solutions on a one-on-one basis. Which got me thinking about starting up a web based school for people who want to become “Expert Web Automationist” (Yes, I did just make that up).

Actually, you can make a lot of money on freelance sites on projects that require data to be scraped off the web. The LEAST amount of money that I have made on one project was $80 when I was first starting out. Doesn’t sound like a lot of money for a project, but consider that the computer was doing all the work and it only took me 1/2 hour to setup the spider. That is $160/hr!!! And like I said that was the LEAST amount of money that I got paid to scrape data of the internet.

The school that I am envisioning will be for beginners all the way to those people who already have a lot of knowledge about programming for the web. So there will be something for everyone whether you have never written a line of code before in your life, or you are a seasoned web designer. The intent of the school will be to give you enough knowledge so you will be an “Expert” in the field of web automation, and will feel confident taking on any job for cash.

Like I said, I am only thinking about the concept of this school, but I have already put together a rough “curriculum”:

Web Automation Concepts
Web Scraping Tools – To make life easier
Client-Side Web Automation
- Javascript Basics
- IMacros Basics
- Others
Server-Side Web Automation
Regular Expressions
Desktop Automation
Remote Objects/Yahoo PIPES/Other
Advanced Topics (Beating Captcha)

Each one of these topics would have many hours of videos, tutorials, practice scraping exercises, and practice exams. I will also be giving you MY CODE that I have already compiled and use on my own projects so you don’t have to create them from scratch. It is my intent to make you BETTER than I am at the art of scraping information off the internet.

If I do create the course, I will limit the amount of people who can enroll to 250 people per year. The reason being is that I don’t want to train 1000′s of people who will all flock to guru.com and drive the price of scraping down. I also don’t want 1000′s of copies of my code floating around the internet… after all it took me 100s of hours to create the code, so I only want people who are serious having it.

So what is the price??? It sure the hell won’t be $30 or $100. I haven’t narrowed it down yet, obviously because I haven’t created it yet. But if you take this course, you should be able to recoup the cost with a few freelance web scraping jobs.

So here is where I need your involvement!!

If you are interested in this course…. LEAVE A COMMENT!!!!! and let me know that you are interested. If I don’t get enough interest then I will NOT create the course. I don’t want to waste my time? Tell your friends about this if you think they are interested, tell your mother, post on forums, ect, ect.

Also email me on your thoughts on pricing, what you want to see in the course, and anything else you can think of to make the course better. If you are proficient in web scraping already and want to be involved in creating this course then let me know. If you are interested in Beta (or Alpha) testing the site let me know as well.

Notice!!! If I don’t get 100 interested people to comment to me by the 1st week of October 2008 then I will scratch the idea.

[Edit] I just noticed that some spanner is stealing my traffic, so I am extending this offer until I can get my traffic back to normal [/Edit]

Why use WordPress MU??

spyderwebtech — Mon, 07 Jan 2008 21:38:22 +0000

Hey everyone,

Thanks for the great questions and for starting some great conversations. I am finding that the reply to my comments are getting longer and longer, so I figure why not just make the responses a post themselves.

In this post, I would like to make a comment on using WordPress MU as a “template system” and what I have observed and maybe what you can take away from the lessons that I have learned.

First off, I would like to say to everyone else who is on or thinking about starting their own SEO Journey… I personally believe that there is not right or wrong way to ANYTHING on the internet. There are just different ways of doing things. My main objective is to make “systems” so I can produce websites very fast and with as little effort as possible.

To accomplish this goal, I wanted to use WordPress MU as a type of template system so I could just use pre-existing templates and plugin my content with as few modifications as possible.

One question arrose as to why not simply just use a regular install of WordPress rather than WordPress MU. The answer to this questions is simply this… Context! The reason why I decided to use WordPress MU is because I was creating a number of product review websites in which I would have many different categories (toys, fashion, electronics, music, etc) under one domain name.

Where WordPress MU comes in is that MU will allow you to create multiple blogs using only one install of wordpress. So for instance is my main domain was www.reviewsite.com, I could have a:

music.reviewsite.com – for music
fashion.reviewsite.com – for fashion
cars.reviewsite.com – for cars
and on and on…

And like I said this can be done with only one install of WordPress MU.

If you have read Eli’s site you would have learned that subdomain are treated like individual websites as long a you don’t use a lot of inner linking from the main domain or do cross-linking.

Yes, I could have used just a regular version of WordPress and created a whole lot of subdomains from my Cpanel…. but that would mean that I would have to also install wordpress for each subdomain and then setup all the plugins and themes. With WordPress MU you can configure it to have plugins standard with each new blog that you create.

So can you see why I went the MU route? One install… hundreds of “individual sites”… one domain name… preconfigured plugins… preconfigured themes…. all that is needed is the content. I can actually create a new site (actually section of my review site) in literally less than a minute. Not bad huh?

One last comment on WordPress MU… Plugins. A comment was posted that there are more Plugins for the regular WordPress installation than for WordPress MU… That is absolutely correct! BUT… Plugins for WordPress MU are supposed to be designed to increase the functionality of administration of users blogs rather than displaying content on individual blogs.

In the WordPress MU installation you will finding two WP-Content/Plugins directories (where you put your Plugins)… one is for MU and the other is for the individual blogs. You can use ANY plugin for the individual blogs that you can with a regular WordPress installation (just make sure to put it in the right folder). So really there is not limitations to Plugins with MU compared to the regular installation.

I hope this answered some questions as to why I used MU rather than the regular installation of WordPress. Sorry that this ran on a bit… In my next post I will share some of the results that I am seeing…

Remember in Eli’s blog…. that a new site will take around 8 months to show it’s true potential… so until then keep on building and never look back.

Eli you rock…. and thanks guys for the great questions… keep them coming!

SEO Art of WAR – 3 Ways to Spy on Your Competitors

spyderwebtech — Fri, 07 Dec 2007 16:20:47 +0000

Let’s face it, to succeed in the world of online marketing you need to know exactly what your competitors are up to otherwise you could find yourself falling down Goolge’s search results faster than Senator Larry Craig’s pants in an airport’s bathroom.

SUN TZU wrote the ART OF WAR during the 6th century BC as a strategic guide to out-wit, out-think, and over-come the enemy. One of the tactics he used was to use the enemy’s strength against them. These tactics are no more true today online than they were 2000 years ago on the battle field.

Here are three ways that you can use SUN TZU’s tactics to spy on your competitors to see what they are up to and possibly increase your revenue in the process.

Content Momentum – In the world of SEO there is a saying that Content is King. If your competitors are constantly adding content to their websites they are encouraging Internet spiders to come to their sites and crawl them more frequently. And when this happens, more pages will get indexed and there is a potential for the SERPs to view their sites as “more authoritative” than yours. This is what I refer to as content momentum.

Art of War Tactic: Keep track of how many new pages are being indexed by the search engines per time period (day, week, or month – depends on your niche). If you see your competitors adding 10 pages of high quality content every month, you had better make plans to match their efforts or you may find yourself playing second fiddle. Use the search term “site:www.yourcompetitorswebsite.com” in Google to find out how many web pages are indexed.

Link Momentum – In the same respects as content momentum, Link Momentum is just as important. The number and quality of inbound links is a huge factor in placing high in the search engines. So why not know how many links are being added.

Art of War Tactic: Use the search term “link:www.competitorwebsite.com” to determine how many links are being added to your competitors per time period. (Yahoo and MSN have similar ways of finding out how many links each site has). If you see that your competitors are constantly adding incoming links to their site then you can bet their are after your position in the search engines. A way to use all your “enemy” hard work against her is find out who link to her site and get them to link to yours.

Use Your Enemy’s Keywords – Keyword are extremely important in your meta tags, headlines, the body of your content, and in the anchor links point to your site. Is your site optimized for all the keywords for your niche? Why not see what keywords your competitors are targeting and get a bigger piece of the pie?Art of War Tactic: View the HTML code of your competitor’s websites to see what kind of keyword they are using in the keyword meta tages, tags, links and in their description. Using this information, strengthen your site by adding new content, or by modifying your current content to incorporate some of the keyword phrases.

Use these tactics to track what your competitors are doing, and use it against them by doing it bigger and better. Does this sound unfair? You bet it is, but after all this is WAR!

Building A Web Spider – Part 2

spyderwebtech — Wed, 05 Dec 2007 12:08:49 +0000

In my last post, Building A Web Spider – Part 1, I covered how to analyze a website so that you can build a very simple web spider to collect the data that you want, and then I showed you how to use PHP to download that page to your server. In this post I am going to show you how to extract the information that you need and leave the rest behind.

When you download a web page using PHP’s function get_file_contents(), the resulting web page is downloaded and put into a variable as a string. What this means is that all the HTML from that web page is now located in a variable which you can access and manipulate (for instance a variable called “$page”) . The variable will contain not only all of the text that you can see on the web page, but all of the HTML tags as well (

,
,,,etc). We will use these tags to help us find the data that we need.

Remember in Part 1, the purpose of creating a web spider was to scrape the current stock price of Microsoft(MSFT) from finance.yahoo.com. If we look at the results page for Microsoft we will notice so unique characteristics of the current stock price compared to all the other prices listed on the page. The current price of the stock is both bolded and bigger than all the other prices on the page. When we go and look at the HTML for the stock price we see these HTML tags surround the stock price.

32.77

We also notice a tag which we must deal with in the HTML. Remember that all we want is the stock price of MSFT and nothing else which is the 32.77 (which will obviously change when you look at the page). Once again there are a lot of ways to get this value, but the way I am going to show you is by using regular expressions.

If you have heard about regular expression you are probably petrified right now, but really they are pretty straight forward and simple to use if you understand what you are trying to scrape from the web page. What regular expression do is to provide a pattern for a computer program to check for, and if it finds it then it will be recorded. In our example we want to check for a price pattern.

In this example we want to scrape a number off of the page. The number has 2 digits before a “.” and two digits after. So we want to write an experssion to match any number in this form. To do that we will use the expression “\d “

Expression : \d — will match any single numeric digit

So to match two digits we could use \d\d, and three digits \d\d\d, and so on and so on. But there is an easier way. We can match any number of digits by using “\d+” which will match one or more digits.

So to match our stock price on the page we could write a regular expression pattern of:

\d+. \d+

Which would match one or more digits before a period and then one or more after the period. This pattern will work except that in a regular expression the “.” has a significant meaning.

Expression: . — will match any digit, letter, symbol, space… well anything!

If we actually wanted to match a period(“.”) we would escape the period with a backslash like this

\d+\.\d+ — This expression will match any stock price on the page!!!

But remember we don’t want just any stock price, we want the current stock price. That is where the HTML tags come in. Let’s simply add the other tags into our expression.

Our original HTML:

32.77

First we need to deal with the tag. I notice that in the id name there lies the MSFT stock symbol which I am sure will change with each stock that I look up. So I will write an expression that looks like :

\d+\.\d+

As you can see, I simply added the span tag to my expression but I use the “.+” expression to match anything between the quote tags because I know that it will change. Now all I have to add is the and tags to my expression and I am ready to search for my pattern in page that I downloaded. My final expression is:

\d+\.\d+

Now to use this expression in PHP I will use a function called preg_match(). preg_match will match the regular expression to my string and output it to an array. To use my expression in PHP however I am going to have to add a “/” to the beginning and end of my regular expression and we will need to escape all the forward slashes (for instance in the tag) using the backslash (“\”)in my expression.

Also we want to extract the stock price to do that we want to put a “( )” around the portion of the expression that we want to extract. Notice the changes in the code below.

$page = file_get_contents($url) /// this is the code to download the page from the last post

$pattern = ‘/ (\d+\.\d+)<\/span><\/b><\/big>/’;

preg_match_all($pattern, $page, $results, PREG_PATTERN_ORDER );

/// the preg match will use the pattern to check the web page and then put the results in the the $results array

To access the stock price we can get it from the first record of the array and the second argument. In PHP it looks like this.

$stock_price = $results[0][1];

The stock price is now in a variable for you to print off on the page, store in a database or do what ever you want.

Now that you have your spider, you can feed it 1000s of stock symbols and your spider will download all of the current prices within a matter of seconds. This spider may be very simple but you can use the same thought process to gather links to your page, scrape data, or anything else you can think of.

Happy Scraping

Other Tutorials:

Scraping Web Pages Using cURL Tutorial – Part 1

Stumble it!

**************************************************************************

* Looking for a comprehensive course on Web Page Scraping?

* Let me know your interest by commenting on the SpyderSchool Post

* ************************************************************************

Monday’s Report – My Craigslist Weekend

spyderwebtech — Tue, 04 Dec 2007 01:04:50 +0000

A new week… making money and going Mad!

Success! I scored my first click through adsense this week. Pulling down a decent days earnings of $.09 makes this all work the while and I envision what will happen when I have 100s of websites all pulling in cash.. all the wonders of scale-ability!

This week has been pretty challenging with writing more web spider code to automate my SEO journey. My target this week is automating craigslist posts (as much as possible anyway). I must say that I am impressed with the craigslist staff and how much effort they put into making it a complete bitch to automate craigslist postings. Well, I am happy to say that I have fought the battle and won… The downside is that it took me much of Saturday to iron out all the bugs in my craigslist posting script.

My results are starting to come through now in the SERPs. My oldest database website has 32 pages index with no more than a few posts to some forums and the url submissions to the major search engine databases. Yahoo’s spider visits my site nearly everyday and crawls about 60 pages, while Google’s bot visits every other day and crawls only about 6 pages. The odd thing is that in Google I have 32 pages index while in Yahoo I have only 2…. very strange. I have been looking around the forums but I haven’t found a reason.

So what’s next for this week? I am going to be putting up more sites using WordPress MU as my template engine. Is MU better than SMARTY?…. Now it is. I had to do a lot of research to get everything humming and strumming along and figure out how the whole engine worked. For beginners who aren’t ready for a lot of programming, I would highly recommend SMARTY. Just make a very generic template and bust out a whole lot of sites.

Well that’s it for this post. I am going to try to bust out the second post on creating your own web spider. I can tell from my stats on this blog that this is a popular subject… so until then. Happy SEOing

Building A Web Spider – Part 1

spyderwebtech — Sat, 01 Dec 2007 01:30:48 +0000

Whether you want to gather data for a database web site, or determine which links are broken in your site a web spider is a great tool for getting repetitive tasks done. Scraping web pages from the internet is a great way to gather content and ideas for web sites that you want to build. You can build a web spider to check the current prices of your favorite stocks, or even to get the balance of your bank account.

I use web spiders all the time to build databases for my database sites… I have even created them to get my email for me and respond if certain conditions are met. The secret to understanding how to build a great web spider is to first understand the environment that it must work in.

What do I mean by this? Well first you must understand that a web spider is simply a computer program. And computer programs are based upon rules (wait didn’t I hear this in the MATRIX) and these programs will blindly follow these rules until we tell them to stop. So for us to have our internet spider complete the task that we ask of it, we first must create the rules that it will follow. And of course for us to create the rules we must first understand the environment.

To illustrate this point, let’s say that we want to create a web spider to download our favorite stock quote from Yahoo (substitute MSN, or Etrade if you like). Let’s write some pseudo-code for the way we would do this manually:

Goto the main page of Etrade.com

Fill-in form on front page with our stock symbol – MSFT

Hit Enter

Look to see the stock price which is in large font and red

Now let’s see what kind of question may arise when we start creating a web spider from this simple process.

Can I bypass the form by entering a dynamic url (say finance.yahoo.com/stockquote.asp?stocksym=MSFT)?

If I have to use the form, what are the form variables that I need to send and how (GET vs POST)?

Do I need to store cookies or session variables?

How do I get the stock price from the web page after downloading?

the list goes on and on.

You notice that this simple example of getting a stock quote has some questions that need to be answered before creating the rules of your web spider. And the best way to get these answers is by going directly to the source… the web page you are trying to scrape. So let’s get started!

To really figure out how a web site works we are going to need to look at the HTML code as well as the urls that appear in your navigation bar of your browser. So that is where we are going to start.

Navigate to Yahoo finance by going HERE

You will notice the base url is finance.yahoo.com, you can find this at the top of the browser. Next fill in the form for the stock that you want to look up (MSFT) and hit the enter button. You will notice now that the url has change to:

http://finance.yahoo.com/q?s=msft

Apparently the portion of the url that changes the stock to viewed is contained in the “q?m=” portion of the url. Therefore, I can look up any stock that I choose by putting a different value behind the “q?m=” portion. In this case it is very simple to create a spider to look up a lot of stocks by feeding different stock symbols into this url and the downloading the page.

So how do you download a page from the internet? Well PHP has a lot of really cool ways of doing this. I am only going to show you one quick way now. That is to use PHP’s file_get_contents() function.

Let’s say we want to get Microsoft’s stock page. Use the following code:

$url = “http://finance.yahoo.com/q?m=MSFT”; // This is the url for Microsoft

///// The next line will download the HTML and put it into a variable called $page

$page = file_get_contents($url);

echo $page; /// will print the html onto your page

?>

The file_get_contents function might be disabled on certain servers. Hostgato.com is where I have my hosting accounts and I know that it works there. I used to use Godaddy.com and I know that their cheaper shared hosting won’t allow for using the file_get_contents() function.

In my next post, I am going to show you how to extract the values that you want from the page that you just downloaded. So until then play around with the above code by changing the value of the stock symbol.
Stumble it!

**************************************************************************

* Looking for a comprehensive course on Web Page Scraping?

* Let me know your interest by commenting on the SpyderSchool Post

* ************************************************************************