Here is a pic of what I have so far. Note that the list of URLs are not the URLs that have been visited, but they are the ones that were found on the first page. That will change shortly as this is just for debug purposes.
I have been working on this little web crawler and at this point it gets all of the absolute links (doing relative links will be a pain, ugh...) from the given website. My next task is to make it keep going after finding all of the links from the first page. That will be easy enough. But making it check if its been to that page already will surely slow things down a ton, especially after its been to a few thousand sites. But oh well, im just trying to get a rudimentary version completed.
Here is a pic of what I have so far. Note that the list of URLs are not the URLs that have been visited, but they are the ones that were found on the first page. That will change shortly as this is just for debug purposes.
Here is a pic of what I have so far. Note that the list of URLs are not the URLs that have been visited, but they are the ones that were found on the first page. That will change shortly as this is just for debug purposes.
Previous Entry
Project Idea
Next Entry
First Step
Advertisement
Latest Entries
26 Games 26 Weeks
1639 views
Celenite
1186 views
Im back...again!
1034 views
Tunnel Syndrome
1306 views
Borderlands
1211 views
First Flash Game
1007 views
First Flash Game
1304 views
Bullet holes? Yum!
1117 views
Game 2
1067 views
New Project
924 views
Advertisement