I have been working on this little web crawler and at this point it gets all of the absolute links (doing relative links will be a pain, ugh...) from the given website. My next task is to make it keep going after finding all of the links from the first page. That will be easy enough. But making it check if its been to that page already will surely slow things down a ton, especially after its been to a few thousand sites. But oh well, im just trying to get a rudimentary version completed.

Here is a pic of what I have so far. Note that the list of URLs are not the URLs that have been visited, but they are the ones that were found on the first page. That will change shortly as this is just for debug purposes.