Hello, I have built a link checker, that extracts links from files and then checks them to see if they're good or not. I'm coding with C# and using the HttpWebRequest to do the checking. The program uses multi-threading and it normally has about 10 threads going at once. First of all, has anyone else used this class and has it worked well for you? What happens to me is if I just run 1 link it works fine. But when I run about 1,000 links, most of them timeout. I set the timeout for 15000 ms which I thought was really long. I have no idea why it is timing out like this. Could one reason be that lots of the links are from the same host? I am checking many things like http://www.mysite.com/page1 http://www.mysite.com/page2 etc etc.. and all the links from a certain host will come up as timed out. I'm not really sure what to try next If anyone is willing to help I'll post my code or do whatever it takes. Thanks for your help

It may be that hosts have a limit on how often you can access them in a certain timeframe, to protect themselves from DOS attacks. In real life you'd never need more than say 5 hits from the same client in a second.

It may be that you're behind a NAT router and it's not allowing more than a certain amount of dynamic NAT sessions from your host at once.

That seems quite likely.

Mark

thanks for all your replys.

Here is the the code that I use to Send the Request

            HttpWebRequest WebReq = null;            try            {                WebReq = (HttpWebRequest)WebRequest.Create(LinkToCheck.Address.Replace("&", "&"));                WebReq.AllowAutoRedirect = false;                WebReq.Timeout = 15000;            }            catch            {                //blah blah failed for some reason            }            HttpWebResponse WebResp = null;            try            {                if (WebReq != null)                    WebResp = (HttpWebResponse)WebReq.GetResponse();            }            catch (Exception e)            {                 //Get the WebException and find out what kind of error we got            }

And that's basically all I do with the WebRequest end.

markr:
I had this same problem when it was single threaded, and just ran one at a time (the process just took 10 times longer)...could the problem you mentioned still be a problem with a single threaded app?

Quote:Original post by kag1
markr:
I had this same problem when it was single threaded, and just ran one at a time (the process just took 10 times longer)...could the problem you mentioned still be a problem with a single threaded app?

Well, possibly.

Personally I'd get out your favourite network analyser and have a look at the frames on the wire to see what's happening to those timed out connections.

NAT routers doing "dynamic NAT", have only finite space in their nat tables. They therefore, sometimes limit the number of entries per host (so that one host making an excessive number of connections can't deny service to others).

It's possible that the web client isn't properly closing the connections (Microsoft web clients have a history of this), or the router holds the NAT table entry open for a little while after the connection is closed (to allow the host to send negative replies to any long-lost frames that come back etc, to be polite).

It's entirely possible also that the target host has a rate-limiter.

Also your proxy server (or transparent proxy) could be rate limiting something.

Check whether your ISP uses a transparent proxy (Many do).

If you still get the problem with a local network web server in the absence of either a transparent proxy or NAT router, I'd be fairly surprised.

Mark