This topic is 5165 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

Recommended Posts

//Psuedo-codey
class document
{
public document(string url)
{
}

{
{
document doc = new document(link.url);
}
}
}

Now this was waaaaaaaaay faster. BUT as you can imagine this spawned loads of unterminated thread and around the 1500 mark I get an exception saying that Start could not start a new thread. So I modified the code above thus:
//Psuedo-codey
class document
{
public document(string url)
{
{
document doc = new document(link.url);
}
}

{
}
}


Share on other sites
I would make only two threads.

Make two queues. One for the urls and one for die downloaded documents.

One thread looks for new urls in the queue, starts downloading and puts them in the document queue when finished.

One thread looks for new documents in the document queue and adds new urls to the url queue.

It sounds like you get a high improvement by downloading several documents at the same time. Just try to make several threads of the first kind until you don't get an improvement.

use lock( ... ) when accessing the queue to prevent errors or multiple downloading of the same document.

Well this is how I use multi threading. Separating the bottleneck in a thread to not slow down the rest.

Share on other sites
OK, This seems to work a treat BUT (and this is a limitation on my part not on the part of the proposed solution).

In my original MT solution each thread ran for a small slice of time then died. This meant that the PC was nice and resposive when there was little to do. In this solution I do the following in each thread

void worker_thread(){   while(brunning)   {      //Do interesting things   }}

This brunts up the processor usage up to 100% all the time. Is there a more efficient way to accomplish what I have done? Not done a huge amount of MT in c#, and the little I have has been runnign very short threads so this has never been an issue before.

Cheers

Neil

Share on other sites
Throw in a Thread.Sleep(0) into your loop to give some control back to the processor.

Share on other sites
Damn, I am glad that this works for you. Otherwise I had to review a lot of code ;)

Sleep will solve your remaining problem. But I feel that Sleep(0) does not free a lot time. Don't mind using Sleep(100). Most of the time the process thread will just find an empty queue and do nothing but eating processor time.
I mean the delay time does not sum up. In no case there will be a document pending for more than 100 ms, what is negligible for that purpose.

I guess you can analize it statistically and use a smart dynamic delay time but this is quite far beyond reasonable ;)

Share on other sites
You could also just have an event that you clear when the queue is empty, and set when the queue has an element in it. The worker thread would then wait on the event, this would put the thread into the wait state when the queue is empty, It also allows you to scale a bit better because you can extend the solution to multiple queues across multiple threads easily enough (also, 100ms sleep is not guaranteed to be 100ms, you just surender your chunk of the CPUs time for AT LEAST that amount of time. It could be significantly longer.)

Share on other sites
what about one thread finding all the links out of a queue, and one using 'System.Threading.ThreadPool.QueueUserWorkItem(...)' to follow the link and add it's items to the queue? that should work fairly well yeah? Just use a ManualResetEvent to stop and start the worker thread when the queue is full or not?

Share on other sites
I think i'm going to redesign it a little again. From running it last nightthe downlaods were a real bottleneck. The Queue of documents waiting to be processed was 99% of the time fully processed, but the list of documents waiting to be downloaded was still huge. Alos because it was only downloading the 1 doc at a time I dont htink I was getting maximum efficiency.

I'll report back to say if this works OK.

Neil

Share on other sites
Well the final idea worked a treat.

The main program thread processes html documents put into a queue by downloader threads. I have had to put in some manual thread management, as outside of the debugger the app was running way too fast and I had 1000 downloads running simultansously. I've pretty much expanded it as far as I can as a simple app as the memory requirements go up to 2 gig of virtual memory after a few seconds, so I'm going to have to sit down and think about a sensible database design now.

Thanks to all who helped me along the way!

Neil

• Game Developer Survey

We are looking for qualified game developers to participate in a 10-minute online survey. Qualified participants will be offered a \$15 incentive for your time and insights. Click here to start!

• 16
• 11
• 24
• 42
• 75