Sign in to follow this  

asynchronous file enum & check, thread problem

This topic is 3586 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I want to have one thread enum files in user folder and possible recurse them. and while one thread is doing that i want another checking the file enum data to see if it should be filter in or out. What would be the best way to do this? I have no idea how quick the filters will be. The file can fail (or pass) on the first check or it can go through dozens of checks and fail on the last check. So i may be able to enum 5 files before the first check is done, then do checks 3 files before 1 file enum is done. How should i manage these threads? the enum thread ends when all files has been enum'd and then the filter flag will end when all files enum data has been checked. then the main thread will use the filter files how it pleases. I am mostly worried about locking/unlocking hundreds of thousands of times. I was considering having a volatile ulong fileEnumCount, isFinished; have the filter file check for the two variable and keep doing sleep(0) until isFinished is true and enumScanIndex == fileEnumCount. But that feels to much of a hack

Share this post


Link to post
Share on other sites
Run everything in a single thread.

Compared to CPU, disk is sloooooooow. Actually, it's slower than that.

Putting filter tests into a separate thread gains you nothing. The tests are cheap compared to inter-thread communication. Not to mention memory allocation you'd need for that, or if you use pre-allocated memory, then one thread will always lag behind another, etc...

Threads applied in this manner for this problem are mis-applied.

And if what you're really testing is an enum (int value), and you're testing it for equality against something, then the cost of that is almost literally zero. The cost of context switch would be higher than testing a single value.

Share this post


Link to post
Share on other sites
Actually i am not testing for a enum value, i am enuming the file data and checking its size, filename, time access, etc. then i am testing it though either simple or very complex filters.

I know the HD is slow, but i dont want to scan 1 file, run every test then scan another file. i rather keep the hd enum going so i dont have to wait.

I had the feeling inter-thread communication is expensive. Right now that volatile variable looks like a good solution. Anyone else have any comments about how i should do this?

Share this post


Link to post
Share on other sites
Quote:
Original post by AcidZombie24
I had the feeling inter-thread communication is expensive. Right now that volatile variable looks like a good solution.


They're not a solution. Strictly speaking, the volatile keyword is pretty much useless for multi-threading. Newer versions of MSVC give you certain guarantees, but I would feel very uncomfortable relying on them.

Share this post


Link to post
Share on other sites
Quote:
Original post by AcidZombie24
I know the HD is slow, but i dont want to scan 1 file, run every test then scan another file.

Why not ? As Antheus said, multithreading this will gain you absolutely nothing here. It will probably slow everything down significantly.

Unless your filters are ridiculously expensive. But then you're doing something very wrong somewhere.

Quote:
Original post by AcidZombie24
I had the feeling inter-thread communication is expensive. Right now that volatile variable looks like a good solution. Anyone else have any comments about how i should do this?

Your current 'solution' is just asking for race conditions or deadlocks. You'll need to properly secure them with mutexes, condition variables, or something similar.

Share this post


Link to post
Share on other sites
Quote:
Original post by Yann L
Why not ? As Antheus said, multithreading this will gain you absolutely nothing here. It will probably slow everything down significantly.

Unless your filters are ridiculously expensive. But then you're doing something very wrong somewhere.


even with a volatile variable solution? i am still thinking this is better. I may paste the code up to see what you guys think. I am sure i can do this w/o taking a significant thread hit.

Quote:
Original post by Yann L
Quote:
Original post by AcidZombie24
I had the feeling inter-thread communication is expensive. Right now that volatile variable looks like a good solution. Anyone else have any comments about how i should do this?

Your current 'solution' is just asking for race conditions or deadlocks. You'll need to properly secure them with mutexes, condition variables, or something similar.


What is so unsecured with a volatile? only 1 thread is updating it (increase only) and the other thread is only reading it to see if there is more data to process.

Share this post


Link to post
Share on other sites
Quote:
Original post by AcidZombie24
even with a volatile variable solution?

The volatile modifier has absolutely nothing to do with it. Unless you are doing very time consuming operations on the listed files (eg. parsing them or similar), multithreading is a poor choice for this task.

Give us an example of the most complex filter you have.

Quote:
Original post by AcidZombie24
What is so unsecured with a volatile? only 1 thread is updating it (increase only) and the other thread is only reading it to see if there is more data to process.

Volatile is not a synchronisation primitive. And if you really want to multithread this process, then you'll need a lot more than just a one variable single-way communication channel.

Share this post


Link to post
Share on other sites
come to think of it. the enum will destory the data so i would have to copy the struct if i wanted to do that, which i dont >_<.

btw, all i would need is 2, isFinished, fileCount. i check proccessedFileCount against fileCount, when == i check if it finished. if so, it ends. otherwise it sleeps and waits.

*kicks self*

Share this post


Link to post
Share on other sites

This topic is 3586 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this