Sign in to follow this  

STL Vector & Multithreading

This topic is 4306 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I seem to be having some problems with code relating to std::vector and multithreading.... I create a thread, the threads process calls a function which fills a std::vector with data. Just before the thread returns I test the data in the vector and it is good. After the thread returns I test the data in the threads calling process, and it appears some of the data in the vector is corrupted, not all of the data, only some. This only happens on Debug builds and it only happens the first time the build is launched. If launched a second time the data in the vector is fine. If rebuilt, the first time it is launched it will have corrupt data again... This problem doesn't arise in release builds.... Here is some pseudo-code
Quote:

int Main()
{
    CreateLoaderThread( LoaderThreadProcess );    // _beginthreadex

    WaitForLoaderThreadToFinishExecution();       // Wait for thread to complete

    TestDataInSTLVector();                        // Some data is corrupt
}


__stdcall LoaderThreadProcess()
{
    GetFirstData();

    while( DataIsGood() && DataExists() )
    {
        LoadSTLVectorWithData();
        GetNextData();
    }

    TestDataInSTLVector();   // Data is fine
    ExitThread();
    return;
}


int LoadSTLVectorWithData()
{
    DataStruct dataStruct;

    InitializeDataStruct(&dataStruct);   // Initialize all struct members

    dataStruct->data = 0;             // Set struct data here

    vecVector.push_back(dataStruct);     // Push struct onto vector
}

This isn't how my code really is, but it's a fair enough approximation for this question I hope. The fact that the problem only happens on the first launch of a fresh Debug build led me to beleive that it was the result of a variable not being properly initialized. Then I started testing the data in the vector in the thread procedure just after the vector is finished being filled, and the data is fine at this point within the thread procedure. The data in the vector isn't corrupted until after the thread procedure has exited. I'm wondering if anyone has any suggestions... I'm using Visual Studio 2003. Thanks... [Edited by - shaitan on February 28, 2006 11:10:32 AM]

Share this post


Link to post
Share on other sites
Quote:
Original post by bakery2k1
Try removing the call to ExitThread, and simply returning from the thread function.

I will give that a shot....

Quote:
Original post by NotAYakk
Contents of LoadSTLVectorWithData() please. =)

I've edited my original post to show pseudo-code for the
LoadSTLVectorWithData() sequence..

Share this post


Link to post
Share on other sites
Does the vector contain pointers, or full-copies of the struct in question?

Do the structs contain any pointers?

What does the "corrupt" data look like? (In hex ideally -- 0xFEEEFEEE? 0xBAADF00D? 0xFEFEFEFE? 0x00000000?)

The bit patterns of corrupt data can tell you alot. Some compilers in debug mode blit cleared/deallocated/uninitialized/beyond top of stack memory with characteristic patterns.

One possibility is you are sticking pointers to data on your thread's stack into the vector. When the thread exits, the thread's stack in debug is likely to be cleared/written over with a bit pattern, but in release it will be left alone.

Share this post


Link to post
Share on other sites
Quote:
Original post by bakery2k1
Try removing the call to ExitThread, and simply returning from the thread function.



I've just finished trying that and it didn't help. Thanks for the suggestion anyway.


I'll give a little more information on what I have done.

The entire code in question resides within one DLL....

Basically what I am doing is calling a thread to load graphic resources into memory.
I'm using a thread so the owning process can perform its rendering duties without having
to wait for the graphic resources to be loaded.

Originally I was using the CreateThread() API call to create my thread... but after this problem
started arising and I did some investigating I switched to the _beginthreadex() method, this appears
to be the better method of thread creation but it had no visible impact on the problem.

I also realized while investigating the problem, that I had the compiler set to use the single-threaded
runtime library, I changed the switch to "Multi-threaded Debug DLL", this had no visible impact on the
problem either. (But seems to be the correct setting for a Debug Build of a DLL which creates a thread(s))


I've tried various combinations with/without _endthreadex() and/or ExitThread() at the end the threads
process function, this didn't help, thanks anyway bakery2k1, it was a good suggestion and something that
I had not tried.

This is an annoying problem, mainly because it is impossible for me to pin down exactly when it started
and what I may have changed in the code to make it happen. I was in the process of adding (and debugging)
sections of code functionality, it wasn't till I had finished adding and debugging much code that I even
realized that this was happening, (It took me a little bit to figure out that this problem was happening
only on the first run of a freshly compiled debug build).




This is an excerpt of relevant sections of my debug log:

Quote:

Start DataLoading Thread

Load Data to vector using loop

winGreenSmoke1.bmp rsGraphic.m_pMemPosition: BMøž

winGreenSmoke2.bmp rsGraphic.m_pMemPosition: BM
winGreenSmoke3.bmp rsGraphic.m_pMemPosition: BMøž

winGreenSmoke4.bmp rsGraphic.m_pMemPosition: BMøž

winGreenSmoke5.bmp rsGraphic.m_pMemPosition: BMøž

winGreenSmoke6.bmp rsGraphic.m_pMemPosition: BM8)

After all data is finished loading, test it for validity

MEMPOS: BMøž

MEMPOS: BM
MEMPOS: BMøž

MEMPOS: BMøž

MEMPOS: BMøž

MEMPOS: BM8)

Exit From DataLoading Thread


Now test data again in the owning process that created the DataLoading Thread

MEMPOS: Memory: BMøž
-- Most data is fine
MEMPOS: Memory: BM
MEMPOS: Memory: BMøž

MEMPOS: Memory: Ý%MÁé?xaqÉ8jG'9ÒŽ“..... etc.... -- Some data seems corrupt
MEMPOS: Memory: +è)Ñ‘]q¥@Õv‚bž’¸&èF..... etc....
MEMPOS: Memory: Ç÷Inˆ¬©t‚"qÓ“@°NÉÖŠšK..... etc....


The data that appears to be corrupted seems to always be at or near the vector.end() ussually the last
three entries in the vector are corrupt.

Share this post


Link to post
Share on other sites
Quote:
Original post by NotAYakk
Does the vector contain pointers, or full-copies of the struct in question?

Do the structs contain any pointers?


vector contains full-copies

structs contain pointers
Quote:


struct DataStruct
{
int iWidth;
int iHeight;
char pszName[MAX_CHAR];
unsigned char* pMemPosition;
unsigned int uiMemDataSize;
LPDIRECT3DSURFACE9 pSurface;
}



The only struct member that seems to be affected is the pMemPosition.
All other members seem to be unaffected.

Quote:

What does the "corrupt" data look like? (In hex ideally -- 0xFEEEFEEE? 0xBAADF00D? 0xFEFEFEFE? 0x00000000?)

The bit patterns of corrupt data can tell you alot. Some compilers in debug mode blit cleared/deallocated/uninitialized/beyond top of stack memory with characteristic patterns.

One possibility is you are sticking pointers to data on your thread's stack into the vector. When the thread exits, the thread's stack in debug is likely to be cleared/written over with a bit pattern, but in release it will be left alone.


Sample Data that appears corrupt
Quote:

Memory: Ý%MÁé?xaqÉ8jG'9ÒŽ“gI½ HOé°X9ЫLµÕ—;
Memory: +è)Ñ‘]q¥@Õv‚bž’¸&èFH8ã®°‰Óu`ry@
Memory: Ç÷Inˆ¬©t‚"qÓ“@°NÉÖŠšK¦x48­žf¤C«
S@0‡HÊ

Share this post


Link to post
Share on other sites
Lets look at pMemPosition; then.

Is pMemPosition itself being corrupted, or the data that pMemPosition points to?

(you may have to print-debug this, and see if pMemPosition is changing on you, or if it is the data pMemPosition is pointing to that is changing)

Share this post


Link to post
Share on other sites
Quote:
Original post by NotAYakk
Lets look at pMemPosition; then.

Is pMemPosition itself being corrupted, or the data that pMemPosition points to?

(you may have to print-debug this, and see if pMemPosition is changing on you, or if it is the data pMemPosition is pointing to that is changing)


It appears that it is the data that is being corrupted.
I've tested the address value of the pointer and it seems to remain
correct in the thread and after the thread.

Thank you very much NotAYakk, for taking the time to consider my problem...
It is much appreciated...

Share this post


Link to post
Share on other sites

I should probably mention that the Data that pMemoryPos points to
is a subsection of a larger chunk of memory that was allocated with
malloc()

Example:
Quote:

Allocated Memory Chunk
[[DATA_SET_01][DATA_SET_02][DATA_SET_03][etc...]]


What I wonder about is the fact that the problem happens only on the
first execution of a Debug Build, and not on subsequent executions of
the same build.

I've had that type of problem before and it has always been the result
of a variable, ( usually a char[] ), not being properly initialized.
I suspect that when this type of problem happens, on the first execution
the improperly initialized variable causes a lock-up/crash, but at the
same time something happens within Visual Studio(?) that causes the
variable to act as if it is properly initialized on subsequent executions.
When I've had problems similar to this in the past, the release build
would consistently lock/crash.

This does not appear to be the case this time though.



Share this post


Link to post
Share on other sites
Who owns that large block of memory?

Something that is easy to check:
Is free() ever called? (if you call free() on any pointer into a malloc'd block, the entire block is deallocated. Pointers into deallocated blocks degenerate into random garbage, and behave differently in debug and release)

What compiler are you using?

Those printouts of your memory -- can you print them out in hex format? (printf "0x%08x" on 4 byte boundries) (I don't know the ASCII-binary mapping off the top of my head, and I'm lazy -- but I can recognize some hex bug fingerprints. ;))

My next question is, could you be accidentally overwriting data from one [data_block] into the next? (unlikely, if it is actually happening as you exit the thread...)

Who owns the vector in question -- is it global, heap allocated, or on a stack somewhere?

Do you have any code in destructors you have written? Are any of them in the stack of the thread? What objects are on the stack of the thread at the time you check and make certain that your data is not corrupt?

Share this post


Link to post
Share on other sites
Quote:
Original post by NotAYakk
Who owns that large block of memory?

It is owned by a global object that manages memory for my app

Quote:

Something that is easy to check:
Is free() ever called? (if you call free() on any pointer into a malloc'd block, the entire block is deallocated. Pointers into deallocated blocks degenerate into random garbage, and behave differently in debug and release)

free() is called once when the global object that manages the memory is destroyed.

Quote:

What compiler are you using?

MSVisualStudio 2003 (MSVSC 7?)

Quote:

Those printouts of your memory -- can you print them out in hex format? (printf "0x%08x" on 4 byte boundries) (I don't know the ASCII-binary mapping off the top of my head, and I'm lazy -- but I can recognize some hex bug fingerprints. ;))

I can give it a shot...

Quote:

My next question is, could you be accidentally overwriting data from one [data_block] into the next? (unlikely, if it is actually happening as you exit the thread...)

No, its not possible that I'm overwriting data, I'm loading the data,
checking its validity while loading, then checking directly afterwards
in the thread and all data is valid.
Then directly after the thread is recognized as exited, I check the
data again and the corruption shows.

Quote:

Who owns the vector in question -- is it global, heap allocated, or on a stack somewhere?

It is owned by a second global object that manages graphical resources.

Quote:

Do you have any code in destructors you have written? Are any of them in the stack of the thread? What objects are on the stack of the thread at the time you check and make certain that your data is not corrupt?

There is code in the destructors of both the global objects I've mentioned
but those objects are created and destroyed only once when the application
starts and when it exits.
As to questions regarding the stack of the thread, that subject is a little
over my head at this time, something I need to spend some time on.

On another note, I've rewritten large sections of this project a number of
times since I've been working on it. I have no doubt I will be doing this
again and if the problem persists at that time I will have to pay special
attention to it as I'm adding that module/functionality back to the project.

Thanks again for you help NotAYakk.

Share this post


Link to post
Share on other sites
Quote:
Original post by Zahlman
Why are you malloc()'ing stuff rather than using new? Especially if you're smart enough to make use of std::vector?


This is the one and only time in my app that malloc() is used.
All other memory allocation is done with new.

When allocating memory for objects I consistently use new, the reason I used
malloc() in this situation, is because I was working with tutorials and books
to complete that section of code, and that is the way it was generally done
in the code that I was using as a study aid.

Pretty much all of my memory allocations are done for objects, this is the
only situation that I can think of off hand where I'm allocating memory
for a varialbe and not an object.

This is what I'm doing:
Quote:

unsigned char* pMemBlock = (unsigned char*)malloc(MemBlockSize);


I'm wondering if I should allocate memory for a char[] instead?
Quote:

unsigned char* pMemBlock = new unsigned char[MemBlockSize];


Although it is of interest to think about, I'm not sure if this would
have much to do with the problem I'm experiencing.



Share this post


Link to post
Share on other sites

This topic is 4306 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this