Sign in to follow this  
janta

memcpy not copying

Recommended Posts

Hiya :) Could anyone explain me how this is possible:
// assuming pDest, pSrc and size are all valid
memcpy(pDest, pSrc, size)
assert( memcmp(pDest, pSrc, size) == 0 ) // ASSERTION FAILS !
The platform is Windows XP 64bit edition. Any clue would be (god damn) appreciated [smile] Regards, JA Edit: whoooooho looks like I forgot a fundamental thing. I'm gonna check whether my pointers overlap...

Share this post


Link to post
Share on other sites
Have you stepped through with a debugger?

Looked at the generated assembly?

I wouldn't hazard a guess without both of these things

Share this post


Link to post
Share on other sites
I've done more debugging. Both pointers are perfectly valid, the byte count is valid. There is no overlap.

Ok, here's my code. I cant believe what I'm seing. Either I'm very stupid, or I'm very tired, or very cursed [smile]


template <class T> T* Duplicate( const T* pObj, int numObjects)
{
T *pNewObj = new T[numObjects];
memcpy( pNewObj, pObj, bytes );
return pNewObj;
};


In order to to debug, I did this:


template <class T> T* Duplicate( const T* pObj, int numObjects)
{
T *pNewObj = new T[numObjects];
u32 bytes = sizeof(T) * numObjects;

memcpy( pNewObj, pObj, bytes );

// detect differences
if( memcmp(pNewObj, pObj, bytes ) )
{
_asm int 3;

// find out what is different
for(u32 i = 0 ; i < bytes ; ++i)
{
if(pSource[i] != pDest[i])
__asm int 3;
}

// give it another try
memcpy( pNewObj, pObj, bytes );

// Still not good ???
if( memcmp(pNewObj, pObj, bytes ) )
{
_asm int 3;

for(u32 i = 0 ; i < bytes ; ++i)
{
if(pSource[i] != pDest[i])
__asm int 3;
}
}
}

return pNewObj;
};


After the first memcpy, sometimes the memcmp will detect a difference. The small loop right after that always tells me that all the bytes are identical EXCEPT 4 adjacent bytes (as if someone had written a int = 0 to the data, forming a "hole") Everything before and after that hole is identical to the source buffer. The offset of that hole from the begining of the buffer may vary, and is no special value. The next memcpy (second attempt) always fills the hole and no further differences are detected...

This is so damn weird, and I'm prety sure it must be something stupid I just overlooked...

The very same executable file generates no such error under Windows xp 32bit, only under Windows xp 64bits (but that could be a coincidence)

Share this post


Link to post
Share on other sites

template <class T> T* Duplicate( const T* pObj, int numObjects)
{
T *pNewObj = new T[numObjects];
memcpy( pNewObj, pObj, bytes );
return pNewObj;
};


Where is "bytes" coming from? Mars? [smile]

[EDIT] AH... I see you fixed that down lower.

Well... this sounds like some type of CRT library mismatch to me... something built against a static library (of your own) that is out of date, that type of thing.

Share this post


Link to post
Share on other sites
Is there a reason why you aren't using std::copy? Or something like this:


template <typename T>
std::vector<T> Duplicate(const std::vector<T>& objects)
{
return std::vector<T> ret(objects);
}

Share this post


Link to post
Share on other sites
Quote:
Original post by Verg

template <class T> T* Duplicate( const T* pObj, int numObjects)
{
T *pNewObj = new T[numObjects];
memcpy( pNewObj, pObj, bytes );
return pNewObj;
};


Where is "bytes" coming from? Mars? [smile]

[EDIT] AH... I see you fixed that down lower.

Well... this sounds like some type of CRT library mismatch to me... something built against a static library (of your own) that is out of date, that type of thing.


The game is built under Windows XP 32bit with the Multi-Threaded DLL (/MD)

if I replace the memcpy by a loop of my own, everything is just fine


for(u32 i = 0 ; i < bytes ; ++i)
pDest[i] = pSource[i];


@Driv3MeFar: Yeah there is. I'm not coding at home, I'm working on a large codebase like 1M lines of code and I cant afford to refactor everything. The team who originally designed the software made the choice not to use stl.



Share this post


Link to post
Share on other sites
Quote:
Original post by janta
Quote:
Original post by Verg

template <class T> T* Duplicate( const T* pObj, int numObjects)
{
T *pNewObj = new T[numObjects];
memcpy( pNewObj, pObj, bytes );
return pNewObj;
};


Where is "bytes" coming from? Mars? [smile]

[EDIT] AH... I see you fixed that down lower.

Well... this sounds like some type of CRT library mismatch to me... something built against a static library (of your own) that is out of date, that type of thing.


The game is built under Windows XP 32bit with the Multi-Threaded DLL (/MD)

if I replace the memcpy by a loop of my own, everything is just fine


for(u32 i = 0 ; i < bytes ; ++i)
pDest[i] = pSource[i];



Well if everything is as you say it is... then the wrong code is being executed as a result of your call to "memcpy". Can you step into the memcpy.asm file?

Is "memcpy" being redefined somewhere?

A last resort would be to roll your own "memcpy" using the inline assembler (unless you're building on that 64-bit compiler...)

Share this post


Link to post
Share on other sites
Quote:
Original post by janta
@Driv3MeFar: Yeah there is. I'm not coding at home, I'm working on a large codebase like 1M lines of code and I cant afford to refactor everything. The team who originally designed the software made the choice not to use stl.


Well, std::vector and std::copy are not part of the STL, they're part of the c++ standard library.

If you don't want to use a vector that's fine, but I can't really see why you wouldn't use copy here:


template <typename T>
T* Duplicate(T* pIn, int numObjs)
{
T* ret = new T[numObjs];

std::copy(pIn, pIn + (numObjs * sizeof(T)), ret);

return ret;
}

Disclaimer: This is untested, my pointer math is probably off.

Share this post


Link to post
Share on other sites
That's interesting. Looking at std::copy's signature I saw it required some iterators as inputs, I did not think that mere pointers could be used as iterators. I'll try this, thanks.

@Verg: Well, memcpy is used at least a bazillion times throughout the project with no errors. The game runs really fine except when it randomly crashes. I don't think I mention the randomness by the way. Most of the times memcpy does its job, then once in a while, boom.

Also, I've taken a look at the assembly and everywhere memcpy is called there is a jmp at the same adress, so I think I can assume always the "same" memcpy is being called. I sould step into though, to see more of what really happens...

Share this post


Link to post
Share on other sites
Quote:
Original post by Driv3MeFar

template <typename T>
T* Duplicate(T* pIn, int numObjs)
{
T* ret = new T[numObjs];

std::copy(pIn, pIn + numObjs, ret);

return ret;
}


EDIT: Fixed bug.

Share this post


Link to post
Share on other sites
Quote:
Original post by dalleboy
Quote:
Original post by Driv3MeFar

template <typename T>
T* Duplicate(T* pIn, int numObjs)
{
T* ret = new T[numObjs];

std::copy(pIn, pIn + numObjs, ret);

return ret;
}


EDIT: Fixed bug.


Oops. I figured I'd fucked something up. Thanks for the fix [grin].

Share this post


Link to post
Share on other sites
If the problems are appearing randomly that sounds like you've stomped on a heap structure somewhere. Perhaps the OS thinks it put some heap structures in your memory and is altering a pointer within those structures (setting the 4 byte quantity to 0). Can you set a memory breakpoint on the buffer and see what triggers it?

Share this post


Link to post
Share on other sites
I've tried this already. I think I can watch a 4 bytes memory space, but not a full buffer of > 1kb (or much more sometimes)
And the memory corruption isn't always at the same address. Plus, I only control one thread, all other are system created (DirectInput, XAct, IO, etc.)

At this point I'm starting to suspect a hardware failure but I wont jump to that conclusion just yet

Share this post


Link to post
Share on other sites
This may be helpful or not... but maybe you can get more information (even if just a little) by logging the content of both buffers after the copy?

Cheers
-Scott

edit: Sorry nevermind... For some reason I read everything except for in the post right after you were like "In order to to debug, I did this:". Guess I need to wake up...

Share this post


Link to post
Share on other sites
Quote:
Original post by janta
@Driv3MeFar: Yeah there is. I'm not coding at home, I'm working on a large codebase like 1M lines of code and I cant afford to refactor everything. The team who originally designed the software made the choice not to use stl.


Had I a billion dollars, I would pay off everyone I could find in similar positions to quit their jobs and arrange boycotts of the responsible companies. It seems nothing else will kill these (code) monstrosities.

Quote:
That's interesting. Looking at std::copy's signature I saw it required some iterators as inputs, I did not think that mere pointers could be used as iterators. I'll try this, thanks.


Of course pointers are iterators. Why do you think the iterator interface is designed so they look like pointers as much as possible? Remember, template code doesn't care about differences between types; it cares about syntactically valid constructs with the given type (ad-hoc polymorphism).

Share this post


Link to post
Share on other sites
All right, if someone is still interested by my personal life, I realized that another thread (a system-spawned thread, probably XACT since problems occurs right after the audio module has been initialized) is currupting my data. I don't know how and when though. So memcpy is not guilty, as was to be expected.

Now I think I'm gonna have a hard time debugging this...

A quick question: do you know of any tool or method to watch a given range of memory area?

Side personal question to Zahlman: In what field do you work ? Do you have some experience with professionnal game programming ? I've worked in a couple of big professionnal commercial shipped projects and had far more problems with those that used STL that with those that did not. Of course I wouldn't draw any general conclusion from that fact.

Share this post


Link to post
Share on other sites
If the corrupted memory location is the same between runs you can place break points when a memory location changes. It's very useful for finding overwrites.

Share this post


Link to post
Share on other sites
Quote:
Original post by janta
I've worked in a couple of big professionnal commercial shipped projects and had far more problems with those that used STL that with those that did not. Of course I wouldn't draw any general conclusion from that fact.

Since it's the game development business we're talking about there's a possibility that they didn't have much experience with the template library to begin with, which could possibly be one of many explanations.

Share this post


Link to post
Share on other sites
Quote:
I've worked in a couple of big professionnal commercial shipped projects and had far more problems with those that used STL that with those that did not. Of course I wouldn't draw any general conclusion from that fact.


One problem might be the STL is old... Very old. We now work with the SC++L. Next problem, few take the time to learn how to properly use it. This is almost worse then not using it. It took me about a week to figure out the basics and with a bit of brain power the rest was easy to figure out as I used it.

Now I find people in the games industry are scared of certain changes. They're normally fine with hardware (though a lot ran crying over multi-cored processors I think lol) but things like using a new library freaks them out and they actively or passively resist it however they can.

I had a friend a year ago finish school and not have a clue what the SC++L was. He figured he could write better code. I then informed him the people that wrote it are far better coders then him and I are (at least currently, I have dreams darn it!) and its heavily tested code used by tens of thousands of other developers. He now loves it. Go figure. I think the slap to the back of his head helped a bit too but I have no scientific proof.

Share this post


Link to post
Share on other sites
Quote:
Original post by janta
I've worked in a couple of big professionnal commercial shipped projects and had far more problems with those that used STL that with those that did not. Of course I wouldn't draw any general conclusion from that fact.

Wow. This is unexpected (although VC6 had many problems in its SCL implementation, and VC7.1 did shoke on some blurbs too). The SCL is quite a good library, and when it's used correctly it doesn't add bugs to your software.
Now, I've only used it for a few years.


Share this post


Link to post
Share on other sites
Quote:
Original post by Emmanuel Deloget
Wow. This is unexpected (although VC6 had many problems in its SCL implementation, and VC7.1 did shoke on some blurbs too). The SCL is quite a good library, and when it's used correctly it doesn't add bugs to your software.
Now, I've only used it for a few years.


Well, again this is my personal experience and by nature my job involves a LOT of debuging, so that might be where the pain came from

Share this post


Link to post
Share on other sites
Quote:
Original post by janta
Quote:
Original post by Emmanuel Deloget
Wow. This is unexpected (although VC6 had many problems in its SCL implementation, and VC7.1 did shoke on some blurbs too). The SCL is quite a good library, and when it's used correctly it doesn't add bugs to your software.
Now, I've only used it for a few years.


Well, again this is my personal experience and by nature my job involves a LOT of debuging, so that might be where the pain came from


Of course, as with everything else in C++, if you're not careful, and don't know exactly what you're doing, the SC++L (or STL, if you prefer) can cause a lot of hard-to-debug problems. In that, it's no different from the standard C libary (and memcpy), or, well, any other part of the language.

So I'd guess that the bugs your team encountered might (at least partly) have been due to some team members who weren't familiar enough with the library.
(Which isn't at all a claim that they're bad coders or anything, just the plain fact that C++ will happily blow up in your face, if you aren't at least 110% sure what you're doing, regardless of whether you're working with the core language, the C or C++ standard libraries or anything else. And if some of them weren't used to the SC++ library, well, bugs might easily happen)

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this