Strange Visual Studio Release Mode Behavior

Started by
10 comments, last by SmkViper 9 years, 8 months ago

I am encountering a strange issue when attempting to build my project for release mode. It works fine in debug mode, but my Vector3D object seems to be returning garbage when I am setting one vector equal to the other.

I've been wondering if it has something to do with not having a operator=, except the default one given by the compiler, but I'm not sure. It is strange. I was also initially wondering if the compiler was executing my code out of order due to release mode compiler optimizations, but then I noticed that my Vector3D was just garbage values for some reason.

This is what I am doing:


Vector3D playerPos = mainPlayer->getObjectPosition(); // Doesn't do anything in release mode (playerPos remains as garbage)
playerPos = this->WorldToScreen(&playerPos);

All that getObjectPosition() does is:


Vector3D& GameObject::getObjectPosition()
{
  return this->position;
}

Thank you for any insight you can give me on solving this issue.

Advertisement

Is it possible that mainPlayer->position hasn't actually been initialized at that point in the program? When I see an accessor appearing to spit out garbage in release but not debug, that's usually the first thing I check.

Also, how are you checking for these garbage values? If you're only seeing them in the debugger and the program is otherwise behaving as expected (or even if it isn't), that's not too unexpected. Very optimized code can make a debugger very confused.

I just checked, I added a std::cout statement above to check, and it is initialized to the expected result. It seems to be having issues placing the result into the Vector3D for some reason.

The problem seems to be duplicated within debug mode when I create a custom assignment operator:


Vector3D& Vector3D::operator=(const Vector3D& rhs)
{
  this->set(rhs.x, rhs.y, rhs.z);
  return *this;
}

If I get rid of that assignment operator it works fine in debug mode.

Here is some more code on what I am doing:


  float x = mainPlayer->getObjectPosition().x, y = mainPlayer->getObjectPosition().y, z = mainPlayer->getObjectPosition().z;
  std::cout << x << " | " << y << " | " << z << std::endl;
  Vector3D playerPos(x, y, z);// = mainPlayer->getObjectPosition();
  std::cout << playerPos.x << " | " << playerPos.y << " | " << playerPos.z << std::endl;
  Vector3D nPlayerPos = this->WorldToScreen(&playerPos);
  std::cout << nPlayerPos.x << " | " << nPlayerPos.y << " | " << nPlayerPos.z << std::endl; // INVALID RESULT

I was inspecting the matrices within WorldToScreen and they appeared out of order in the release mode, however, that may have just been the compiler debug mode in release confusion.

That looks fine to me given the context you've provided. What happens inside of set()? What are you expecting to see and what sort of "garbage values" are you seeing, exactly?

set() sets the x y and z component of the Vector.


void Vector3D::set(float x, float y, float z)
{
  this->x = x;
  this->y = y;
  this->z = z;
}

The value I'm expecting to see after the WorldToScreen is: | 512 | 384 | 0, which is what I get in debug mode without the custom assignment operator, but in release mode I'm getting:
| 4.57501e+013 | 2.47714e-039 | 4.26061e+013

I can't help but wonder now if the release mode is shifting my matrices around for some reason, I'm not sure tho, I don't think they would be nor how. I'll print those out to see if they are for some reason.

A full rebuild might help if something is corrupt between different libraries.

I second the 'probably uninitialized' situation.

Of course, it might not be that object that is uninitialized. Something else could be uninitialized, which in turn causes something else to stomp on it.

While it is harder to diagnose, you can still use a debugger in release build. I'd go with some breakpoints when you think you initialized it (verifying it by stepping through the dissassembly window) and some memory breakpoints on the object itself to see if someone else is stomping on it.

There definitely seems to be something wrong with my library when my project is accessing it. I call the WorldToScreen function in my library and get those invalid results, but if I pull out that code from within my library and execute it manually in my other project it produces the correct results.

Vector3D playerPos(x, y, z);// = mainPlayer->getObjectPosition();
std::cout << playerPos.x << " | " << playerPos.y << " | " << playerPos.z << std::endl;
Vector3D nPlayerPos = this->WorldToScreen(&playerPos);
std::cout << nPlayerPos.x << " | " << nPlayerPos.y << " | " << nPlayerPos.z << std::endl; // INVALID RESULT


This is an invalid result from WorldToScreen - and not the vector assignment itself - from what you're implying with this code. It could be the vector but from this context it appears to almost certainly be a problem in your function there. Especially with you explicitly passing a pointer to it (why?) I'm thinking you may doing some kind of illegal aliasing violation in that code.

Vector3D playerPos = mainPlayer->getObjectPosition(); // Doesn't do anything in release mode (playerPos remains as garbage)


Depending on a few things, this is probably going to invoke your copy constructor operator, not your copy assignment operator.

Vector3D& GameObject::getObjectPosition()


It's not common for get*() functions to return a non-const (lvalue) reference. Are you sure this is the behavior you want? It's not likely the cause of the bug, but depending on various overloads you have defined, it may be suspect. Prefer to return a value or a const reference and have a separate set*() function. It's common to think that a const reference is the safest choice, but check how well your compilers handle returning by value. Compilers are getting better at it and in some cases will handle return-by-value more efficiently than return-by-const-reference.

my Vector3D object


Why are you writing your own in the first place? Even on the AAA space I want to whack people who write their own (I do understand the draw - I've done it myself far more often than I should). These kinds of bugs just don't exist in well-tested third-party libraries. Plus, it's almost guaranteed that they'll be better optimized and more featureful. Math libraries are a commodity; don't write your own.

If in doubt, http://glm.g-truc.net/0.9.5/index.html is a perfectly valid choice, even if using D3D. DirectXMath is also a good choice, even if using OpenGL. vectormath works (in Bullet) and many other 3D middlewares include their own.

I can't help but wonder now if the release mode is shifting my matrices around for some reason, I'm not sure tho, I don't think they would be nor how. I'll print those out to see if they are for some reason.


This is very possible depending on what kinds of tricks you're pulling in your matrices. I've seen a lot of "fancy" math code that abuses aliasing rules and can lead to all kinds of perfectly valid but unintended output after optimization.


I've been wondering if it has something to do with not having a operator=, except the default one given by the compiler, but I'm not sure.


Rule of Zero. You _want_ to use the compiler-generated copy/move constructors/operators (and default constructor and destructor) if at all possible, since they'll be correct more often (pretty much always) than your own (easily buggy due to subtle requirements on those constructors/operators).

There's absolutely no good reason to write a custom copy operator for something like a Vector3d. Maybe you can justify a default constructor if you want to ensure that Vectors are zero-initialized. In that case you should still strongly prefer prefer NSDMI (non-static data member initializers) and a compiler-generated default constructor.

Sean Middleditch – Game Systems Engineer – Join my team!

Thanks for the insightful posts. It turns out the problem was definitely in the WorldToScreen function, I was returning a local variable as a reference, which I should have known not to do, but that was the problem.

I changed the function to:


Vector3D GameZone::WorldToScreen(const Vector3D& worldPos);

Instead of:


Vector3D& GameZone::WorldToScreen(const Vector3D& worldPos);

Thanks for the insightful posts. It turns out the problem was definitely in the WorldToScreen function, I was returning a local variable as a reference, which I should have known not to do, but that was the problem.


Get in the habit of compiling with the highest warning level and with all warnings treated as errors. Microsoft's compiler is quite capable of diagnosing this mistake for you and catching it long before it turns into a forum thread. smile.png

Sean Middleditch – Game Systems Engineer – Join my team!

This topic is closed to new replies.

Advertisement