Sign in to follow this  

Challenging STL magic

This topic is 3737 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I have used STL widely in a large WinXP project. Last week I decided to try again STLport (version 5.1.3). I have been using the STL that comes with VS2005 and it has worked quite well... However, as I expected, STLport is quite a lot faster and I would like to keep using it, but: While all else seems to work. Something goes wrong with the vector normals that are calculated for bumpmap textures. Now I am narrowing down the possible causes: Visual inspection and logfile written values suggest that bumpmap textures have some correct values and repeated patterns of wrong values. Hardcoding some patterns to bumpmap generating function seems to work sometimes. Before that failed values come from sinf and cosf using calculations. After that the values go are stored in an array and put into DirectX 9 managed textures using LockRect and memcpy (no STL there). I have multiple threads, but this all is done in a single thread without interference. I have very few pointers in the whole program. Turning STLport debug range checking does not catch any out-of-range uses. STL is not used to generate the data. However std::map is used to store information about bumpmap texture piece updates. (Not actual data, just updated index numbers for both key and value.) Everything works ok in debug mode and also in release mode if I disable VS2005 compiler optimizations. If I do not disable optimizations but change floating point model from the VS2005 project settings, then the wrong values change and sometimes work but not reliably. Without STLport everything works, but is slower. Questions: Are other people using STLport with a VS2005 project? Have you encountered similar problems? Good ideas about finding a bug that does not show in debug mode at all? Thanks in advance for help and encouragement... [cool] Osmo Suvisaari waspgames.com

Share this post


Link to post
Share on other sites
I looked at STL port some time back. The supported compilers list stops at VC6 and gcc 3.0.

Then I stopped looking into it.

I'm not saying that that list is correct, or definitive, but MVS 7,8 and 9 are perhaps worth mentioning if you're dealing with anything even remotely C++ related. It's not like they are new products. VC6 is only 10 years old now.

Share this post


Link to post
Share on other sites
Quote:
Original post by Osmo Suvisaari
I have very few pointers in the whole program.
Turning STLport debug range checking does not catch any out-of-range uses.
STL is not used to generate the data.
However std::map is used to store information about bumpmap texture piece updates. (Not actual data, just updated index numbers for both key and value.)
Hmm... Are you changing the the data values (std::map <key_type, data_type, [comparison_function]>) inside the map without first removing them? This is at least one way of possibly corrupting the data as the inner data structure invariant doesn't necessarily hold in that case any more. (I'd bet it's a tree and changing some internal node in which the ordering is based doesn't trigger any tree rotations.)

This is something VC++ compiler didn't catch earlier at least whereas some g++ compilers will report as an error when compiled with strict enough compiler options.
Quote:
Original post by Osmo Suvisaari
Questions:
Are other people using STLport with a VS2005 project?
Have you encountered similar problems?
Good ideas about finding a bug that does not show in debug mode at all?
Those pesky heisenbugs. ;) But anyway, that is the only thing I can think of and that is a pretty common data structure related error.

Share this post


Link to post
Share on other sites
Quote:
Original post by Antheus
I looked at STL port some time back. The supported compilers list stops at VC6 and gcc 3.0.

Then I stopped looking into it.

I'm not saying that that list is correct, or definitive, but MVS 7,8 and 9 are perhaps worth mentioning if you're dealing with anything even remotely C++ related. It's not like they are new products. VC6 is only 10 years old now.
STLport-5.1.3 seem to be supporting also the compiler shipping with VS2005. Some release notes. And besides, I tested the new TR1 stuff with my version of VS2005 in order to do some algorithms regarding automata minimisation (pesky coursework I want to do in environment initially Windows and not in some *nix) since I need hashmaps. [smile]

Share this post


Link to post
Share on other sites
I think that is a bug in my own code or configuration, and STLport is ok...

But it is certainly possible that there is something wrong with using STLport with VS2005. It would be good to know for certain before I spend days and nights debugging this... [cool]

This bug is probably a pointer bug, out-of-range bug or a multithreading bug in my own code, or in the configuration.

It is ironic that changing STL broke something that is the least STL using part of the project. That means probably that the error is somewhere else but only causing problems at that point due memory corruption, or... a STLport compatibility issue?

-Osmo Suvisaari
waspgames.com

Share this post


Link to post
Share on other sites
Progress update, pragmatic approach:
I have not found any bugs in my own code that would cause this.
However I have isolated the code that causes errors when optimized using

#pragma optimize("", off)
and
#pragma optimize("", on)




Everything works ok if I disable optimization before the end of one class declaration in one class header. I turn the optimization back on right after that like this:

#pragma optimize("", off)
}; //class declaration ends here
#pragma pack(pop)
#pragma optimize("", on)




I wonder how that does affect the code as the disabled part does not contain any functions? I think it should not have any effect as Visual Studio documentation states that:
Quote:

The optimize pragma must appear outside a function and takes effect at the first function defined after the pragma is seen.


Only special thing in that class is the use of #pragma pack lines before and after class declaration.
And removing that does not fix the problem, and I have used #pragma pack in similar fashion in 5 classes.

Conclusion:
The problem is now gone, but I am not satisfied by the solution, because it is probable that the bug is still there and this has only hidden it...

-Osmo Suvisaari
waspgames.com

Share this post


Link to post
Share on other sites
Ok, here is the whole header file ActiveMapBlock_logical.h:


#pragma once

#include "Globals.h"
#include "ActiveMapBlock.h"

#pragma pack(push)
#pragma pack(1)

// Logical map block used in game mechanical calculations
class CActiveMapBlock_logical : public CActiveMapBlock
{
unsigned short CalculatedRelativeMovementCost;

char Rainfall;
char Warmth;

unsigned char DefaultPopulation; //read from defaults, not saved
unsigned char DefaultTechnologyLevel; //read from defaults, not saved
int DefaultFlags; //read from defaults, not saved

unsigned char Population;
unsigned char TechnologyLevel;
unsigned char Roughness;

int Flags;
int PreviousFlags;

public:

void SetFlagOn( CActiveMapBlock::enumFlag gFlag)
{
Flags |= gFlag;
}

void SetFlagOff( CActiveMapBlock::enumFlag gFlag)
{
Flags &= (0xffffffff ^ gFlag);
}

void SetPreviousFlags(int gFlags) {PreviousFlags=gFlags;}
void SetFlags(int gFlags) {Flags=gFlags;}

int GetCurrentFlags(float gPhase) {if(gPhase<=0) return PreviousFlags; else return Flags;}
int GetFlags() {return Flags;}
int GetPreviousFlags() {return PreviousFlags;}

virtual void LoadDefaultFrom(CBuffer & gBuffer);
virtual void SaveDefaultInto(CBuffer & gBuffer);

virtual void LoadFrom(CBuffer & gBuffer);
virtual void SaveInto(CBuffer & gBuffer);

void SetRainfall(float gValue)
{
Rainfall=static_cast<char>(gValue*127.0f);
}
float GetRainfall()
{
return Rainfall/127.0f;
}

void SetWarmth(float gValue)
{
Warmth=static_cast<char>(gValue*127.0f);
}
float GetWarmth()
{
return Warmth/127.0f;
}

float GetCalculatedRelativeMovementCost()
{
return static_cast<float>(CalculatedRelativeMovementCost)*0.01f;
}

void SetCalculatedRelativeMovementCost(float gValue)
{
if(gValue<=0.01f) CalculatedRelativeMovementCost = 1;
else if(gValue>=255.0f) CalculatedRelativeMovementCost = 25500;
else
{
CalculatedRelativeMovementCost=static_cast<unsigned short>(gValue*100.0f);
}
}

float GetPopulation() {return static_cast<float>(Population);}
float GetTechnologyLevel() {return static_cast<float>(TechnologyLevel)/255.0f;}
float GetRoughness() {return static_cast<float>(Roughness);}

void SetPopulation(float gPopulation)
{
if(gPopulation<=0) Population=0;
else if(gPopulation>255) Population=255;
else Population=static_cast<unsigned char>(gPopulation);
}
void SetTechnologyLevel(float gTechnologyLevel)
{
if(gTechnologyLevel<=0) TechnologyLevel=0;
else if(gTechnologyLevel>=1) TechnologyLevel=255;
else TechnologyLevel=static_cast<unsigned char>(gTechnologyLevel*255.0f);
}

void ImproveTechnologyLevel(float gImprovement)
{
SetTechnologyLevel( GetTechnologyLevel()+gImprovement );
}

void SetRoughness(float gRoughness)
{
if(gRoughness<=0) Roughness=0;
else if(gRoughness>255) Roughness=255;
else Roughness=static_cast<unsigned char>(gRoughness);
}

void CalculateRelativeMovementCost();

static float GetRoadMovementCost(float gPopulation, float gTechnology, float gRoughness);
float GetRoadMovementCost();

CActiveMapBlock_logical(void);
virtual ~CActiveMapBlock_logical(void);

#pragma optimize("", off)
};
#pragma pack(pop)
#pragma optimize("", on)




-Osmo Suvisaari
waspgames.com

Share this post


Link to post
Share on other sites
How about removing the pragma pack(1) and the corresponding pop statement? It could be possible that your other code excepts to see some other (default) alignment than the one you have you have explicitly declared. Moreover -- I can't know for sure -- it could be that your packs and pops interleave so that something goes wrong.

Anyway, it feels at least that the bugs that appear during optimisations are due to the fact that the optimisations rely on some other data alignment than the one you have explicitly set in your code. The other parts can't possible know that you have set some other member alignment than the default. Please see pack (C/C++) at MSDN for more information.

[EDIT]Consider, for instance, the one-line functions that will most likely be inlined during the optimisations. Now there's a real change the other code is doing something with your data that is aligned differently than it excepts it to be (due to the fact that there are private member with different alignment involved there).[/EDIT]

[Edited by - Naurava kulkuri on September 22, 2007 1:38:23 PM]

Share this post


Link to post
Share on other sites
void SetFlagOff( CActiveMapBlock::enumFlag gFlag)
{
Flags &= (0xffffffff ^ gFlag);
}

Hmm... I guess you know the architecture you are using also..? [smile]


int GetCurrentFlags(float gPhase) {if(gPhase<=0) return PreviousFlags; else return Flags;}

Making a floating point comparison isn't that exact also (in any of the functions you are doing it).

I also wonder, why so much casting? I assume you are trying save some space here with all those data type choices, packing and then casting, but have you considered that there will some extra instructions also there that take some space? It could be that your data structures are huge and using the data types you are using is justified, but otherwise I'd go with floats or whatever you are casting to all the time.

It feels also a bit suspectible that the combination of all that excessive casting (and perhaps alignment) you lose some accuracy at some point. The problem is further compounded with those floating point comparisons that aren't accurate in any case unless you can guarantee some discrete set of values that are separate enough for those comparisons to be accurate.

[Edited by - Naurava kulkuri on September 22, 2007 2:25:05 PM]

Share this post


Link to post
Share on other sites
This is a bit off topic but...

Yes, you guessed the reason.
The data size has been the only bottleneck in this one. Executable code size will not be a problem in the project. (Speed is, but not in this part.)

Those floating point comparisons are accurate enough:
Zero and negative values are accurately recognizable and the upper limits just have to be checked for overflows when assigning values to less accurate small integer types.

It's all justifiable and correct. Believe me. [cool]


-Osmo Suvisaari
waspgames.com

Share this post


Link to post
Share on other sites
Quote:
Original post by Osmo Suvisaari
This is a bit off topic but...

Yes, you guessed the reason.
The data size has been the only bottleneck in this one. Executable code size will not be a problem in the project. (Speed is, but not in this part.)

Those floating point comparisons are accurate enough:
Zero and negative values are accurately recognizable and the upper limits just have to be checked for overflows when assigning values to less accurate small integer types.

void SetWarmth(float gValue)
{
Warmth=static_cast<char>(gValue*127.0f);
}
float GetWarmth()
{
return Warmth/127.0f;
}


In this case, for instance, then if gvalue is something like, say, 1.01 in SetWarmth your intention is to get very likely a negative value returned from GetWarmth? Or that if the value is less than 1.0, you don't mind those rounding errors?
Quote:
Original post by Osmo Suvisaari
It's all justifiable and correct. Believe me. [cool]
I'm just a bit concerned that you use the values somewhere, as you obviously are using and even if you check for upper and lower limits, the results aren't correct due to compounded error somewhere else in your program.

Consider, for instance that if you are using those floating point values in STL data structures as comparison keys, there can be comparisons that are equal when they shouldn't be or not equal when they should be. Moreover, if the values are sorted, say, according to this Warmth value, you are really sure you have taken into account all this (and documented it with care) and that the elements you get out in that case aren't perhaps the ones you really want?

It could be that the problems aren't even in this class (I can't know) but are just showing here. Neither the IEEE 754 nor the C++ standard specify how transcendental functions (sin, cos, exp etc.) should be implemented and exactly what should the calculations yield in each and every implementation. Quicly looking over to STLPort directories I see there are "cmath files" there. It could also be that there are some implementation differences playing a role here.

There has been also a discussion related to this recently, Esoteric concepts in c++.

Share this post


Link to post
Share on other sites
Quote:
Original post by Vorpy
Did the code work properly before optimizing it for space?
Or was is optimised for space right from the beginning? ;)

I also suspect that since there are packing and casts and everything, some of the floating point behaviour doesn't play nicely along with the optimisations. Like that it could be actually implementation defined when certain switches are turned on. Also disabling the optimisations (optimize (C/C++)) with a pragma and then turning them on seem indicate towards this possibility.

Of course Visual Studio cannot catch the incompatible switches if there aren't any, but something is done in the code. In any case I feel it needs too much digging to documentation to find this out, but I'd be delighted if someone could confirm my suspicions or take some alternative and valid guess.

Share this post


Link to post
Share on other sites
Yes, the class itself is 3-years old and the size reduction was made over a year ago, and has worked ok.

Overflowing the warmth would just cause deep snow and glaciers appear in the hottest parts of tropical areas... No problem there.

The rounding errors do not accumulate since I am not using the approximate values as new input.

...

The surface normals that go wrong are not calculated from these values in any way. Only problem in this class is the fact that with STLport if it is compiled with all VS optimizations, then those vector values totally elsewhere in the program get abnormal real values and NaN values for x y z.


-Osmo Suvisaari
waspgames.com

[Edited by - Osmo Suvisaari on September 23, 2007 4:20:10 AM]

Share this post


Link to post
Share on other sites

This topic is 3737 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this