# Can you tell me what he means by "precision"?

## Recommended Posts

I tried posting on the XNA site as well, but didn't really get any advice so I am hoping someone here can help. I've tried using this tool called Luminix Shader which is an exporter for materials from 3d studio max. I can now successfully load a model with multi-texturing, but the lighting is giving me these bizarre pixelation / striations on my model (dark red on top in picture below) on my development machine. The co-author stated that is may be a "precision problem" but didn't share any more and so I'm officially confused! I'd love to find an exporter from 3ds max that just works for exporting materials!!

Anyway, the education is appreciated. The picture and original forum post (including the shader fx and x file) is below.

http://www.lumonix.biz/yabb/Attachments/Capture.PNG

http://www.lumonix.biz/yabb/YaBB.pl?num=1287068261/0

##### Share on other sites
I assume that he's talking about "floating-point precision". Though I didn't have time to look at the article. It's a pretty simple concept, and is often a problem with arithmetic in programming. I'm sure you already understand that computers run off of binary (1s and 0s) instructions, right? Well, they also use 1s and 0s to represent data, such as this text you're reading. There are also different basic data *types* which are found in almost every programming language. Like "byte", "short", "int", "long", etc. For this to make any sense to a computer, each one of these types must have a fixed size, aka "fixed-width". A "byte" has 8 *bits*. A bit is every space it has to store a 1 or 0. For instance, this is a byte has the value of 125 in binary: 01111101

Here's where "precision" becomes a problem. Every data type has a certain "range", which is the distance between the smallest and largest value it can represent. A byte, for instance, can only represent the numbers from 0-255. Why? Because it only has 8-bits. That means the largest number possible to store in a byte is (in binary) 11111111, which is 255 in decimal. The smallest value possible for a byte is 00000000 (zero in decimal). All eight bits are used whether the value is 0 or 255, because it's fixed-width. So a byte isn't very precise for working with larger numbers, but it's good for small values and representing tiny chunks of data. For larger values, one would have to use a "short" (aka "word"), an "int" or maybe a "long", which have 16, 32 and 64 bits, respectively. But these can only represent whole numbers, and cannot have a decimal place or fractional part.

You may already know of the type "float", which CAN store fractional parts, and allows you to work with values like 2.5 or -0.000456. A float is technically termed a "single-precision floating-point data type". Just like the type "int" (or "dword", aka "double-word"), it has a total of 32-bits. But it uses them differently so it can represent fractional parts of numbers. So a floating-point type can usually either use its 32-bits to represent a larger whole number or it can use the bits for precision in storing fractional values, like the result of dividing 1 by 6439245 (1 / 6439245). But open Calculator and try that calculation. What happens? It has to represent that value using scientific notation (at least mine does). The reason is because it doesn't have enough bits in some inner variable to store the 100% correct answer in plain form (I'm guessing at that, lol. Just bear with me for the sake of example). And Calculator is probably using the type "double" (aka "double-precision floating-point data type") for such calculations, which have 64-bits (twice as many as a regular float). So to sum things up, every data type has a fixed-width in a program. It can't have any more bits of storage even if it needs to represent a bigger value. If you exceed the amount of bits allowed, you can have different things happen. Sometimes a system has an undefined and unpredictable behavior. Sometimes some type of "overflow exception" will occur. But most often, the extra bits are just truncated (chopped off), and you LOSE that data. Thus is the precision problem! Think if you tried to store the value of 300 in a byte (only 8-bits allowed, but 300 = 100101100 in binary, and that's 9 bits!). What would happen? This is what would happen -> 00101100 That is only 44! See, we just lost all that extra data. All basic data types, at some point, can suffer the same problem.

Hopefully you fully understand it now. You didn't state how much you knew about this topic, so I didn't know if I was answering a complete beginner or someone with more knowledge. So I apologize if this was too lengthy or too simplistic for your taste! :)

##### Share on other sites
It [i]looks[i] like you have geometry very close together and they're z-fighting. Basically this means that the precision of the depth buffer is inadequate to represent the distance between the overlapping triangles. Is that the case here? Are you perhaps drawing the same geometry with multiple passes?

##### Share on other sites
A few things:

First and foremost, thanks for the answers. Keinmann, WOW, great answer. You're right, I didn't really clarify my question and you post brought back my college years of hardware and software theory but it was a great answer, none-the-less and a good read. MJP, also, thank you for your answer.

My question becomes, where would be the best place to look to resolve my issue?

Lumonix claims that their ShaderFX product will output HLSL without me having to write any HLSL code. So I haven't changed the fx file at all (except to change the cullmode from cw to ccw). The model, from what I know, was created in MAX as a simple cube primitive, a material was applied, and the file was exported using Lumonix giving me an x file, an fx file, and 3 texture files.

I don't believe I am drawing my object twice. My "cube" class inherits a drawable base class called "model".

[Edited by - rm2000 on October 18, 2010 8:04:03 AM]

##### Share on other sites
For anyone interested, I've discovered the solution to the problem and posted my answer on the Lumonix site. It turns out the only issue was that I had to set the generate tangents property to true for the model properties.

## Create an account

Register a new account

• ### Forum Statistics

• Total Topics
628275
• Total Posts
2981762

• 10
• 11
• 17
• 10
• 9