I know this thing used to be popular when deferred shading hit the scene, as people would do things like writing linear depth values into RGBA8 render-targets (i.e. splitting a float into 4 8-bit integers, then reconstructing the float afterwards).
I'm trying to do this on SM3, which means no bitwise logic (i.e. |, >>, &, etc).
In my specific case, I'm trying to split a float into four 2-bit values, though the logic should be the same as if I were trying to split it into four 8-bit values.
In both cases below, I'm assuming that the input value [font=courier new,courier,monospace]data[/font] is from [font=courier new,courier,monospace]0.0f[/font] to [font=courier new,courier,monospace]1.0f[/font] inclusive.
The code that I keep finding everywhere (converted from 8-bit to 2-bit) is as follows:
//splitting:
float4 shift = float4( 1, 4, 16, 64 );
float4 abcd = frac( data * shift );
abcd -= abcd.xxyz * float4(0, (float3)1/4.0);
//reconstructing:
float reconstructed = dot( abcd, float4(3*64,3*16,3*4,3)/255.0 );
However, this code is buggy. It's almost right, and it looks mostly right when using the 8-bit version, but the flaw becomes much more exaggerated when you bring it down to a 2-bit version. The problem is that frac never returns 1.0, so you skip over certain values (causing banding), and you can never encode the input value of 1.0f.The code that I came up with is as follows, and it was mostly through a process of trial-and-error that I arrived at this result.
//splitting:
float4 abcd = frac(data * (255.0/float4(256,64,16,4)));
abcd = floor(abcd*4)/3;
//reconstructing:
float reconstructed = dot( abcd, float4(3*64,3*16,3*4,3)/255.0 );
If anyone has any links to other methods, or general advice in this area, please share!
N.B. The magic numbers in use are:
255 = max 8-bit value (2-bit * 4 components)
3 = max 2-bit value
4 = num 2-bit values
1,4,16,64,256 = shift by 0, 2, 4, 6, 8 places.