I want to try out different formats for storing normals at vertices and compare quality to pick the one that suits me the best. Most formats are either simple to pack to have functions that do that for you (for example - D3DX_FLOAT4_to_R10G10B10A2_UNORM). With this one I do not know how to do it correctly.

Looking at the source, that function also saturates your input to [0, 1], so you'd have to do that first to use the same method, with (x + 1) / 2. Then modify the scale to match R11G11B10, which means 2047 for x, 2047 for y, shift with 11 instead of 10 for y, and still 1023 for z but shift by 22. W is skipped.

In the unpack function replace the shifts and scales the same way, shift 11 for y and 22 for z, divide by 2047 for x and y, and the mask for x and y is 0x7ff instead of 0x3ff.

And to restore your values from [0, 1] to [-1, 1] use x * 2 - 1.

Actually never mind, you're trying to use a float format. Sorry about that.