Hi, I am trying to setup picking in my project, but there is strange issue, which not let me do this. Vecto3.Unproject returns NAN always.

I found out that its because of Matrix.Inverse inside that method and as a result inversed matrix have 0 in all fields.

All transmitted parameters are correct.

So, I wonder isis that correct that from WVP matrix during inversion I always receive Zero matrix?

I leave here some code to be sure I transfer parameters correct:

ZNear = 0.01;

ZFar = 10000000000000;

mouse coords and ViewPort also correct;

But both vectors receive NAN as a result value;

private void Pick(int mouseX, int mouseY, ref Matrix wvpMatrix, BoundingBox boundingBox)
        {

            Vector3 mouseNearVector = new Vector3(mouseX, mouseY, basicCamera.ZNear);
            Matrix m = Matrix.Identity;
            m = Matrix.Invert(wvpMatrix);
            Vector3 pointNear = Vector3.Unproject(mouseNearVector, 0, 0, GraphicsDevice.Viewport.Width, GraphicsDevice.Viewport.Height, basicCamera.ZNear, basicCamera.ZFar,
                wvpMatrix);

            Vector3 mouseFarVector = new Vector3(mouseX, mouseY, basicCamera.ZFar);
            Vector3 pointFar = Vector3.Unproject(mouseFarVector, 0, 0, GraphicsDevice.Viewport.Width, GraphicsDevice.Viewport.Height, basicCamera.ZNear, basicCamera.ZFar,
                wvpMatrix);

            Ray ray = new Ray(pointNear, Vector3.Normalize(pointFar - pointNear));
            float distance = 0;
            bool intersect = boundingBox.Intersects(ref ray, out distance);
            D2DText = "Intersect: " + intersect.ToString() + " distance = "+distance;
            intersect = false;
        }

Does Vector3.Unproject() method working correct or I am missing something?

Also it works really strange - sometime it returns true if no model under mouse and in the same time returns NAN for distance or return true and real distance value when no model under cursor is present.

It very strange that sometime picking is working at least somehow and the other time its not working at all.

No effect - both points are still NAN with 1 and 1000. But with 0 and 100 far vector is calculated, but near vector still NAN with any value I give it.

So, I wonder isis that correct that from WVP matrix during inversion I always receive Zero matrix?

Not all matrices are invertible. The determinant of the matrix should be non-zero for it to be invertible. Try using Matrix.Inverse on the identity matrix - the result should be Identity. If that works, maybe your input matrix is the problem.

Also, your ZFar value is way too large. D3D (and maybe SharpDX as well?) converts 64-bot doubles to 32-bit floats internally, and the maximum integer that can be represented accurately as a 32-bit float, without precision loss is 16777216. Once you go above this value, weird things start to happen because of the precision loss.

Not all matrices are invertible. The determinant of the matrix should be non-zero for it to be invertible. Try using Matrix.Inverse on the identity matrix - the result should be Identity. If that works, maybe your input matrix is the problem.

Also, your ZFar value is way too large. D3D (and maybe SharpDX as well?) converts 64-bot doubles to 32-bit floats internally, and the maximum integer that can be represented accurately as a 32-bit float, without precision loss is 16777216. Once you go above this value, weird things start to happen because of the precision loss.

Ok, I tested identity matrix and it works correct, so the problem could be in my projection matrix where I set znear and zfar values.

And I have one question regarding all this:

does mouseNearVector and mouseFarVector needs to contain ZNear and ZFar values corresponding? and Unproject method also needs to keep the same ZNear and ZFar? Or values can differ?

And why I can`t use for ex 100 000 as zFar or 0.01 as ZNear? In that case how I will create ray which could reach the far plane of camera view if this is possible by backbuffer and projection matrix?

does mouseNearVector and mouseFarVector needs to contain ZNear and ZFar values corresponding? and Unproject method also needs to keep the same ZNear and ZFar? Or values can differ?

Sorry, I can't tell you if your existing values are ok. You'll have to test that yourself.

IMHO yes, you should try to keep all your values between -16777216 and 16777216. If possible, you should use values even closer to 0 then that, so that any operations done by D3D on them don't go under/over these values either. There is a good explanation of this here: http://stackoverflow.com/questions/12596695/why-does-a-float-variable-stop-incrementing-at-16777216-in-c

Also, I think that your initial ZFar value is already NaN by the time you pass it to Unproject. There's no way such a large number can fit in a float. Maybe you also get a compiler warning if it's a float and the initial value is too large so it sets it to NaN? Or if it's a double, you'll get a warning about it being converted to float when you pass it to Unproject...

About ZNear: if you set it too small, you will get terrible aliasing. I don't remember why this is, but the recommentadion is that you set it to at least 0.5.

In that case how I will create ray which could reach the far plane of camera view if this is possible by backbuffer and projection matrix?

You don't need to create a vector (ray) that reaches that far by Unproject. Just create a normalized one, and then you can use scalar multiplication to make it whatever length you want. You can also convert your normalized vector to use doubles instead of floats for this if you need it to be longer than 16777216 (does SharpDX have a Vector class which uses doubles?). Just keep in mind that when you pass those double values back to D3D, they will be converted back to float...

Also, I think that your initial ZFar value is already NaN by the time you pass it to Unproject. There's no way such a large number can fit in a float. Maybe you also get a compiler warning if it's a float and the initial value is too large so it sets it to NaN?

No, ZFar is valid when I pass it to unproject and also my projection matrix works quite well without any problems, but i will try to descrease it to the valid float value.

OK, I tried and it works, but why noone points me that 6th and 7th parameters in Unproject() is not ZNear and ZFar, but Viewport.MinDepth and ViewPort.MaxDepth which is 0 and 1.0f correspondingly?

So, at last I coul unproject it and now it is not NAN, but I have the second question:

How to increase its length?

It return false when I far from the model and when I near - it always treu even if I not looking at the model. I suspect that I am inside bounding box and thats why it returns true.

SO, could you please tell how correctly make ray a maximum length possible?