I thought that i have completed the 3d picking but after testing it few times ,i found that it is a little bit(too much) inaccurate.

i am using ray casting with bullet everything is ok with that, but the thing is i can't get the accurate 3d coordinates according to the mouse position on the viewing plan .

here's my code 

i do the same thing for the topleft corner.

retInversePro() is returning inverse(view)*inverse(projection);

this much error i am having 

(the white line just shows the box diagonal in the 3d world on the viewing plane,which shows the error the end of it should be within the cursor point but it is not)

The end of the white line shows the where mouse button is released.

and this is how i am storing the mouse position 

i can also post code about how  i am mapping the position of cursor (device coordinates ) to the NDC.

It would be a lot of help for me if someone could tell me what is going wrong.

Thanks in advance.

I can't see anything incorrect in the code you've posted.

The error appears to be only in the Y direction. Do you get the same error at all positions on the screen moving vertically from top to bottom? Does the error change when you point the mouse at geometry closer to the screen, or does it remain constant? Do you get an error in the X direction at any point?

Ya as i move towards the window corners it also shows disturbance in x and y both the directions. 

 I Don't think its related but 

1 hour ago, Uttam Kushwah said:

returning inverse(view)*inverse(projection)

☝This is returning a floating point matrix and all other things are double based data. Could it be the bug ?. 

13 minutes ago, Uttam Kushwah said:

Ya as i move towards the window corners it also shows disturbance in x and y both the directions. 

 I Don't think its related but 

☝This is returning a floating point matrix and all other things are double based data. Could it be the bug ?. 

I would not expect floating point error to produce this bug, no. Can you debug plot the ray you're casting? That could help distinguish a ray calculation error from a ray collision error?

Actually I am currently rendering ortho projection which do not show the ray rendered as they are perpendicular to the camera viewing plan,  I have to switch to perspective projection.however I am able to render them. I will post images. 

1 minute ago, Uttam Kushwah said:

Actually I am currently rendering ortho projection which do not show the ray rendered as they are perpendicular to the camera viewing plan,  I have to switch to perspective projection.however I am able to render them. I will post images. 

This is kind of my point: if you can see them then something is wrong, no?

i am getting the correct result now.

i don't how and what is the problem but if i run the program with integrated graphics the ray casting doesn't work properly,

but for recording purpose i tried to run the program with my nvidia GPU and i was shocked, it worked ! but i thought that how it worked properly this time,than tried it again with integrated graphics than i figured out that its the gpu's thing.

now the question is why this thing is happening.

1 minute ago, Uttam Kushwah said:

i am getting the correct result now.

i don't how and what is the problem but if i run the program with integrated graphics the ray casting doesn't work properly,

but for recording purpose i tried to run the program with my nvidia GPU and i was shocked, it worked ! but i thought that how it worked properly this time,than tried it again with integrated graphics than i figured out that its the gpu's thing.

now the question is why this thing is happening.

That's... odd. Is your window the same size in both cases? Does glfwGetCursorPos() report the same values for all four corners with both cards? Does your method of finding the values for retHeight() and retWidth() match up with these values?

ya the window is of the same size and it works in any case whether i go full screen or something else.

5 minutes ago, Irusan, son of Arusan said:

Does glfwGetCursorPos() report the same values for all four corners with both cards? Does your method of finding the values for retHeight() and retWidth() match up with these values?

i didn't try that much but i guess i have too.And one thing how am i supposed to find these thing exactly twice (because it is impossible to drag same size of box at same position on two different tries unless hardcode them).