I actually understood what you meant by inverting one of the component of matrix but I don't know how to do that in opentk because modelViewMatrix variable is a Matrix4 struct. This doesn't change anything:
modelViewMatrix.M13 = -modelViewMatrix.M13;
I removed that minus in UnProject function, so now:
and if I change
mouse.Y = viewport - y;
mouse.Y = viewport - y - 130;
then it gets converted almost fine. I've implemented zooming, so when I zoom in the view, I can see that X coord isn't very accurate too.
maybe there is another way to convert that?
Also, I forgot to mention that my application is 2D if that makes any sense.
EDIT: it's also possible that unproject function is wrong, I took it from opentk forum written by someone... I see learning opengl is really difficult .