Okay then, my next guess would be to change this (but after reverting the previous changes I suggested).
vec.Y = (2.0f * mouse.Y / (float)viewport.Height - 1);
Try removing the '-' and see if that works.
Also, sorry I didn't better explain what I was doing with the matrix suggestion. 13 is the column based matrix's location of the Y translation coordinate.
| 1 0 0 X |
| 0 1 0 -Y |
| 0 0 1 Z |
| 0 0 0 1 |
I simply suggested you invert it. I also had to do this to get my vertex projection code to work properly.
Hopefully, my suggestion helps a bit better this time. Sorry for any confusion.
EDIT: This may or may not matter for you, but I had to invert the incoming Y coordinate and the Y component of the translation vector in my modelview matrix in order to fix the problem.