Thank you Samith, just one question. In which space t and s are distances eye-near/eye-far? If I look in my projection matrix (built via glm::perspective) at position 3,3 and 3,4 what I see is something like -1.03 and -1 where my clipping planes are [0.1, 8]
Yep. The s and t values are computed given the near and far planes. In my post above I said that s and t are two values that satisfy the equations (s*f + t)/f = 1 and (s*n + t)/n = -1, but I didn't provide any actually solution for s and t
If you are interested, the solutions for s and t can be found like so:
(sf + t)/f = 1
(sn + t)/n = -1
// here you have two linear equations and two unknowns. you can solve
// for s and t with your basic high school algebra techniques
eq1: sf + t = f
eq2: sn + t = -n
// subtract eq2 from eq1
eq3: s(f-n) = f+n
s = (f+n)/(f-n)
// substitute s into eq1 and solve for t
eq4: sf + t = f
f(f+n)/(f-n) + t = f
f(f+n) + t(f-n) = f(f-n)
f(f+n) - f(f-n) = -t(f-n)
ff+fn - ff+fn = -t(f+n)
-2fn/(f+n) = t
// now you can check s and t given a near and far plane of 0.1 and 8.0
// s = (f+n)/(f-n) = (8-0.1)/(8+0.1) =~ 1.025
// t = 2fn/(f+n) =~ -0.19
Some caveats: these computations depend on some nitpicky details about your projection. For example, if you use your projection matrix to transform a right handed coordinate space into the left handed clip space, then the original equations will be slightly different (ie s*-f + t = f instead of sf + t = f, etc). Also, the s and t values will be found in your projection matrix in the 3,3 location and 3,4 location if your projection matrix is column major, but they'll be in 3,3 and 4,3 if your projection matrix is row major.
I don't know enough about your exact environment to give you all the answers, but I hope the algebra I've posted at least gets you started down the right path.