Focusing on number 1 (player shooting raycasting) I tried doing something close to what you said, but it's not working as expected, can you show an example?
This is how I do it in my engine:
void Camera::getPickRay(int screenX, int screenY, Vec3& rayOrigin, Vec3& rayDirection, Math::CoordinateSpace space)
float vx = (((2.0f * screenX) / (float)mGraphicsEngine->getScreenWidth()) - 1.0f ) / mProjectionMatrix._11;
float vy = (((-2.0f * screenY) / (float)mGraphicsEngine->getScreenHeight()) + 1.0f ) / mProjectionMatrix._22;
rayOrigin.set(0.0f, 0.0f, 0.0f);
rayDirection.set(vx, vy, 1.0f);
Mat4& worldMatrix = mParentNode->getLocalToWorldTransform();
Vec4 o(0.0f, 0.0f, 0.0f, 1.0f);
Vec4 d(vx, vy, 1.0f, 0.0f);
o = o * worldMatrix;
d = d * worldMatrix;
Basically you give the function the screen coordinates, a ray origin vector, a ray direction vector and the space you want the ray in (world or local). Since this is the camera we are talking about, the local space is effectively the view space.
So, first you get the ray in view space. For that we need the projection matrix of the camera, the screen coordinates and the screen size. If you are cool with with view space you just fill in the vectors and you are good to go.
If you want the ray in world space you need to transform it with the world matrix, aka "local-to-world-matrix", aka "view-to-world-matrix". If you happen to have the "world-to-view-matrix" instead, like most camera implementations, the one you want is just the inverse of that. You fill out two 4D vectors (the origin with W=1, the direction with W=0) and transform them by the matrix. After that you should probably normalize the direction.
Voila, you now have the "start-point" of the ray (which is still just the camera position btw) and a unit-length direction vector of the ray, both in world space. Multiply the direction with whatever huge number you want to get a long ray cast.
Does this help?
Edited by GuyWithBeard, 03 December 2012 - 10:52 AM.