When you move the camera, the shadow seems to shift. Therefor I guess, that your back transform works up to the projection part (viewport.
Well, check your back transformation pipeline (view->world->light space->light frustum), something like this
C = camera transform (in world space) L = shadow camera transform (in world space) Assumption: you got a pixel position pos_v in view space reconstructed from the framebuffer // 1. View to world space pos_w = C * pos_v // NOT the inverse camera transform // 2. world to light space pos_l = inverse(L) * pow_w // 3. Project on shadow frustrum pos_final = L_proj * pos_l // 4. get shadow texel shadow_texel = 2dShadow( shadow_map, pos_final)