Yes, the camera position, what happens if you stand still and rotate the camera only ? In this case the "terrain texture" should not change, if it change, your InverseView matrix is most likely broken.
An other test, if you only move the camera along the lookat,right axis (no rotation) the color pattern should stay the same (like a projected texture pointing along the up-vector centered at the camera).