Think of the displacement map as a 2D signal that's sampled by the projected grid. All sampling techniques, projected grid included, have to take into account aliasing. In order to fight it, follow these steps:
- Build, either at runtime or offline, a full mipmap chain of the displacement map. Experiment with different downsampling filters, like box filters, simple point filters (which can be OK for displacement maps) or more advanced ones.
-In the shader, calculate the derivatives of the displacement map texture coordinates with respect to the projected grid x,y coordinates. You have some options here: forward differencing, ddx and ddy (not applicable in my vertex shaders) or analitically. I'd suggest the last option. Please ask me for formulas if you get stuck.
-Fetch the displacement map by passing the derivatives (tex2Dgrad in DirectX).
-For procedural waves (sin, cos etc), exploit the derivatives to filter them out analitically.
You might want to add a parameter (in the [0,1] range) that multiplies the derivatives in order to let artists control the tradeoff aliasing vs detail in the scene.
Also, as described in the original projected grid paper, the projective camera does not necessarily match the main camera. By changing some of its parameters you can bias the sampling and, for example, add more detail in the distance.
Last, wobbling is inevitable if the projected grid is too coarse. What is the resolution of the screen and the projected grid in your application ? I suggest a ratio of less than 4 before the two.