I'm having issues wrapping my head around how tesselation and collision can work. My understanding of terrain tesselation is based on the nvidia demo/whitepaper: https://developer.nvidia.com/sites/default/files/akamai/gamedev/files/sdk/11/TerrainTessellation_WhitePaper.pdf

For example, an animated character moving across a (heavily) tesselated terrain such as in the demo. Given that tesselation is a shader technique, how can the two ever work together?

EDIT: Also if the tesselation is the result of shaders, how on earth do you apply lighting and shadows to it?

Typically these 2 things aren't related - the terrain can be tesselated independently from collision handling. The tesselation of a terrain is usually based on the point-of-view and the distance of terrain chunks, while the collision testing needs to work on a data which isn't view point dependent.

If you'd perform collision tests on the tesselated terrain then funny things could happen - consider that your camera is moving and the tesselation changes radically: objects that were on terrain would be in air or maybe inside the terrain. This could make objects fall or maybe get boosted in astronomic speeds (because they are suddenly inside an object). Believe me, I tried this years ago.

Keep the physics separated from the graphics and your life will be easier. Make a separated physical model for the terrain. This is logical since you could be running the game without any graphic output (ie. dedicated server).

Cheers!

As Kuana said, keep a fully detailed mesh as your physics mesh (heightmap, whatever) which is separate from your visual mesh and have it loaded as long as anything can interact with it.

As for your lighting and shadows, as long as your lighting and shadows uses the depth buffer/lighting buffer or what-have-you, then it should be fine.

EDIT: Also if the tesselation is the result of shaders, how on earth do you apply lighting and shadows to it?

For shadow mapping this is ok as long as you use the same tesselated mesh for the shadow map rendering - there will be visual artifacts if the mesh geometry rendered in the shadow map isn't the same as the mesh geometry used in the lighting calculations.

However, sometimes you may need to optimize the shadow rendering and use maybe a less detailed mesh - you'll need to study what kind of geometrical changes cause less rendering artifacts.

Cheers!

In some cases you might not be able to get the same level of detail on your physics terrain and tessellated terrain mesh (for reasons xyz). On one project I worked on some clever guys at NV came up with an approach which would evaluate height of the ground on GPU at the positions of the various character bones, and then offset the bones if needed, to make sure that there was no issues with the characters sinking into the ground. Overall it worked really well. (technically not correct since you would have cases were one leg would be shorter than the other for example, but not very noticeable in game)