This is an idea I've thought about on and off for a while but never tried out. Essentially my thinking was that rather than formulating an animation as a description how the positions of vertices change, what if the vertex positions are constant and the animation how the shape of space changes.

Thinking of it like how a magnet distorts field lines is the most visual representation I can think of.

It obviously isn't a generally useful approach, much more suited to organic animation, but I'd be interested what people think. Or maybe this is already used and has some names I can look up online.

I think I saw some of these spatial deformations you mention in 3DS Max, under "space warp" (iirc). Not sure if it was for animation or not, but basically it warped space inside a given volume according to some displacement function. Well, actually, it just warped the vertices and objects within - the space itself remained Euclidean.

I'm not sure how applicable this is as a general-purpose solution, though. Generally, the more information you want to encode in your animation spatial transformation, the more complex the transformation will be (information theory) so unless you have really simple or self-similar animations (like fractals) I'm afraid the spatial transformation would end up being approximately as complex to evaluate as just displacing every vertex the traditional way (if that makes any sense). But it could have its uses, you mention organic animation, that could be an approach worth pursuing.

You could kind of say that this is how skinned/skeletal animation works. You don't save animation data for each vertex, instead you've got animation data for a collection of "bones". Each bone's data is basically an independent transformation of 3D space.

Each vertex then has a collection of bone indices and weights, and uses them to decide which "bone spaces" will affect it, and by how much. e.g. an elbow vertex might be weighted 50% to the forearm space, and 50% to the upper-arm space, resulting in a soft/organic transition at the elbow-joint when the forearm and upper-arm spaces are transformed differently (hence the name -- skinned animation).

This can also be used to create hard edges/transitions - if every vertex is only associated with a single bone (with a weight of 100%) - so it is a very general technique.