The core of a material 'definition' is what I'm calling a grain pattern. Basically, it's a box, in which you put a number of different 'grains,' and arrange them in some way. Grains can take on a pretty wide range of shapes - spheres, cubes, capsules, cuboids, noisy variants of those primitives, and compound variants of those primitives - and have things like mass associated with them.
So you've created a basic grain - let's say it's capsule-shaped, with a slightly wider radius in the middle than at the edges. You add it to the grain pattern (and because it's the only grain type present, it is given 100% weighting). You set up the density of the material - a balsa wood type thing would have a lower density than hardwood, for example - which determines how many grains are added to the box. You give them a random distribution of positions, but don't go quite so crazy with the orientations - you make them aligned along the Z axis, with a small variance.
Then you define grain-grain 'collisions.' When two of these capsules overlap, what should happen? Should they repel each other so that they stop overlapping? Should they coalesce? Should they 'attach,' creating a join at the contact point which requires a certain threshold force to break? You set up a collision for this grain with itself, and set it to 'attach' mode. And suddenly, your capsules are wood fibers, aligned and connecting.
Once you've got that, you can use the box to fill a solid volume in your level. When you set off an explosion next to the volume, it can be subdivided into little boxes, and as the effects of the explosion reach the wall, the touched boxes are 'converted' to grain form and the physics computed from there.
When you're done, surface-finding techniques can be used to map polygons over the exposed grains, perhaps even using normal maps to preserve some detail while keeping the polycount low.
It's not so easy to see how it would effect a wood type material... but imagine something like slate. Flat, rectangular grains, arranged in an offset grid pattern (so that the centers of grains on one layer are aligned with the off-centers of neighbouring layers). Make yourself a wall out of that and apply a force to it... you'll leave a hole which, when viewed from the other side, should show a radial pattern with each layer seperating at a different point.
And the great thing is, these materials only ever need to be created once, really. Once you've got "slate," you can apply it to any volume, regardless of the colour of the slate texture you slap on the top.