From what I understand, Zbrush is not running in D3D or OpenGL. So the limitations of those technologies doesn't really apply. It is polygons like Hodgman said, but not traditional OpenGL or D3D mesh polygons. There are a bunch of features that bend the rules in Zbrush.
Edit: I can imagine that the insert mesh is done by grabbing the border edge vertices, doing some math to find out which ones are closest between the two objects and drawing lines between the meshes. I don't know what math they use to build the surface triangles between the objects but it is probably some of the math from Dynamesh and/or Decimation Master.