I also like the idea of offsets relative to the position of the offset, although I wonder what would happen if you had a negative offset;
Theoretically it wouldn't matter and would still work.
...
I assume the deleter would look something like this:

Yep, that's how you'd clean up your allocation.

Yeah, negative offsets "just work", seeing I used a signed integer type. Also, seeing that an offset of 0 doesn't go anywhere (it takes you to the offset value itself, which isn't useful) I can use 0 as a "null pointer" value.
On the C# side (when generating data files), I've written an extension class for BinaryWriter (I've also added functions like Write32, Write16, etc, to make my file-writing code more explicit and obvious), that I use like this:

writer.Write32(buffers.Count());//num stream formats
long tocStreamFormats = writer.WriteTemp32();//offset
... later ...
writer.OverwriteTemp32(tocStreamFormats, writer.RelativeOffset(tocStreamFormats));//overwrite placeholder with relative offset
for (int i = 0; i != buffers.Count(); ++i)
{ ...

Do not put API-dependant values in your blobs (such as DXGI_FORMAT). Well, you can, but it's going to look wrong in the long run

There's nothing wrong with doing this, as long as you only support one API per platform and are ok with shipping different data archives per platform, and you've got a build-system that can automatically generate your files.
e.g. on every console game I've worked on, the data has had to be compiled once for PC, PS3 and 360 (with some files being shared across platforms, but many being different per build). If we have to modify some detail per platform, you just increment the version number on that model exporter, and the build system automatically recompiles all the models.

The upside to using API-specific structures in your files is that your loading code can be extremely simple (with no parsing step required), but yes, the downside for a cross-platform game is that you'd either have to ship different model files, or your model file would have to include structures for both D3D/GL/etc.

I am worried, though, about how I should deal with getting the VertexLayout to the renderer. I'm not sure if I should store it per mesh, load it similarly to how I'm loading textures, or just have each material specify.

I treat vertex formats as an external dependency, like textures, materials, etc...
Also BTW, I don't store texture-handles in the model file, only material-handles. The material can specify the shader, uniforms, textures, etc...

The complicated part with Vertex Layouts is that they're a bridge between both models and shaders. I deal with this by having a "streams.lua" file (or multiple of them), which describes model-side vertex layouts and shader-side vertex layouts, and which model layouts can be used with which shader layouts.
When writing my shaders, I also have to specifically name the shader-side vertex layout that I want to use from this file.
e.g. For a shader that requires position/normal and 2 tex-coords, and a shadow-shader that only requires position, I could write in this file:

StreamFormat("testStream",
{
    { -- stream 0
        { Float, 3, Position },
    },
    { -- stream 1
        { Float, 3, Normal,   0 },
        { Float, 2, TexCoord, 0 },
        { Float, 2, TexCoord, 1, "Projection1" },
    },
})
VertexFormat("testVertex",
{
    { "position", float3, Position },
    { "normal",   float3, Normal },
    { "texcoord", float2, TexCoord, 0 },
    { "texcoord", float2, TexCoord, 1 },
})
VertexFormat("testShadowVertex",
{
    { "position", float3, Position },
})
InputLayout( "testStream", "testVertex" )
InputLayout( "testStream", "testShadowVertex" )

When compiling a model, I look at the which vertex-formats the current mesh needs to be compatible with (based on which shaders the artist have applied to the mesh), and then find the set of stream-formats that are compatible with all of those vertex formats, and then compile the model using the stream-format from that set that best matches the data that's present in the DAE.

Above, there's only 1 stream format, which defines a position stream and a normal/tex0/tex1 stream. These could be stored in different vertex buffers, or the same vertex buffer using offsets. If the DAE didn't include some of those attributes, then the model would produce an error during the data-build process.

The graph of build-rules to create a model (.geo) file look something like this for me:

Source     C# Intermediate        Game Data

      +- temp/mat
      |
.DAE -+
      |
      +- temp/geo ------------+
                              |
                              +- data/geo
                              |
              +- temp/streams +
              |               |
.streams.lua -+               |
                              |
       +- temp/vlayout -------+
       |
.hlsl -+- temp/hlsl
       |
       +- temp/technique