You could go up to 32 bits exactly and transfer them as D3DDECLTYPE_UBYTE4 but this seriously does smell of premature optimization. Try just transferring the full normal texcoords as floats anyway - you'll probably find that you're not really bound at this stage of the pipeline at all and that any attempt to reduce the data size doesn't make a blind bit of difference.

I assume the context to be PC games. It might look different when you're running on a console.

On DX9 level hardware, UBYTE4 is indeed the best you can get. You can do some bit swizzling in the shader to combine 2x 4bit into one of these 4 bytes, but you can't specify anything less than 32bit. Thus this compression scheme is only useful if you can make other use of the remaining 3 bytes.

A warning from experience: it won't help you much.

- It can save a lot of GPU memory, but this is only useful if you're handling literally millions of instances. If you're doing just a few tens of thousands of instances, I wouldn't waste my time on it.

- It can save quite some memory bandwith, but I never found this to be the limiting factor. The best I got from compressing my instancing vertex structure from 56 bytes down to 20 bytes was 30% performance gain.

- It can save you transfer bandwith in case you're updating the instancing data every frame. In that case I'd say it's worth the hassle, but I'd wager you have other problems then.

- It won't help you in any other case.

A few months ago I wrote a voxel renderer that splatted millions of textured quads. I first tried to use instancing, but it was slow as hell. 4 million quads resulted in ~15fps on my Geforce GTX 460. When trying to find the bottleneck I noticed that all counters of NVPerfHUD together only accounted for 30% of the frame time, and 70% went to "somewhere". Then I tried Visual NSight, which was buggy as fuck but at least could show me the real cause: the Input Assembler. Then I removed all instancing and stored four unique vertex structures per quad, with a total 80 bytes per quad, and I got to 55fps. For the very same geometry, and four times the GPU memory bandwith. Something is happening on those modern cards that I can't explain. An ATI GPU showed the same behaviour.

It's also the case that for huge numbers of objects you're more likely to bottleneck on fillrate (and potentially overdraw, depending on the type of object) than on vertices. This can be observed with particle systems and would be true of foliage too.

Im just a bit disappointed, that no solution exists for transferring custom-sized data pieces, as it can be a problem if transferring millions of data packets.

Well, graphics cards can't deal too well with data smaller than 32 bits, especially unaligned data, so what you'd save in memory bandwidth, you would lose in processing efficiency. Really, you should get everything working first, and then benchmark (if you still notice a slowdown once everything is in place).