I had a little more success writing a VLC Media Player plugin, which basically works by feeding the request into my codec system and using this to decode the video.
then, later got the VfW driver to work, turns out the issue was mostly that some of the logic was broke, and this was revealed after throwing together a mechanism to print messages in a log file.
also created a new codec I had called BTIC1D in an attempt to have higher image quality than RPZA / BTIC1C (mostly for storing intermediate video data, in contrast to BTIC1C which is "nearly ideal" for decoding to DXT1 or DXT5).
it is a Block-VQ codec supporting Alpha, Layers, and HDR, using YCgCo and 4:2:0 chroma sub-sampling, and YUV bit-depths of 11:9:9 for LDR and 9:8:8 for HDR (with a 4-bit exponent), and also using Deflate for entropy-coding.
bit-rates are currently fairly high (30Mbps for 480p30 at 80% quality), and decode speeds "could be better" (decode is approx 120 Mpix/sec at present, for a single-threaded plain-C decoder), but it works (*).
most of the decode time goes into things like converting blocks to RGBA and also dealing with Deflate.
*: during design I was expecting it to break 200 Mpix/s or better, so the performance is a little disappointing, but alas...
it is fast enough to probably do 1080p60 or 2160i30, but the files would be huge.
probably better compression at similar decode speeds (and probably faster encode) could be possible if using WHT or similar instead (as-in the BTIC2 family).
now having usable codec drivers opens up more options, mostly as fully custom codecs can now be viewed in normal video-players (if albeit lacking many features). otherwise, it would probably require a specialized video player or similar to see videos using the extended features. but, in the common case, it works...
but, I seem to be wasting a bit too much time on all this, not really getting a whole lot else done...
ADD, some info:
and its predecessor:
no code currently available, might do so eventually if anyone is interested.
Revived a past effort (and finished implementing it), mostly to compare things (see how BTIC1D fares against a more conventional design).
interestingly, the bitrate difference between BTIC1D and BTIC2C isn't particularly drastic, however, the decoding speed of BTIC2C is currently a fair bit lower (once again, mostly bogging down in the YCgCo->RGBA conversion, *1, but currently only pulling off about 35-40 Mpix/sec with an optimized build), though the encode speed is a bit faster.
temporarily dropped a few things to get it implemented more quickly though.
its implementation was mostly done by copy-pasting parts from several of my other codecs: JPEG, BTIC1D, a few parts from BTIC2B (never fully implemented). does bring up idle thoughts of if it could make sense to hybridize VQ and WHT+Huffman (say, allowing both within a shared bitstream).
*1: less certain is the reason for the big speed difference here, though cache patterns could be a big factor.
likely, things may be changed to perform color conversion on a block-by-block basis rather than via a big monolithic conversion pass.