Honestly in that case I'd have just changed the header and called it a new format =P It's based in JPEG, but the changes you mention sound serious enough to be considered its own new thing, really.
interestingly, JPEG doesn't really have a "file header" in the traditional sense, but is instead a sequence of escape-coded markers (IOW: the byte value 0xFF is magic, with the following byte as a marker, ...). this makes it a little "interesting" in a few ways...
typically for these files, I am using a "BTJ" extension, and generally calling the format "BGBTech-JPEG", and has enough options that it can actually generate a range of sub-formats, ranging from those resembling and compatible with conventional JPEG, to the significantly altered (and incompatible) NBCES variant, which is pretty much "beaten beyond recognition" (most of the internal headers and tables are different, the block-coding is different, it incorporates parts from PNG and Deflate, ...).
a related format is "BTIC" ("BGBTech Image Codec"), which was partly based on NBCES, but is internally based more around secondary compression of DXTn images (and high speed decompression), so is really its own format (during implementation I decided against trying to shoe-horn it in with BTJ and NBCES). it still retains a few elements from JPEG, mostly in terms of reusing a few of the markers (SOI/EOI, APPn, JPGn -> SYSn, COM, ...), and some of the high-level structure from BTJ and NBCES, but is otherwise its own format (it uses its own internal headers and similar, and apart from these markers, there is little else in common).
the main thing here is that the original JPEG markers were used to build a TLV format for BTJ, which is then expanded on and used as part of the general structure for building the other formats (*3).
BTJ basically just used them to add layers and a few additional informational headers.
BTJ-NBCES basically used them to implement new headers (and mostly discards the original JPEG headers, *1);
BTIC, basically similar, but has no headers or other structures in common with NBCES (*2).
*1: the NBCES headers are basically stored together as a big glob of bits, using a fixed-width bit-field to identify various headers, and typically with the headers themselves entropy-coded (vaguely similar to the strategy used in Deflate). some minor tweaks are also made to the FF escape-coding to make it more space efficient (owing to the loss of required JPEG compatibility, and nested packaging with each level adding an FF -> FF 00 conversion gets expensive, making a more efficient encoding scheme desirable...).
*2: most of its structures are byte-based, and currently no entropy coding is used, mostly to allow very fast decoding at the expense of compression ratio.
*3: sort of like RIFF, which was used as the base for AVI/WAV/RMI/... but, this TLV format offers a few more interesting possibilities: optional longer marker names (FOURCC/EIGHTCC/ASCIZ), distinguishes optional vs must-understand, allows for open-ended markers (paired start/end markers), as well as for resynchronization (re-aligning with an in-progress data-stream, mostly as the markers can't validly appear in the payload data). multiplexing could also be possible (likely via an additional wrapping layer).
currently, I am using the current FOURCC codes (for AVIs):
"MJPG": Motion JPEG and sometimes very-conservative subsets of BTJ;
"MBTJ": Motion BTJ;
"MBTC": Motion BTIC.
I originally considered using a similar packaging for "BTAC" (BGBTech Audio Codec), but was lazy and just used a fixed file-header instead (sort of like BMP).
decided to leave out stuff mostly about possible future directions and TLV escape tagging (BTJ vs my network protocol vs MPEG and friends...).
short answer: my network protocol (SBXE) and MPEG use longer (but different) escape-sequences, but there are various tradeoffs here.