cant help much - irregular logic code is just irregular...
Possibly use a C macro for the stuffing of the ctb.. to reduce it to one line and another for ctb ..
yeah, it is possible...
a lot of code is written fairly quickly and so tends to develop repeating patterns and some verbosity.
some things are left expanded out in case I see a way to make them faster, where putting things in macros can sometimes hide potential optimizations, or sometimes make some issues for looking at and stepping through stuff in the debugger.
though, for example, "*_memcpy8()" is actually a macro in this case (chooses a target-specific mechanism to copy 8 bytes).
I looked at it for a while looking for processing commonality and that was the closest/easiest/clear thing just to shrink the long code length down a bit.
Those 2 macros I mention seemed to be simple patterns reused more than a few times and they can just stuff 4 given parameters (the assignment equations) into the 0,1,2,3 and 4,5,6,7 offsets of the of 4 consecutive ctb array slots - it coukld shrink the function length some significant amount (I usually use ALL uppercase for macro names to make them stand out clearly, but thats just my style) I'd make the macro name short so you wont have the params running off the right side (I also indent only 3 blanks so I can have quite long 1 line macro param sets fit)
The other thing I notice (I think Ive done something like this code at one time) is to document the mode data formats (words bytes bits) and command codes Right in with the Code at teh spots that carry out each variation. (also maybe name the encoding variations even to make it more mnemonic).
Also Im not sure of the effects on the flow clarity but those 'modes' ---can they be broken out into a seperate function each mode to get rid of some of the nesting?
Again, its quite irregular and very few common processing variations to not just spell most of it out linear like that (I dont really see that going away)