The samplers in the hardware generally have a fixed list of formats they can interpret from the GPU cache and/or memory. This list corresponds to the texture formats supported by a given hardware piece.

4 channels is a practical upper limit for the fixed formats, since fetching RGBA colors is overwhelmingly the most common usage scenario for the samplers.

SM4 and up (or D3D10 and up) support integer instructions, so you could easily do custom packing/unpacking of 128-bit texels in pixel shader to introduce your own pixel format. You'd need to do your own filtering if you require it, as the filtering of GPU-supported formats is the responsibility of the samplers.