in my case, things are organized recursively roughly by category, for example:
lighting / shadows / ...
rendering voxel terrain
3d character models
texture and shader / material management
game logic / server-side logic
entities / AI
client-side / input-handling / UI
world loading/management stuff
voxel terrain stuff
console related stuff (processing commands, cvars, ...)
stuff for image-processing and video data / codecs / ...
stuff for audio-processing (sound effects, mixing, ...)
assorted math stuff
vector and matrix functions
primitive types (AABB / OBB / ... collision-checking, ...)
things like polygon clipping and similar (clip triangle by plane, ...).
memory manager / GC
dynamic type and object facilities
script VM VM / interpreter
within this, things may be broken up into specific topics, for example, within image and video stuff, there might be files or groups of files dedicated to specific image formats or codecs, as well as specific data representations (stuff related to DXT5 or BC7, for example).
however, there may be some amount of overlap, for example, several formats may share things like the DCT / IDCT transforms, as well as a lot of special case-code, such as YCbCr 4:2:0 (or YCbCrA 4:2:0:4) to DXT5 or BC7 conversion logic.
but, this may be kept separate from the renderer, which may just be like "give me those video frames as BC7", and may not really care how the video-decoder stuff does so (or all the hair that goes into real-time BC7 transcoding, or its quality limitations, ...). likewise, the video-codec logic may not need to care about the specifics of how the texture-manager streams stuff to the GPU (PBOs vs glCompressedTexImage2D calls vs ...), its main goal just being to fill the buffers quickly. likewise, the video-codec may have some stuff otherwise irrelevant to the renderer, such as logic to allow it to be callable from VfW or DShow or plug into VLC, ...
beyond this, generally things are organized by similarity.
whatever -> DXT5 block-level conversion
variants based on input format (RGBA, BGRA, YUV420, ...)
variants based on secondary requirements
slower but higher quality (such as for texture loading)
faster but lower quality (such as for video playback)
DXT5 -> whatever (have DXT5, need RGBA, BGRA, or YUV420 or YUV422, for example)
image-level functions (convert a whole image, flip image, or convert+flip, ...)
like above, but all over again for BC7...
likewise, but for floating-point textures...
so, one might end up with a source file dedicated to various "whatever to DXT5" conversion functions, another for "whatever to BC7 mode 5", ...
elsewhere in the engine, it may be a different set of files dedicated to each specific enemy type (enemy-specific behaviors and animations, ...).
or, in an interpreter, files dedicated to various arithmetic operators or array load/store operations or field get/set operators, ... (because we might need versions of the operators specific to various operand types, ...).
a lot of this is not necessarily because anything is particularly difficult, but mostly because of performance constraints and special cases and variations, which may in turn sometimes result in a lot of hair and bulk.