in my case, things are divided up more like this:
there is currently a library which manages low-level image loading/saving stuff, and also things like DXTn encoding, ...
this was originally part of the renderer, but ended up being migrated into its own library, mostly because the ability to deal with a lot of this also turned out to be relevant to batch tools, and because the logic for one of my formats (my customized and extended JPEG variant) started getting a bit bulky.
another library, namely the "low-level renderer" mostly manages loading textures from disk into OpenGL textures (among other things, like dealing with things like fonts and similar, ...).
then there is the "high level renderer", which deals with materials ("shaders") as well as real shaders, 3D models, and world rendering.
internally, they are often called "shaders" partly as the system was originally partly influenced by the one in Quake3 (though more half-assed).
calling them "materials" makes more sense though, and creates less confusion with the use of vertex/fragment shaders, which the system also manages.
materials are kind of ugly, as partly they seem to belong more in the low-level renderer than in the high-level renderer.
also, the need to deal with them partly over on the server-end has led to a partially-redundant version of the system existing over in "common", which has also ended up with redundant copies of some of the 3D model loaders (as sometimes this is also relevant to the server-end).
this is partly annoying as currently the 2 versions of the material system use different material ID numbers, and it is necessary to call a function to map from one to another prior to binding the active material (lame...).
worse still, the material system has some hacking into dealing with parts of the custom-JPEG variants' layer system (in ways which expose format internals), and with making the whole video-texture thing work, which is basically an abstraction failure.
this would imply a better architecture being:
dedicated (renderer-independent) image library for low-level graphics stuff;
* and probably separating layer management from the specific codecs, and providing a clean API for managing multi-layer images, and also providing for I/P Frame images (it will also need to deal with video, and possibly animation *1). as-is, a lot of this is tied to the specific codecs. note: these are not standard codecs (*2).
common, which manages low-level materials and basic 3D model loading, ...
* as well as all the other stuff it does (console, cvars, voxel terrain, ...).
low-lever renderer, which manages both textures, and probably also rendering materials, while still keeping textures and materials separate.
* sometimes one needs raw textures and not a full material.
high-level renderer, which deals with scene rendering, and provides the drawing logic for various kinds of models, ...
probably with some separation between the specific model file-formats and how they are drawn (as-is, there is some fair bit of overlap, better would probably be if a lot of format-specific stuff could be moved out of the renderer, say allowing the renderer to more concern itself with efficiently culling and drawing raw triangle meshes).
the uncertainty then is how to best separate game-related rendering concerns (world-structure, static vs skeletal models, ...) from the more general rendering process (major rendering passes, culling, drawing meshes, ...). as-is, all this is a bit of an ugly and tangled mess.
*1: basically, the whole issue of using real-time CPU-side layer composition, vs using multiple drawing passes on the GPU, vs rendering down the images offline and leaving them mostly as single-layer video frames (note: may still include separate layers for normal/bump/specular/luminance/... maps, which would be rendered down separately). some of this mostly came up during the initial (still early) development of a tool to handle 2D animation (vs the current strategy of doing most of this stuff manually frame-at-a-time in Paint.NET), where generally doing all this offline is currently the only really viable option.
*2: currently, I am still mostly still using my original expanded format (BTJ / "BGBTech JPEG"), mostly as my considered replacement "BTIC1" wasn't very good (speed was good, but size/quality was pretty bad, and the design was problematic, it was based on additional compression of DXTn). another codec was designed and partly implemented which I have called "BTIC2B", which is designed mostly to hopefully outperform BTJ regarding decode speed, but is otherwise (vaguely) "similar" to BTJ. some format tweaks were made, but I am not really sure it will actually be significantly faster.
more so, there are "bigger fish" as far as the profiler goes. ironically, my sound-mixing currently uses more time in the profiler than currently is used by video decoding (mostly dynamic reverb). as well as most of the time going into the renderer and voxel code, ... (stuff for culling, skeletal animation, checking for block-updates, ...).