If you're looking at light hitting the surface as a wave that's encountering a change in IOR, then refraction is correct. Fresnel's laws say that some of the wave is reflected and some is refracted -- those are the two options.
I wouldn't really call it "refracted", it's more a notion of incoherent scattering. In other words, "specular" means "not scattered", whereas refraction *is* a form of local specular reflection.
For the reflected portion of the light wave, the angle of incidence and angle of reflection are always equal, which makes it a "specular reflection". This light can't be discoloured by the surface due to absorption, because it never enters the surface.
The rest of the wave (the part that isn't reflected) only has one other option - to be refracted (i.e. transmitted through the IOR boundary). The angle of transmission is bent slightly (meaning it's not "specular"), though this is irrelevant to us when rendering solids.
Once this light has been refracted/transmitted into the opaque object, then scattering/diffusion/absorption can occur. It bounces around inside the object, mostly being absorbed but with parts of it managing to make it back to the surface again to be re-emitted in some random direction.
So it's also valid to say that diffuse lighting equations account for scattered/diffuse light, but scattered/diffused light is refracted light (the only way that light can get into the diffuser, assuming it's not emissive, is to be transmitted/refracted through it's surface).
The Lambert diffuse model ignores specular reflection (all light is refracted/transmitted into the surface), it's then assumed to be perfectly diffused (every possible exitance angle has equal probability), and if you use a white "diffuse texture", then it's assumed that no absorption occurs and that all of this refracted light eventually makes it back out.
Likewise, the Phong specular model ignores diffusion/refraction (all light is reflected off the surface without entering it at all).
To account for all of the light that hits the surface - the reflected portion and the refracted/transmitted portion, you need to combine both these BRDF's together (or use another BRDF that does account for both "diffuse" and "specular" light).
During the diffusion process inside the surface, it's also usually assumed that the polarisation of this light also becomes randomly distributed (it becomes "unpolarized" light, or has mixed polarisation). Whereas on the other hand, reflected light doesn't really interact with the surface, so it's assume that it's polarization is essentially unchanged. Most renderers ignore polarization though, and simply assume that all light is an equal mix of all possible polarizations.