Thanks @MJP for the incredibly informative reply! that's a tonne of useful information.
This was the approach taken by Bungie for Halo 3, and was discussed in their SIGGRAPH presentation as well as their course notes.
More recently, we presented some info at SIGGRAPH about our lightmap baking for The Order.
Ideally i would love to do something like either of these implementations but it isn't currently feasible for this particular project, its also slightly beyond my current capability.
You could also try computing a "dominant direction" from your lightmap info and treating that as a directional light, but these approaches can sometimes suffer from interpolation issues where the dominant direction doesn't match up from texel to another.
I took your advice and implemented a hacky phong specular with a dominant direction, It produces much better results than my previous implementation! here is my code, please feel free to pick it apart and point out any mistakes or suggest any improvements, Im still very much a novice. Though LOOKS right ingame........
#if( PHONG && !PS4)
{
float3 vEyeDir = normalize( g_EyePos - worldPos );
half SpecExponentMap = tex2D(SpecExponentSampler, coords.xy).r;
float lumer1 = ( Luminance( lightmapColor1.rgb ) );
float lumer2 = ( Luminance( lightmapColor2.rgb ) );
float lumer3 = ( Luminance( lightmapColor3.rgb ) );
float3 vLightDomDir = normalize (lumer1 * bumpBasis[0] + lumer2 * bumpBasis[1] + lumer3 * bumpBasis[2]);
float3 vHalfAngle = normalize( vEyeDir.xyz + vLightDomDir.xyz );
float flNDotH = saturate( dot( worldSpaceNormal.xyz, vHalfAngle.xyz ) );
SpecPhong = pow( flNDotH, SpecExponentMap ); // Raise to specular exponent
SpecPhong *= pow( saturate( dot( worldSpaceNormal, vLightDomDir ) ), 0.5 ); // Mask with N.L raised to a power
float3 d_irradiance = 0;
d_irradiance.x = dot(bumpBasis[0], vLightDomDir);
d_irradiance.y = dot(bumpBasis[1], vLightDomDir);
d_irradiance.z = dot(bumpBasis[2], vLightDomDir);
d_irradiance*= d_irradiance;
d_irradiance = d_irradiance.x * lightmapColor1.rgb + d_irradiance.y * lightmapColor2.rgb + d_irradiance.z * lightmapColor3.rgb;
SpecPhong *= d_irradiance; // color the specular from the d_irradiance
float3 vSpecularTint =1; // leaving this in for now, models have the ability to tint specular
float fSpecMask = 1.0f;
fSpecMask = baseColor.a; // hardcode for now
//fSpecMask = lerp( normalTexel.a, baseColor.a, g_fBaseMapAlphaPhongMask );
//fSpecMask = lerp( fSpecMask, baseLum, g_fBaseLumPhongMask );
SpecPhong *= fSpecMask * vSpecularTint;
float3 g_FresnelRanges = float3 ( 0.3, 9.0, 28 ); // test vaules to match models material // need to hook up to material
float fFresnelRanges;
fFresnelRanges = Fresnelcustom( worldSpaceNormal, vEyeDir, g_FresnelRanges );
SpecPhong *= fFresnelRanges;
//to do look at this again.
#if ( ( CASCADED_SHADOW_MAPPING ) && ( CASCADE_SIZE > 0 ) )
{
// Apply csm shadows
//SpecPhong.rgb *= shadowscalarpassthrough;
}
#endif
//note note - not sure why we would want rim lighting on brushwork so left it out. but could bring back for cheap;
}
Thanks!