How do the modders,who introduce new shader effects,know what input layout they must be expecting in the vertex shader and what constant buffers are being set,etc..are they reverse engineering the shader files?
For example:Freelancer
This isn't really a complete answer but there are lots of GPU debugging tools (pix, etc..) that'll catch the DirectX calls with their arguments, intercepting the raw shader code (if it wasn't already available) and everything relating to constant buffer layout, vertex layout, etc... so I would assume they reverse-engineer and study it this way.
If the mod can be fully self-contained within the shader, it's a lot easier, but otherwise they could be injecting new additional code in the executable (kind of like Fraps does to capture DirectX frames) to enable additional behaviour and special effects. This is also possible if the source can be decompiled and made sense of to some extent (like Minecraft, the code is obfuscated but you can still kind of understand where you need to add your modding code, and then recompile the jar file).
There was a "hack" to add bloom to Half-Life 1 engine games (pre-steam) a while back.
As I recall, it provided it's own OpenGL32.dll which when placed next to the half-life .exe would take precedent over the normal one (provided by WIndows or driver).
It would then simply pass through the calls to the original one, but hook the parts needed to add bloom.
This probably isn't a very conventional way (though a lot of multi-player cheats are made this way) but I thought it was quite clever.
