This isn't really a complete answer but there are lots of GPU debugging tools (pix, etc..) that'll catch the DirectX calls with their arguments, intercepting the raw shader code (if it wasn't already available) and everything relating to constant buffer layout, vertex layout, etc... so I would assume they reverse-engineer and study it this way.
If the mod can be fully self-contained within the shader, it's a lot easier, but otherwise they could be injecting new additional code in the executable (kind of like Fraps does to capture DirectX frames) to enable additional behaviour and special effects. This is also possible if the source can be decompiled and made sense of to some extent (like Minecraft, the code is obfuscated but you can still kind of understand where you need to add your modding code, and then recompile the jar file).
Edited by Bacterius, 15 January 2013 - 02:04 AM.
The slowsort algorithm is a perfect illustration of the multiply and surrender paradigm, which is perhaps the single most important paradigm in the development of reluctant algorithms. The basic multiply and surrender strategy consists in replacing the problem at hand by two or more subproblems, each slightly simpler than the original, and continue multiplying subproblems and subsubproblems recursively in this fashion as long as possible. At some point the subproblems will all become so simple that their solution can no longer be postponed, and we will have to surrender. Experience shows that, in most cases, by the time this point is reached the total work will be substantially higher than what could have been wasted by a more direct approach.
- Pessimal Algorithms and Simplexity Analysis