So here is the idea: each time a client wants to connect to the server, it has to download & replace the client EXE from it, pretty much like a patch.. but every time it connects. The trick is, the downloaded EXE is different each time and incompatible with its previous version, including the network protocol, function adresses, etc..
When the server detects an incoming connection, it picks up one EXE from a pool (which can be arbitrarily as large as you want) and sends it to the client. All the EXEs use a different network protocol and once the download is complete, the server is set to only understand the specific protocol corresponding to the client. If there's still no reconnection after 10 seconds, for example, the process has to restart from the beginning.
Some drawbacks i can see:
- additional bandwidth (if the EXE is 1 MB, that's 1 MB of bandwidth consumed each time a client connects).
- the server has to understand all the possible protocols at the same time
- the number of client EXEs has to be pretty large
- each time the client EXE is patched/improved (by the developpers), many versions have to be recompiled.
- not 56K friendly. ADSL is not problem because downloading 1 MB is only a couple seconds.. but 56K modem users will have to be patient.
The advantages:
- if a hacker modifies his local EXE, the server will send him a new one when he connects. If the hacker ignores the newly downloaded EXE and tries to connect to the server with his hacked version, the protocols will be incompatible and the connection refused.
- if the hacker downloads a new EXE, wants to hack it and connect to the server with this hacked version, he has to do it in less than 10 seconds.
- a hacked version cannot be redistributed because it would be invalidated as soon as average-Joe connects to the server.
The flaws..?
- could the hacker modify the EXE in less than 10 seconds ?
- still does not fix the content-modification problem (like modifying textures/models or external DLLs).
Thoughts..?
You can try an asset (including DLL) hashing system, where a hash of a random asset is checked periodically - if the hash fails there's an asset hack on the client, flag it dirty on the login box, and a new asset must be acquired before the client will authenticate. Hash checks can be done on the login box, so shouldn't slow down the game too much. You could even periodically (if you get clientside time) generate hashes from in-memory assets and get those checked - would make run-time swapping of assets more difficult.
If you package potentially exploitable code into its own DLLs and keep 'em encrypted (decrypt to memory, then load the DLL from there) you make disassembly very difficult.
You could also package required game-server negotiation information in 'up-to-date' DLLs, such as D-H primes and generators. Might make a smaller download than the whole exe, and eases the burden of modifying protocols all the time - the clients simply won't be able to negotiate an encryption key.
Just a couple of ideas I've been working on myself.