the possibility of script-insecurity is mostly because the script-language in my case is largely due to the existence of a root-accessible C FFI.

code running under a non-root user generally lacks direct FFI access, and generally has to go through a "_setugid" wrapper, though I could possibly add support for being able to mark various C functions as callable via untrusted code (vs requiring a wrapper class).

clients hacking servers is likely slightly less of an issue at present, as there is currently no way to directly send code to the server (except possibly via console commands, but these are filtered), and currently no RPC is used (RPC had generally also been seen as a potential security risk). most normal communication is via list-format messages (except partly with voxel data, which are sent mostly as flattened out byte-arrays).

Yes, the clients should never send script back to the server, that would be a basket full of snakes to keep secure..

What seems to be missing here is that it sounds like you are not sandboxing the scripting on either side and instead possibly sharing a VM environment? For our usage in a full client/server networked model each side had a unique VM instance and each one allowed access to only functionality defined for that side. What I mean is that there were basically three tables of functions which were registered with the VM's, on the client the "client and shared" tables were registered, on the server the "server and shared" were registered. As such there was no possibility of the client even knowing of the server accessible functions and even if they did get a list, they didn't have "transmission" ID's which would map a message/RPC/whatever to call those functions, it would just be an invalid and dropped message.

If I'm taking something out of context, please correct me but it sounds like a shared VM over client and server which sounds like a bad idea.

both run the same script VM, but generally they don't directly share memory or object handles (all communication is via passing messages).

more so, you couldn't physically share a VM instance over a network anyways, this just wouldn't work.

the code sharing would mostly be pushing globs of code over to the client (as messages) and essentially executing them on the client (basically, as a big "eval").

functionally, it isn't much different from something like a "centerprint" event (where a message is sent to be displayed in the middle of the users' screen), or a "stufftext" event (where commands to be executed are sent to the client's console, with scrubbing).

this code would be specific to the client though (IOW: it wouldn't and couldn't run on the server, if anything, because the available features and APIs would probably be different).

theoretically, the code could be sent to the client as bytecode, but this could potentially introduce more versioning issues, as the bytecode isn't really entirely nailed down, and may be more prone to change than the source-language.

originally, the script VM was mostly written for the server end, but has ended up being used for some client-side stuff as well.

not sure why a person would want/need different scripting VMs for the client and server though (at least in both cases, the same language can be used, ...).

also, the scripting VM largely predates much of the rest of the engine (my scripting language has been being developed since ~ 2004, whereas my game project didn't really start until around late 2010).

as for gameplay hacking:

the main obvious hacks are mostly, if control if player movement were moved to the client, it would be possible for a hacked client to rig up things like flying/noclip/teleporting/...

I.e. the standard hack types because most games do want to run the movement on the client in order to make it as responsive and smooth as possible. The only "absolute" solution is to say "don't do it" but that sucks because then you have all the latency issues, "too" smooth movement if the server runs as lower update rates than the client graphics updates etc. If you can get away with leaving the latency between player saying move and acting on it (or hide it, harder but possible, I pushed this solution for a prior game but ended up overruled because it wasn't coded like "normal" game objects anymore ) then that is still your best bet for non-hackable. Mostly folks go with the client controls movement though because it is basically the same as a single player game and everyone knows how to code it. Anyway:

With some work you can put in a "90%" catch the bastards system which is not impossible to hack (NOTE: teleporting can always be caught, that's easy to determine server side without much work), it just raises the bar from being something anyone that downloads Cheat Engine can do, to something someone will likely get kicked offline/banned a couple times before they figure out enough to work around it. The idea is to be sneaky but not intrusive to your code, yet add some layers of double checking things.

I'm not sure how much detail in terms of ways to catch cheaters you want but basically if they have the executable available and you have moved movement to the client, it is really your only solution. It's not perfect and you just have to deal with the fact that some percentage of cheaters will get away with it. As long as the number of cheaters is fairly low, it's worth ignoring them if 99% (hopefully 99+%) are foiled.

I already have server side movement, and it works ok for now.

the main drawback is that this could potentially require a lot more CPU power / memory-use / ... per-client on the server.

doing stuff on the client could likely allow more clients on a given server by reducing the resource requirements required to manage each of them.

possibly though, this could be made a server-level policy decision, rather than hard-coded into the game though. possibly, an entity flag or similar could be used to say whether the client or server currently is managing player movement physics, and a global setting could control whether client-side movement physics are enabled.

sv_clientsidemovement 1

add:

the uncertainty here is whether the client side physics code should be itself a part of the clients' binaries, or itself just streamed as script code.

I was originally thinking of making it as streamed script code, but granted, having as part of the actual program binary could also make sense (and would probably be less effort and also higher-performance).