A game that properly supports mods is going to have some kind of "entry point" for the mod in question -- usually something like a definition file of some sort. It could just be a text file in the mods folder (e.g. modname_mod.txt) that lists components and dependencies. Yes, your engine has to be aware of it, and has to know how to load them. Modding in games that don't actively support it typically involves editing data files or DLL injection.
This can all be done lazily. When the user clicks "select mods", for example, the mod loader can scan the mods directory for entry points. When the user selects an entry point, the mod can then load the requisite dlls ('plugins'), mods, or resources. If the user adds a mod, quits to the main menu, and then clicks "select mods" again, the engine could theoretically reload the mods again. Realistically though, most engines I've seen load mods at startup (or have the selector in a pre-game splash screen a la mount and blade or some paradox games), and it's nice to be able to do that for things like "mod manager" mods. It's worth noting that you can however do all of this dynamically, and it shouldn't be performance-impacting if you do so at the right place and time.
As for sharing resources, that should be fine. Just require a certain model in the models folder, or a certain sound in the sounds folder. Mods should ask the game engine to load resources the game engine can load. If multiple mods need the resource, you can just use a reference counting solution or something to track cleanup.
Version incompatibilities are a thing, but that's just like game version incompatibilities. Either the server needs to require a specific version of the mod, or it needs to automatically give that version to clients. As for your synchronization questions, that all depends on the operation. If the client modifies the UI to be purple or something, who cares? Anything the client can do in an RTS that affects other players should be sent to other players. If the client says "I have a tank here with 10,000,000 damage", the server can say "I know what your tank is, and I know you've given it two upgrades and it has 30 damage". That kind of validation isn't a modding issue, it's a fundamental anti-cheat issue that the core game needs anyway.