How do the settings in NVidia control panel work? This is confusing to me because engines no longer use fixed pipeline and rendering is done very differently in every game, so, for instance, how would the graphics card know which shader is responsible for ambient occlusion, and how to disable it without messing up the entire rendering process?
How does NVidia CP work?
It doesn't, it is basically a driver hack and nvidia has a list of games for which a given hack(such as AO) works, support for new games is added with driver updates. (nvidia added AO support to starcraft 2 wings of liberty in the R265 driver for example)
Having seen the code in question inside the NVIDIA driver, I can tell you it's not magic. You know a game pipeline - either a specific game, or a particular family of games, has a particular way of rendering. Based on the sizing and usage of a render target, for example, you can identify it as the main scene RT, or G-buffers, or post processing, whatever. And then the driver simply sneaks in extra rendering passes, or drops rendering passes, or does whatever other clever thing someone's decided to add.