Does it always work the first time?
More like does it ever work first time. In general there's always a bit of back and forth with it. The more planning that went in the more it pays off and good communication between the developers themselves is crucial, in fact you can gauge how communicative they were just by looking at the amount/ugliness of the glue-code between systems.
Is the main programmer the person that does the merging?
Sometimes, but not always. As SillyCow mentioned the important thing is to have an API boundary that both sides can agree on and work with. I find these APIs work best when created by one person who has taken the time to understand what the other guy needs.
I would think it would require a lot of communication and planning before even writing the code.
It can and probably should. A lot of the time though these things arrive organically.
How do they even keep track of this massive project and the flow of the program?
By having well-compartmentalised systems with clearly defined API boundaries. The overall flow is oft defined by a sort of backbone, like the main game loop, or a messaging system with a reflex-arc triggered by something tangible (like a user clicking a button).
This organisation doesn't always arrive though - I've inherited projects where the communication between systems is blurry with complex, hard to tease apart interactions and dependencies. Keeping track of things then becomes something that only the original developers can do, so access to pick their brain is pretty darn useful.
When they start the project, is it usually a prototype just to see if it works before merging more systems?
I like prototyping, I think it's seriously under-valued as an approach. A lot of the time a prototype isn't made or, if it is, then it'll just be a single "prototype" and regardless of it's successes/failures proceeds to become the full-fledged product anyway.