The RPG I'm working on uses a custom C++ engine and Lua for scripting. It's open source, well commented, and has a decent amount of documentation, so maybe you could snoop around what we've done there and get some ideas.
- Code - https://bitbucket.org/allacrost/allacrost
- Wiki - http://www.allacrost.org/wiki/index.php/Main_Page
- Relevant code locations: game/src/engine/script/ (Scripting Engine), game/src/modes/map/map_event.h (Map events), game/lua/scripts/maps (Lua map scripts)
I've been thinking about putting a video together on our youtube channel to explain how the map exploration code in the game works and how you build a script with it, but unfortunately won't get around to that until next month at the earliest. Here's some general designs that we've implemented.
First, we use Luabind to connect our C++ and Lua code. Luabind allows us to specify which classes and functions we want to be available within our Lua files. Most of the code that processes what happens on a map happens on the C++ side (or Lua calling into the C++ functions). For example, path-finding, animations, and drawing all happen in C++. The things that happen in Lua for a map are some large functions that populate the map with sprites, objects (like treasures), zones (defined areas of the map used for different purposes), and events (things that happen on a map). Lua will also have a number of small custom functions to do specialized things that we need, for example shaking the screen during an earthquake scene, moving the camera to a different location, or telling a sprite to move to a specific destination.
Most of the class objects we create in Lua we send a call to the corresponding manager class in C++ to handle that object. So every sprite we create is sent to a sprite manager, for instance, so that the sprite manager can figure out which sprites need to be updated, drawn to the screen, and handle collision detection between sprites. When we do this, we also tell C++ "I created this object in memory. I'm now giving it to you with the responsibility of destroying it when you are done". This isn't something you have to do, but it helps us to keep all object destruction contained in a single place rather than rely on Lua's garbage collector. (We use special arguments in our Luabind binding code to make this happen).
Now one thing I want to describe in detail is our event management system. Our maps all have an event manager, which holds event objects of various types. You may have an event that moves a sprite to a destination, an event to display a dialogue between characters, an event that causes a battle to occur, an event that plays an audio file, and so on. We also have custom events that contain pointers to Lua functions to run. Events can be chained together so that after one event finishes, another will begin automatically. This is useful for sequences where we want to have several things happen one right after another. You can specify how long to wait after an event finishes before the next one to begin, or you can start another event at the same time that the current event begins. Every event has two functions: a "Start" function that runs when the event begins, and an "Update" function that updates the event as necessary, and returns true when the event is finished. So a sprite move event would specify the desired location for the sprite in "Start" and the "Update" function wouldn't return true until the sprite reaches that destination. This is a simple system that offers great flexibility, which is what you need to build maps that are interesting.
The image below hopefully helps illustrate this idea of event chaining. We use Lua code to setup an event chain and to check for the conditions when an event chain should begin (for example, when a sprite walks into a certain area of the map). Lua tells the event manager to start the first event in the chain, and from there the C++ code handles the rest, processing each event until the sequence is complete (or processing it infinitely if there's a loop in the event chain).
I know that's a lot of ideas and words I'm throwing around, but hopefully some of what I said makes sense and helps you get a better idea of how C++ and Lua can co-exist. This was one of the most difficult problems for me to figure out myself, so my best advice to you is to experiment and figure out what works best for your situation. Maybe it's something like we've done, or maybe it's something entirely different. There's really not a right or wrong answer to how to mix Lua and C++ together.