Summary of my nav-data: built from 3D world grid as maps are already built on a grid (it's X-COM'ish), then merged into largest area bounding boxes (if two boxes can't merge, check if they could former a larger volume if merged were the remainder cut away), cells know if they're adjacent to “sky” in which case they can path to any other “sky” touching cell like an off-mesh connection in any other nav-mesh, and when stuff gets blown up the merge routines get called a bunch of times.
Primary jank is that if things have cell masking rules then those are different nav-meshes - so far there's only 1 of those, for cells big monsters can just bulldoze through unbothered, but it's probably going to become a problem as more things come along for “I can go through here, but I have to pop a nova first to level the place.”
Biggest headache for me has been convincing flight motion with regards to queueing at chokepoints. As my system is grid based to the design grid there's also the added bit that the aperture of actual passage is not assured to be the size of a cell, but could be a small window with varying traits (barred, glass, broken) that a CrocoBat can fit through but a straight up dragon can't do diddly.
I deal with queueing (flyers need to circle like vultures if they can't hover) by refitting a collection of curves to the nav cell placed at the top of it. Flyers will head up and circle in these stacks of curves until their turn. The nav-cells know if they're attached to open-sky, in which case this circling behaviour continues outside of the map area and in the “sky” area, which is outside the map-grid. They enter their curve by boids'n it up along a path through the center of the curves, causes them to swoop low, up, and out like a mushroom cloud. Right now I've just got a timing control on a flyer-type basis that controls how tight the queue delay is, so CrocoBats will basically swarm right on through as fast as they can while GatorSkyEels (eh … it flies … IDK … RIP my naming sense) putz for a bit as they are chonk. This gets reused for idle behaviour, so the fliers will go into circling mode.
Those curves do get moved around, as undulating flyers have different spacing requirements and there's some room for some noise vertically. The fitment to the cube is XZ (Y is Up) done as Laplacian deform with Eigen to fit the box. Curves were a nice solution here as I can … like look at them and roughly understand WTF is going to happen as well as use them for the entry and exit strategy of this state.
I would really like to ditch having the separate box network for flyers and work with grid itself, but I feel I would end up having it anyways just to have the information required to setup the circling layouts. There are non-grid maps, but those are like classic DOOM maps in that they're sectors with height and they get detailed with various lofts and tessellations - so there's just a merge of sectors whose ceilings levels are the same with limitations on height difference - also, flyers are strongly avoided on those maps outside of gimmick rooms that will lock you in with them and a battle situation just makes them boids.
For the boids'ing everybody reports their flight corridor accounting for their undulation ranges and how far a boids looks depth-wise into connected cells depends on the volume of the cell. Things are allowed to go crazy, if need be, but everbody is rocking that front-line repulse like leader-following (though I use a cone). Rules applied vary on the flyer, CrocoBats swarm and thus gather, GatorSkyEels are basically Rookie slaying machines and do their own thing that all lesser boids avoid which creates a nice scatter and you can actually see “ahhh, damn, here comes the big one.”
Dealing with erroneous situations has basically been a content problem. Right now I'm just using some key poses with some inertialization and a few other parameters for interpolating these shock response poses with some wiggle bone dynamics on wing joints. Unsure how tractable that will be moving forward into final art - jank may shape what's acceptable for the final look.
---
Flyers have definitely been easier than serpentine navigators, though I do admit to copping-out and pulling some MonsterHunter jank with flyers. Serpentine has me losing hair fast enough to consider scrapping it, but I'd sooner scrap complicated interiors than the gigantic snakes. There are swimmers … it's fake knee-high FX water (or oops … you're dead, lava) … those were easy.
Nowhere near done, still have to visit ambush behaviours both in regards to introductions of a flyer and as part of that navigation leaping from sky-touching-cell to sky-touching-cell.
