Poly B in that example is horribly inappropriate anyway. It shouldn't connect to the top and bottom walls at all. Instead, the corners of the blue boxes should be connected to each other making area B represent the area between them.
That allows you to simply mark poly B as being off-limits for agents that can't use it.
Thank you IADaveMark. I have thought about this situation, but even poly B is generated as you say, this problem is still exsit. Or it would be an easy way to do some algorithem base on an appropriate generation?
You might want to take a look at the Recast and Detour library, which provides automatic generation of navmeshes. A part of the parametric structure that you populate to generate the navmesh includes a member for agent radius and another for agent height, which affect the generation of the navmesh. A solution to the problem of agents of multiple sizes is to generate multiple navmeshes, and roughly classify the agents into a fixed number of size categories equal to the number of navmeshes. This way, you don't have to complicate the actual pathfind with additional logic to cull polygons based on agent size.
Always shrink your navmesh for agent size.
We have tested this problem at work and we found that generating several navmeshes - as JT suggest - is far more efficient (less memory AND less cpu cost) than generating one navmesh that supports several agent radii.
Thank you for the advise of multiply navmeshes, Steadtler and JTippetts. But here comes another problem. The project I'm working on is an RTS game, so it involves dynamic addition and removing of obstacles like buildings, and it'll be complicated for me to generate multiply navmeshes dynamically in an efficient way.
I've read the
pathfinding of Company of Heros, the obstacles change frequently and it uses grids not navmeshes, and that's the way I used in past too, the reason why I want to replace it by navmesh is that I couldn't find an efficient and precise way to smooth the path, it means the path should turn only at the corner of obstacles( I'm using a line of sight test to every waypoint choosed by bisection method ). The effect in Starcraft2 is what I want to achieve ( obviously the buildings in Starcraft2 are on grids, and the path of one unit is very smooth and precise ), not flocking behavior, just one unit one path. Could you give me some advises?