I was a dingus and posted this in the gameplay section as I thought it would be more relevant to that forum but then saw all the threads about visibility in here and wanted to cross post it. Sorry if this is a foul. If this is a huge issue feel free to delete and close the thread in the gameplay forum:
I'm working with some friends on a top down, stealth based, 2D tiled game. One of the issues we're facing is how to have the AI detect that they can see the player.
The reason this is difficult is because our levels have the concept of layers (ie: climb a ladder in the scene to get from the lower layer (1) to the upper layer of the scene (2). This factors into gameplay because we want the AI to be able to see players that are beneath them, unless the players are against the wall or close enough to the wall that the AI shouldn't be able to see them. See the attached image for an example, red is the AI, green is the player, and yellow is the detectable range that the AI can see. Note how the ground cuts off the AI's frustum.
The issue is that half the team wants to push for using a 3D collision system in our 2D game. Meaning everything in the game will have a 3D collision shape in this 2D game where the height is nonvariable except for when you're on different layers.
This seems wasteful to me and I'm trying to think of a way that this system can be implemented simply using 2D AABB's and basic trigonometry. I'm trying to come up with a solution for this in 2D and wanted to see if anyone else had some thoughts. Its safe to assume that we know the height difference between the ai and the player, the distance of the AI from the edge, and the distance of the player from the bottom of the cliff. The map is tiled and it would be fairly trivial to map the player/ai to the nearest tile (the sacrifice of accuracy isn't an issue) so its possible that there's some use in mapping the ai and player to a tile and figuring out visible tiles from that info.
This may be the one problem that we can't solve in 2D and that requires us to go with a 3D system for checking, but I'd really like to avoid that if at all possible.
If I can clear anything up or give more detail please feel free to let me know.