Actually your suggestion seems quite sound. Using an AAB that completely encompasses your AI's FOV could be done for a quick check to see if the player could be in the enemy AI's FOV. Then perform a raycast from the enemy to the player to determine if the player is actually in the line of sight of the enemy. Raycasting can be done using DDA. Which is pretty quick if you are only casting one ray.
I am not sure about your layer concept. If you mean the upper halfof the screen is a different layer vs the lower half. Then the same grid or matrix you use for collision detection is the same one you cast your ray into.