I need some line of sight algorithms for a 3D enviorment that's bathed in fog of war. The ground won't be flat. it can go below zero, or above it.

And I'd like to add in some buildings with multiple floors. The world is built with tiled prefabs. but the tiling only comes into play when the player enters combat. Think of XCOM. I've already figured out how to do the shader, now comes to play with figuring out visibility on the CPU. I'd like it to be fast enough to be ran on multiple characters, as well as keep awareness of height differences and buildings.

My first though for a basic visibily calculation was to compute a parabola in local space relative to the character. Either run a DDS over it, or take incremental steps over it to get my rays for testing at the end of the field of vision, then revolve it to scan for clear sight, tiles and occluding angles.

But this scould easily have plenty of problems.

Ray casting around a radius sounds like the most simple solution. What exactly are you trying to find though? Do you want to know if the player sees an enemy? If that is the case, you can check if the enemy is within the range of detection, then cast a ray and see if the depth matches the distance(if something is occluding your view from the enemy), if it matches, you see the enemy.

If you need to run this algo across multiple players based on their position I assume, it is important that you do an initial filter so you are not running redundant scans. Depending on your game logic, if things exists on a grid, that too could make it much easier to accomplish this.

Could you clarify what you are trying to find line of sight on and what sort of information you want to collect?(Enemy, ally, distance, height, ect...)

Sorry, hadn't slept very well so my brain isn't quite working.

The line of sight algorithm is for the fog of war, which takes into consideration of the terrain's height. Basically the occluded world. Building's and it's floors, and geometry that'd be marked to occlude it. I'm not quite concerned about checking for visibility against AI right now as that can be pretty simplified.

I should also note that I'm avoiding the use of a volume texture to try and help save space. TO help put it in perspective, I'm chunking the world together with tiling similar to a height field landscape. There will be set paths on how the player is allowed to transition.

Each fog of war texture for a chunk will be a 64x64 L_8bit texture. And it'll just be composited together by the CPU before being uploaded to the GPU.

It sounds like you're referring only to the rendering? Something like Jagged Alliance 3? Like this:

[attachment=32530:jaggies.jpg]