Some question to the audio experts. I want to handle sound in a dungeon/cave like environment. The first question is, how to occlude sound properly.

I.e. when a creature is behind a wall, I should damp or turn off the sound it makes. On the other hand, when a open door is next to the wall, I would hear the creature, even if it is not visible. To make it even more complicated, the sound would come from the direction of the door, not the wall. Even more complex: What if there are two doors, one to the left and one to the right of the player and a wall between player and creature ?

Something like this:

C
-| |-------| |---
P



To be honest, I don't have the time to create an accurate sound landscape, nor will I add some audio libs which handles occlusion (EAX?). I would like to hear some feasable/common tricks of handling this kind of issue. I have a waypoint system which could be utilise to handle/fake sound reflection, but I don't know if this kind of fakes are too clumpsy.

This is a difficult problem in the general case, but what you could do is trace a bunch of rays coming from the monster's mouth (or a crude approximation e.g. rays coming from the monster's center in an hemisphere centered on the direction in which he is facing), have them bounce around the scene (you will need to add sound materials to your geometry to describe how sound will interact with the scene). Then you count up the number of rays which reach the player. The ratio of those which make it over the total number of rays traced corresponds to the volume ratio. If you already have collision detection implemented then tracing rays and intersecting them with the scene should be easy. You also want to implement diffraction as it is extremely noticeable for sound waves (as their wavelength is pretty big).

A more efficient method is to trace rays from both the monster and the player, and then checking whether a connection exists (and summing them up in a similar way). This is very efficient for multiple monsters or in general sound emitters since you only need to generate the player rays once for all the monsters).

Because this is a random process, you will need to smooth the resulting ratio somewhat because you will obtain slightly different ratios every frame which will cause crackling if used naively.

Anyway that's the "almost physically correct" method. I'm sure many approximations exist such as baking the sound landscape. I would go for the physically correct version myself as it doesn't sound too computationally intensive and if done right can be stunning (you inherently get 3D sound using that method, by the way, so if you have compatible hardware it's pretty awesome).

This also handles every imaginable situation. If there is a wall between the creature and the player then the player will hear only muffled sounds (because the wall doesn't stop 100% of the acoustic rays, but rather something like 85-90% - this is what I mean by sound materials). If there is a door on the side, some sound rays from the monster will make it through the door and hit the player, which will increase the sound made by the monster.

The raytracing approach sounds interesting, but the implementation overhead seems to be quite high and performance could be an issue too.

After looking for a solution I found out, that unreal uses a simple occlusion handling by reducing the sound radius to ~65% when a simple line-of-sign test failed. Any other options ?

Look up sound propagation (the technical term for what you want to do). Most geometric techniques will trace either rays or frusta to sample visible surfaces in the environment, then perform diffraction/reflection path validation on those surfaces. Numerical simulation is more accurate but several orders of magnitude slower. Either way, you are going to have to do some sort of ray tracing to perform occlusion tests, even if you do something simple.

I'm in the process of developing a sound propagation middleware library called GSound. It's free to use for evaluation and for non-commercial applications, licensing for commercial use is negotiable. My paper which outlines the ray tracing approach that GSound uses is also online at that page if you want to get an idea of how it works. This system will produce all normal environmental acoustic effects (reverb, reflections, diffractions, transmissions) automatically and fast enough to be used in games.

The basic interface to the library is fairly simple and can be used in conjunction with or as a replacement for tools like FMOD (though the feature sets are not identical, we focus only on sound propagation right now).

Let me know if you have any more specific questions.

Quite interesting stuff
Atleast I will choose the simple way.

Just a thought:
Did you consider to link your sound system to a physics engine ? Most games already utilise phyiscs engine and use it to hold an abstract version of their world. There're some good open source phyiscs engine which could be extended to work with your audio engine.