Well you could do two things:

1. Use soft particles to remove this hard penetration.

2. Use yours/anothers physics library to detect collision between the particles and the surroundings and then do the desired action.

Sometimes smoke can occur somewhere, but it should never penetrate other meshes such as a door or building.

Usually not a problem, as long as the back side is not visible. While a cloud of smoke half-disappearing into the turret of a burning tank is not 100% realistic, its usually considered acceptable, at least from the outside view. from the inside the turret view, smoke that's outside should definitely not penetrate the hull.

How do I prevent smoke particles from penetrating meshes?

I take it you have a specific case. what is it?

it may be something that can be handled with drawing order and manipulating zbuf test and zbuf write. "overlapping" type issues can often be handled simply by drawing things in the correct way (with respect to things like zbuf, blends ops, filters, etc), and/or in the correct order.

You probably need to go the physics route.

A couple of approaches i can think of:

1.Remove all smoke outside the 'room' (if the room is a box you could just iterate through all smoke and remove offenders. You could also remove all smoke behind a given plane or shape)

This wont affect the behaviour of the smoke in any way, inside the room it will appear the smoke is just going through the roof (but it wont appear outside)

2.Make the smoke particles be actual physical bodies (spheres, points, whatever works) with some special physics. You would probably make them:

-Low mass

-Force pushing them up

-Collisions would not be rigid but smooth so that the smoke can compress. You might want the internal collisions keep a greater distance than collisions with the world so that the smoke can go through small holes but still be relatively sparse.

3.Represent the smoke using a volume (a 3D grid for example) where you render each cell that has smoke as a particle (or use some other technique of rendering it)

You would model the spreading of the smoke inside the grid. You could mark which grid cells have obstacles to not spread there.

1. will be the simplest especially if the room/obstacles are simple and you can easily check which particles are inside/outside them. It will look ugly though but if the smoke is dense at all and wont move too fast nobody will notice.

2. will integrate better with your world and work well with dynamic obstacles. Its more scalable than 3. because the smoke particles dont need a denseish grid to back them. However its also poorly scalable in that a lot of particles will require a lot of collision checks and stuff.

3. might allow for realistic rendering of the smoke because you can use fancy GPU raycasting to render it (???) The physics of the smoke wont come as easily as with using physical bodies for each particle, but it should be faster for dense smoke with lots of particles/cells than 2. Id say this works best for static emitters and environments.

I don't get what you mean, the smoke particles are ALWAYS facing the camera, so there is nothing called back side at all, because the back side never face the camera.

the backside of the surface being penetrated, not the backside of the smoke quad.

IE smoke is outside the door. some smoke penetrates the front side of the door (the outside of the door). camera is on the backside of the door (inside the room). smoke penetrates through the door into the room, camera is on back side of door, and can see smoke sticking into the room through the door. camera on backside of penetrated surface = problem.

typically one would clip particles that collide or near-collide with surfaces that should be non-penetrable (and where the penetration can be seen from the back side of the penetrated surface <g>). either simply clip them or actually deactivate the particle.