I found a free slot this evening to sit down and implement a hit scanning routine for the engine.
As a precursor step I had to implement a spatial hashing data structure to store all the objects in the world in while providing fast overlap detection.
For my hit scanning routine I ray march along the ray in question looking at all of the map cells and checking if any objects were hashed into them. Any object was, then it gets clipped against the level geometry, as well as tested for distance from the ray. If its completely clipped or too far from the ray it gets discarded. Finally the closest of all the remaining objects is returned.
As you can see from the video below, it seems to be working okay in most situations, so I don't expect to need to change my approach much.
You can see a debug visualization of the spatial hash debug drawn in the top left corner of the window.
Next task is either monster sprite animations or particles and blood spray.
That's about the same process with what I did.
I first checked for any mobs within the hit angle based on distance the distance to the shooter. Then I do a ray trace between the mob and the player for any level geometry or doors, and grab the closest success as the hit. I'm pretty sure that's not the best way to do it, but it's been working well so far.
If I need to do something similar in the future, I think I'll go with something closer to your method.