You could grab an SDK from Unreal Engine or CryEngine (the free ones, under an indie license) and as an added bonus you would have a lot of very AAA art assets to test it on.

I'm not sure, though , if it is a good idea to skip the basics of the rendering and jump right into the lighting equation of the participating media - those are some pretty hardcore papers.

Perhaps you already have that low-level experience - just not in the form of the engine you wrote for yourself (e.g. at work) ?

I'm sure you will find lots of plugins and info on Blender - that gets mentioned every other day or so over the forums. Houdini - way less, though.

If you are fixed on OpenGL, you can safely disregard this option :)

Hieroglyph 3 is an open source engine that makes it possible to add new effects / rendering techniques relatively easily - but it uses Direct3D 11. There is a number of different effects already available (deferred rendering, light prepass, tessellated terrain surface, water rendering, voxel texture rendering, etc...), which can be used as a guide for implementing something totally new. There is a small community and moderately active discussions, so if you have questions there is help not too far away.

In full disclosure, it is my library... but I still think it may be able to help you with your testing.