I'd suggest trying to pre-voxelize everything and just inject it into your grid. I really don't understand why there was the insistence on re-voxelization every frame, except maybe because the voxel cone tracing paper was more academic than practical initially. Certainly dynamic re-voxelization might be useful for dynamic destruction/other physics/deformation stuff. But really, most of your stuff is just going to a static model, and even animated models can be voxelized and transformed.
Why waste all those resources rasterizing everything multiple times per frame? That being said, if you really do have something that needs to be re-rasterized, you can use Epic's trick of marking things as static and checking each frame to see if they actually need to be re-voxelized. Otherwise they just re-use the previous voxelization. Anyway, it's looking good.