Heh, I'll ask again since it seems my question was skipped: how are you downsampling your voxel data? This might play a role in the issue you're having.
I'm pretty certain I've solved my issue. But for your interest:
The same method that this guy uses:
I use hardware mip-mapping and then filtering with the compute shader by transferring values for each 3x3x3 voxels to neighbouring voxels in each direction.
Btw, I've also posted a video at the bottom of the start of my thread.