It's going to be one hell of a work.
If you think voxel world, I think you're trying to do minecraft. An easy way to deal with collisions is the use of an external library. Bullet goes great with OpenGL since it uses the same coordinate system. Unfortunately, it's likely to run amok as you walk away from the center. Minecraft has an ad-hoc physics and still runs amok - after a few hours of walking from the origin I have been told.
Nonetheless, I strongly suggest to look at a physics library.
Due to the massive amounts of cubes in voxel worlds, you'll probably have to implement some very gross-grained block merge. While I've found cuboids to be fairly fast at both operation and add/remove operations, once you start having ten thousands objects, you'll be taxing the system a bit more than I'd like to and if your chunks are - say 30x30x30 you'll be blasting that big way!