Look into build configuration, there are tools like jam, cmake, premake, etc... They will allow you to generate your project base on configuration settings. For instance in your source, you could have a folder for OpenGL and Directx implementations, then base on the configuration your project would build different files. This is to control which files to build, you still need to work with macros to handle platform specific includes and implementations.
Create a manager with various rendering buckets, base on materials, shaders etc... Then whichever bucket the object belong to, add it to that bucket (probably some sort of reference, id or something, as you might have another manager that manage all the objects). When it time to render, you can render all the objects from different buckets.