I've been using SDL for years and I think it's just awesome to get stuff working crossplatform much easier. But of course it still takes time to get some specific things working in some weird situations. Probably took me a few days to get my little framework working with Emscripten the first time I used it, mostly because of a shader bug which only happened on Chromium.
I think you're in the right direction with designing it to work with different libraries, the final game under your framework/engine shouldn't care what library is creating the graphics contexts or processing input events or whatever. You can even make your own code for handling those later on, if it's really needed. Just don't spend too much time on making some random wrapper classes at first. Make it work with one library first and clean up if/when you realize something could be done better.