[..] it just seems so much easier to do the abstraction on a higher level than the lower one.
Whether it's a generic interface to renderer(s) or something else, abstractions tend to work better at a higher level. Otherwise you may as well stick to the low level code minus the abstraction in the first place.
Designing this well is another issue...
 You seem to have come to these conclusions on your own. Perhaps you are just seeking validation.