Question: why do we suddenly need, in 2014, much closer access to hardware? When hardware has been getting faster at a crazy rate and phones have the same 3D power as gaming rigs used to, why did someone decide "we need a few percent more by completely redesigning all the drivers and APIs"?
If this was happening 10 years ago it would make sense to me, but these days we have such an abundance of GPU power and it is increasing faster than you can keep up!
We don't "suddenly need" it in 2014, we've needed it for quite some time longer now; 2014 just happens to be the year when the APIs, hardware vendors and developers are all coming around to the same kind of thinking (it could have been any year, it just happens to be this year).
Previously there have been other solutions to the problem of API overhead - batching, instancing, primitive restart, texture arrays, draw indirect, etc - but those solutions have their own limitations and have been pushed pretty much as far as they can go. The new APIs just solve the same problems that these other solutions solve, but in a more general and more useful way.