Personally i went from XNA to SLimDX and have not looked back, yes you will have to build a number of things from scratch, but its all still c# and you have more freedom. From your background it seems you might want that.
Or even start with OpenTK/TAU and use Opengl, combines with Mono you could do multiplatform development.