Anyway to get to the point I'm looking for a more minimalist 3D engine that has the bare minimum needed to get started with 3D with a lot of low level control.

To be honest, take opengl or directx. A working framework needs a lot of infrastructure and architecture, which will enforce you to use all its classes, structures to utilize their engine.

Something like SDL 2.0 or DirectX Tool Kit (DXTK) might be close to what you're looking for. They help take care of the inane details of dealing with the platform, give you some nice APIs for doing common things, and integrating other higher-level functionality (like say, Bullet physics library) is fairly well documented by the community, but other than that you can have a straight-up 3D API interface with little or no interjection. But, teh downside for you might be that these things are not engines as you know them -- You'll probably have to roll a fair bit of your own tech (or integrate third-party solutions) for things like materials, models, etc.

That said, many people seem to find things like Unity adequate, and when I come to a point of feeling like an engine or library is stupid, myself, I've learned over time to take a step back and ask myself whether it really is, or if I've just not taken the time to learn it right. Your reaction might be something of a symptom of NIH syndrome, make sure its not because its going to take a lot of time and effort to re-implement the things Unity will give you (not to mention that Unity is debugged and battle-tested).

I'm actually working with UE4 on my current project and I've had a great experience with it so far. With that said, I don't think it'll fit you well with you based on your OP.

UE4 is built more as an all-inclusive framework that you game is built on top of rather than a library that is built into your game. So for you it'll seem just like Unity where they require you to use their style of framework; this is the exact same approach UE4 takes. Also, although UE4 has the source available to subscribers, it is by no means minimalistic as all the core components of the engine are rather tightly knit together. Lastly, if you found Unity's GUI system painful, then you will find UE4's Slate system even more painful. The only way to get a "good" UI system into UE4 is essentially what would need to be done with Unity which is purchase a 3rd-party GUI framework (Scaleform and Coherent UI are the two big ones for UE4 thus far).

I'm not saying UE4 is a bad engine (heck I use it myself), but I'm just saying that it's not going to fit your needs/wants.