As OpenGL version a suggest 3.3+ because almoste all modern graphic card support it and is very similar to OpenGL 4.0.
As GUI library you can use the Qt library.
As math library you can use Eigen (very fast with extensive use of Expression Template and SIMD istruction (SSE, AVX, Neon)).
But you have to learn very well GLSL language. OpenGL is a C API and is not easy to write OOP code. Take a look to (http://oglplus.org/)