I think C# is a fine choice, especially for learning how a game engine works. OpenTK is a great library to use to get started. It has openGL/AL/CL bindings + all the utility, input, math classes you can shake a stick at. My game engine uses an OpenGL 4.3 context just fine with OpenTK and I haven't run into any limitations, aside from my own capability.
I've integrated Lua into my C# game engine and if you're looking at other languages, There's the iron languages (IronPython, IronScheme, etc) which integrate really nicely into C# since they're all managed languages using the same runtime.