How could GLSL even have file includes? GLSL doesn't even know about source files. You would either need a feedback mechanism telling the host application which include file is needed or the host application would have to upload each and every file that might possibly be required by the current source file. To get that right, you would have to parse your GLSL source files anyway to find out which includes you need. It's only a small step towards handling the includes yourself from there.
Linking is a completely different story though, the system knows all symbols generated by its source files and does not need any two-way communication with the host application.
So, I think the way it is done is the only feasible one.
Yeah, looks like you start the program from your IDE, which does not keep the console windows open after the program ends, so after displaying "bonus: foo", the program ends (as intended) and the console window belonging to it is closed by Windows. Do what Aliii proposes. Maybe use cin::get() so you don't have to add more headers.
How can you ever hope to accomplish anything without using formal mathematics? Even your programming language's syntax is formal mathematics (thus the name "formal language"). So by writing a program you do actually use formal mathematics.
Also, without formally finishing your derivations, you will write unneccessarily complex code, which will often lead to slower code.
On the other hand, there seems to be no sensible reason not to use mathematics. Or can you name even a single one to back up your statement?
Just as an addition, I always prefer using the C99 header stdint.h (also available as cstdint in C++) and the types uint8_t and similar and not handling this mess myself. In one company I worked for, this was not done and during a switch to 64 bit I had to update a whole bunch of header files with lots of #ifdefs to take care of this. Had the project used stdint, it would have saved me a whole day of debugging to find the problem and fix it.
Don't go for the 3d engine too quick. Take your time to really understand the basics of programming. Read books or online articles about efficiency and correctness. Do some serious work on 3d maths. Study other engines' structures. Blender may be interesting because it can show you how artists think about 3d data (and how that may differ from a programmer's perspective).
So, yes it is possible to write a polished 3d engine in C++, even on your own, but it is nothing you can do in a few months or (depending on your learning curve, spare time and motivation) even years. Keep that in mind. Also, if your goal is to write a game, write a game, not an engine.
Ruby is a fine language and I see no reason why it shouldn't be usable as a high level scripting language.
You could use OpenGL, but this means you'll have to write a lot of shader code to do this. And since you (I assume) don't have to make it a real time program, you might as well use the CPU instead. If you are allowed to use it in course, take a look at OpenCV, which is one of the most used libraries for image processing / computer vision.
It should have at least Sobel and median filters built in, but they are really easy to implement yourself (and doing so will teach you some of the most importent basic tools and techniques in field), like simple matrix convolutions, image formats and the like.