Hello. I have small problem with OpenGL coordinates. It's been a while since I used OpenGL and back then I used OpenGL 2.1 not 3.3.

So I basically have triangle vertices:

GLfloat Vertices[] = {
0.75f, 0.75f, 0.0f, 1.0f,
0.75f, -0.75f, 0.0f, 1.0f,
-0.75f, -0.75f, 0.0f, 1.0f };


Now I have mouse coordinates for example 512, 384 which is the center at 1024, 768 window.
When I translate this normalized 1.0f equals 1024 and -1.0f equals 0 for X and for Y 1.0f equals 768 and -1.0f 0.
Now I have these coordinates:

896, 672, 0
896, 288, 0
384, 288, 0

I do not care about Z because I need to calculate only the GUI elements, which are on Z = 0;

Now I can calculate if the mouse coordinates are on top of the triangle, but for some reason glfw gives mouse coordinates with top-left coordinate as the 0.0 and OpenGL uses the bottom-left corner as 0.0, this means that I need to turn the mouse yPosition around, but at the center it still would be 384, but any other value would be different. MyPosition = 768 - oldMyPosition;

if((((896 >= MxPosition) == true && (672 >= MyPosition) == true) &&
((896 >= MxPosition) == true && (288 <= MyPosition) == true) &&
((384 <= MxPosition) == true && (288 <= MyPosition) == true)) == true) {
return true;
} else {
return false;
}

Of course this if statement is not the finished one, but should I calculate it this way or should I use some other method?

That is the way I do it. But not using naked constants. Given that you know the window size, you can compute the borders dynamically. That way, when you some day decide to move the initial triangles, the computation will automatically translate to glfw coordinates.

Also, there is no need to compare with "true". The statement "if ((896 >= MxPosition) == true)" is the same as "if (896 >= MxPosition)".

If you have many triangles (or quads) that you want to detect mouse pointer, consider using color picking. That is a technique where you draw all quads with a unique color each, that identifies which quad it is. You draw this to the back buffer, so the user of the program never need to see it.

I know i dont need naked constants nor compare with true, just used these to simplify the post. Thanks for letting me know about the color picking technique. Will look Into that.

If you transform the mouse coordinate into the triangle coordinate system instead of the other way around, you only have to transform one point, instead of three. (or even more if you want to check against many triangles)

Your code is also not checking the triangle, but the bounding rectangle of the triangle. To check the triangle, you can use three line checks, like described in this post: http://stackoverflow...ne-a-point-lies

with point ABC, counter clockwise ordered, if all three "isLeft" is true for lines AB, BC and CA, the point is inside the triangle.

So Something like this, with a triangle.

MouseX = MouseX / (WindowWidth / 2) - 1;
MouseY = MouseY / (WindowHeight /2) - 1;

If(((B.x - A.x) * (MouseY - A.y) - (B.y - A.y) * (MouseX - A.x) >= 0) &&
((B.x - C.x) * (MouseY - C.y) - (B.y - C.y) * (MouseX - C.x) >=0) &&
((C.x - A.x) * (MouseY - A.y) - (C.y - A.y) * (MouseX - A.x) >= 0))

This way i can check more than three points easily?

I would wrap the line test in a function like in the stackoverflow example, that makes for easier to read code. (= less mistakes)
Also a line test is a useful tool to put in your utility toolbox for other uses (for example ai steering behaviors)

Yes, you can use this technique for convex polygons with any number of sides
Concave polygons can be split in several convex polygons.

I havn't checked your code in detail, but it looks about right. Though I think you need to flip the y coordinate in the mouse transformation?

Olof yes of course. I just wrote that if statement on this board. I forgot to flip the mouse Y coordinate around.
Will implement this later for my ScrewGUI.