So what do you think I should do to make this work otherwise I'm almost totally stumped?

again, what exactly are you trying to accomplish?

Okay here is a full explanation for your pleasure

What I'm trying to do is mouse selection in a different way then doing ray collision checking/color picking/index object picking/selection mode.

I'm doing my own mouse selection which I would like to call Area Selection Translation how this works is :

1.) position a quad at the mouse cursor position

2.) When I left click display a quad with a texture that has a border In the texture and colorkey the center using a fragment shader.

3.) Then when you left click and drag the quad whatever is under the quad is translated with it


making for a very simple way of doing mouse selection and skipping out any complexities that might come along with that

Now the problem is I do not know how to position the quad under the mouse cursor even using gluUnproject doesn't provide me with the results I need which is the above step by step ^^

I hope you understand better.

I'm using WIN32 to get the mouse cursor position
OpenGL 3.0 to draw the quad
GLSL to colorkey the center part of the quads texture

here is a image that might help you understand better



here what I got so far obviously there is still a texture problem which I will fix later but -> http://pastebin.com/5eQjfTFP

Ah, okay, I see.

If it was me, what I would do is instead of trying to draw the selection quad in world space, draw it during the UI phase. The UI phase is done after rendering the 3D scene, and is typically done in an orthographic projection. This way, the quad is drawn in screen space overlaid upon the scene, and you don't have to worry about reverse projection into the world and all of the clipping and z-fighting that may occur.

Once the quad selection is done, you can then un-project the screen coordinates of the four corners of the quad into world space using a depth of 0 and a depth of 1 (thus two unprojections per quad corner). The result will be a set of 8 points representing the volume occluded by the on-screen selection quad. You can then test geometry against this volume to see if it is selected or not.

so render the quad in orthographic then select the quad which would happen automatically since I would already have left clicked then unproject after rendered how would I do the two unprojections per quad corner code? also how would opengl know of the corners? when using glVertex3f();

You know where the mouse is, right? And you know how large the quad is? Just calculate screen coordinates for the corners of the quad as offsets from the mouse location. Then reverse project those screen coordinates using gluUnProject.

If you un-project the quad using a winZ of 0, you end up with a quad in world-space co-planar with the near clip plane. If you un-project using winZ=1 you end up with a quad on the far clip plane. Join those quads together into a solid volume, and voila. You have your selection volume.

I don't know how to calculate screen cordinates for the corners of the quad I'm not a math person by nature thats why I was asking for help in the begining anyways.

If your quad is M by N pixels in size, and your mouse is at screen coords (mx, my), then the corners of the quad in screen space are (mx-M/2, my-N/2), (mx+M/2, my-N/2), (mx-M/2, my+N/2), (mx+M/2, my+N/2). Simple as that.

Really, if this is so hard for you to understand, I recommend that you take a break from this and work through some 3D math primers. You are going to need at least a halfway decent grasp on this math if you want to succeed in what you are attempting without having to rely on a huge amount of hand-holding.

naa this is fine I just needed a head start in the right direction so these are equations for calculating [color=#282828][font=helvetica, arial, verdana, tahoma, sans-serif]

[background=rgb(250, 251, 252)]screen coordinates of the pixel demension of a quad?[/background]

[/font]