Hey,

In my editor you can select multiple objects by dragging a screen space rectangle around several objects using the mouse, aka rubber band selection. The editor view is 3D and uses a perspective camera, basically the game running in a special editor mode. I would want the selection to select the 3D scene objects that appear inside the rectangle, nothing more.

My current approach is to transform the corner points of the selection rectangle from screen space to NDC space and do the same for the corners of every world-space bounding box of every potentially hit object. This allows me to do a simple 2D rectangle-to-rectangle collision test to see which objects are inside the selection rectangle. This works, but depending on the shape of the objects, their world transform as well as the camera transform I get a lot of false positives.

Any idea how to do this more efficiently? I was thinking the following might work:

For each potentially hit object (basically all visible geometry):

• Iterate over all vertices of the object and transform them to NDC space
• Create a 2D convex hull out of the vertices
• Perform a rectangle-to-convex hull collision test to see if the object touches or resides in the selection rectangle.

However, the above seems very complex for such a simple feature, not to mention slow, as the calculations would have to be done every time the mouse moves. I guess the convex hulls could be cached until the camera moves, but that only complicates the system further.

I can already perform ray-to-mesh intersection tests perfectly, the problem comes when I want to perform tests with something "fatter" than just a ray...

So, how would you do this?

I've wondered about this myself, from the brief searches online, it looks like most people do it the way you describe, bringing everything to 2d space and doing it there, except usually they use an aabb instead.

I guess the other option would be to create a frustum out of the points of the square being created and then do a frustum check (or maybe a rectangular cube?) against the objects' aabb.

To do this, I would use the screen-space selection rectangle to generate 4 planes that bound the outside of the selection in world space (normals point toward the inside), plus one more plane for the near plane. This is a lot like building a frustum for culling. Then, the objects in the selection are the objects that have some part in front of all 5 planes.

At the coarsest level, you could do this with the AABBs of all objects for speed. Then you do the same test with the vertices on that reduced set and can throw out any objects that have no vertices or triangles in front of all planes.

Nanoha, that won't work because I need to be able to select occluded objects. It is quite common in an editor to have a "pile" of objects that you want to select by drawing a rectangle around the whole pile. Other than that, it's a neat way to handle the problem. I am doing picking through ray casting which allows me to select all objects along a ray (which is important for some stuff I am doing) but I have used the technique you mentioned for generating a visible-object-ID-map for occlusion culling in a tile based dungeon crawler I worked on.

If you pick your location in 3d space using a ray at your initial "click" location and than again at your "release" location, you can using the x, z values to form a 2d box(Directx). If an objects draw location is within those coordinates, select. No need to worry about the y coordinate.

Edit: This will also select your partial object picks as long as their draw location is on the inside of the rectangle. No reason to use color id in a shader blah blah blah, unless their is a specific reason you need to have collision perfect object selection which I wouldn't understand why in an editing environment.

Multiple way possibles :

- Ray intersection using a spacing value for each ray into the rectangle.

- Convert the AABB of each object in screen space and test this rectangle with the selection rectangle.