I am trying to implement accurate selection tool for my level editor that would be able to use both isometric tiles and isometric hexagons. with the same sprite dimensions. I am now focusing on getting regular isometric tile to work properly.

Below is the color coded isometric tile that i will be using (it will be transparent with only lines visible, this one is for explanation only). I want my program to register specific tile if the cursor is only in green area.

The sprite has dimensions of 192x96. How do I calculate if the mouse is only in green area? Since from what I find everything is based on rectangle logic, because I check X, and Y coordinates of the mouse.

From here I have something like

if(mouse.x > 0 && mouse.y < 48)

if(mouse.x < 96 && mouse.y < 48)

select tile

but these coordinates also catch stuff from red triangle area in top left.

Another common technique is to store a copy of that image above (simplified to save memory)

Then to test for a click on a tile you build a list of tiles it could be (usually only two) convert the coordinate into a relative one for the tile and pick from the stored image.

If the colour you pick up is green, you have the tile.

I think it would probably be easier to do a ray polygon check, but this is the way we used to do it when floating point was expensive.

have a look at http://content.gpwiki.org/OpenGL_Selection_Using_Unique_Color_IDs

This is all completely 2D stuff. What I did is, I created grid using one sprite and simply placing it on calculated X and Y coordinates to form a grid.

I am looking into point in polygon stuff, but I am not sure how to do this yet. This is what I am testing with

// triangle to test against
sf::ConvexShape hex;
hex.setPointCount(3);
hex.setPoint(0, sf::Vector2f(0, 0));
hex.setPoint(1, sf::Vector2f(96, 0));
hex.setPoint(2, sf::Vector2f(0, 48));


// check coordinates if inside the triangle
if(Mouse.x >= hex.getPoint(0).x && Mouse.y >= hex.getPoint(0).y)
  if(Mouse.x <= hex.getPoint(1).x && Mouse.y >= hex.getPoint(1).y)
    if(Mouse.x >= hex.getPoint(2).x && Mouse.y <= hex.getPoint(2).y)
    {
	diamond.setPosition(0, 0); // this is tile marker - it will highlight the tile mouse pointer hovers over
    }

Something is off since I am still in rectangle calculation and not triangle. Am I on the right track?

EDIT: I found this - http://www.geeksforgeeks.org/how-to-check-if-a-given-point-lies-inside-a-polygon/ - looks close to the first reply, and this does answer my question. Thanks for help everyone.

const int tile_x = x/192 + y/96;
const int tile_y = y/96 - x/192;

const int tile_x = round(x/192.0 + y/96.0);
const int tile_y = round(y/96.0 - x/192.0);

Or just base your math on the simple idea that your map is a rectangular grid rotated by 45° and 60°, transform your mouse coordinates accordingly and you should get the same result

Wouldn't this actually be more complicated? I would need to create regular greed and rotate it 45 degrees, and than additionally transform mouse coordinates. Seems interesting but I am not quite sure how I would handle data in this case. Would I save out all data as if it was square greed and transform it on run-time?