Hello all, I have a problem with my 2D isometric picking.
I've asked on gamedev.stackexchange too, but so far I have no answer. You can read the stackexchange question here. It contains details. I will write a short version here.

Here's the code I have for drawing the tiles

// Draw code
int col = 0;
int row = 0;
for (int i = 0; i < nrOfTiles; ++i)
{
// XOffset and YOffset are currently hardcoded values, but will represent camera offset combined with HUD offset
Point tile = IsoToScreen(col, row, TileWidth / 2, TileHeight / 2, XOffset, YOffset);
int x = tile.X;
int y = tile.Y;
spriteBatch.Draw(_tiles, new Rectangle(tile.X, tile.Y, TileWidth, TileHeight), Color.White);
col++;
if (col >= Columns) // Columns is the number of tiles in a single row
{
col = 0;
row++;
}
}
// Get selection overlay location (removed check if selection exists for simplicity sake)
Point tile = IsoToScreen(_selectedTile.X, _selectedTile.Y, TileWidth / 2, TileHeight / 2, XOffset, YOffset);
spriteBatch.Draw(_selectionTexture, new Rectangle(tile.X, tile.Y, TileWidth, TileHeight), Color.White);
// End of draw code
public Point IsoToScreen(int isoX, int isoY, int widthHalf, int heightHalf, int xOffset, int yOffset)
{
Point newPoint = new Point();
newPoint.X = widthHalf * (isoX + isoY) + xOffset;
newPoint.Y = heightHalf * (-isoX + isoY) + yOffset;
return newPoint;
}



And here is the code for getting which tile is the mouse on

public Point? ScreenToIso(int screenX, int screenY, int tileHeight, int offsetX, int offsetY)
{
Point? newPoint = null;
int nX = -1;
int nY = -1;
int tX = screenX - offsetX;
int tY = screenY - offsetY;
nX = -(tY - tX / 2) / tileHeight;
nY = (tY + tX / 2) / tileHeight;
newPoint = new Point(nX, nY);
return newPoint;
}


This code is close, but not even close enough for a public beta or something like that (even though this is just a proof of concept, which I will later transform into my game). I have no idea why this code is so close. I drew the tiles by thinking up the formula, and by googling I believe I've seen that it is correct (however it's still possible it's not). The code for picking - what ever I came up with, didn't work. What ever I googled, didn't work. This code I currently have is a derivation of something I googled - it was not really close, but closer than others, so I modified it and it's even closer now - I did it by trial and error as I have no idea behaved and behaves the way it did/does.

Can anyone point out to me my error? Whenever I think about it, it doesn't make sense to me that it behaves that way. I tried reversing the formula for drawing and it didn't pan out which doesn't make sense either. I must be doing something wrong then so please, if someone sees my error, could you point it out?

The actual answer depends on a few things but one thing I can definitely tell you is that there shouldn't be any maths happening between the x and y mouse coordinates. Anyway, this is the way I remember that I did it. It probably wasn't the most clever or elegant but it was simple and I like simple.

You'll want to divide the calculated cursor position by the tileHeight/Width and ROUND to the nearest number. This will need to be done with a double or float because ints automatically truncate (round) up or down depending on their sign (negative/positive). Up for negative and down for positive values. You can avoid changing things via casting to int after calculating using a double or float variable with an integer.

If the tiles have moved, the tile map is bigger than what the screen is capable of displaying, you'll need to calculate the map position of the screenspace 0, 0 tile and offset the values to account for it.

This wouldn't help with transparency (or overlapping tiles), where a much different approach would need to be taken, and would only work if all tiles are the same size which it looks like they are from the source code provided.

Hope that helped.

These are my equations for the ISO-conversions (i use a hard-coded slope of 4:1 pixels (x:y)):
ISO_X,ISO_Z are the left-bottom offset of the iso space


void World2ISO(int wx, int wz, int wh, int& isox, int& isoy)
{
isox = wx - wz + ISO_X;
isoy = (-(wx + wz) / 4) - wh + ISO_Z;
}

void ISO2World(int isox, int isoy, int& wx, int& wz, int& wh)
{
wh = 0;
wx = (isox/2) - (2*isoy);
wz = -(isox/2) - (2*isoy);
}



Maybe that helps you a bit

@MachinaX
What do you mean by "there shouldn't be any maths happening between the x and y mouse coordinates"?
What you're saying is this?

public Point? ScreenToIso(int screenX, int screenY, int tileHeight, int offsetX, int offsetY)
{
Point? newPoint = null;
int nX = -1;
int nY = -1;
int tX = screenX - offsetX;
int tY = screenY - offsetY;
nX = Math.Round((double)tX / (double)tileHeight / (double)tileWidth);
nX = Math.Round((double)tY / (double)tileHeight / (double)tileWidth);
newPoint = new Point(nX, nY);
return newPoint;
}


If this code is what you're saying, then it doesn't work (I mean, after fixing small, insignificant for this, errors like nX being in while Math.Round returns decimal). It selects something way further than under mouse cursor (out of screen). But the code doesn't make sense to me either - why would I divide with tileHeight/tileWidth?
I'm also not sure what you mean by the sentence "If the tiles have moved,...", do you mean the -offset I have in tX and tY variables?
As for the transparent parts, I'll implement a color map for that, but first I want to get to a stage where the rest actually works.

@AticAtac
I couldn't get your code to draw properly so I didn't try your picking code. I'm probably missing something, again... But I'm at work now so not much time to play with this. When I get home I'll play around more a bit.

Thank you to you both for your contributions.

The matrix solution mentioned here is essentially what you want to use to generate the correct formulas. If you plug your tile size into a 2x2 matrix representing your isometric basis, you'll get this:

| TileWidth / 2, TileWidth / 2 |
| -TileHeight / 2, TileHeight / 2 |


I was able to infer the transformation from your IsoToScreen code, since this math:

newPoint.X = widthHalf * (isoX + isoY) + xOffset;
newPoint.Y = heightHalf * (-isoX + isoY) + yOffset;


Is equivalent to:

| widthHalf, widthHalf | * | isoX | + | xOffset |
| -heightHalf, heightHalf | | isoY | | yOffset |


The inverse of that basis matrix is, if I did my mental math correctly:

| 1 / TileWidth, 1 / -TileHeight |
| 1 / TileWidth, 1 / TileHeight |


Which gives us the equations:

nX = (int)((double)(tX / TileWidth) - (double)(tY / TileHeight));
nY = (int)((double)(tX / TileWidth) + (double)(tY / TileHeight));


Might need a little tweaking, but that should be the general form.

Thanks for the math!
However it's not working. It selectes only the tile that would be 0,0 in the original system (the non rotated, top view) and it does so for about width/2 to the left and height/2 above. This made me realize though that I failed to mention my coordinate system (XNA) has it's (0,0) at the top left, not the bottom left of the window.

Yesterday I've tried thinking up some formulas, but to no avail. I was sure this would be correct, but it doesn't work for some reason... I'll have to try again after work..


x = mX / TileWidth + (mY / TileHeight / 2 - maxX / 2) * TileWidth
y = mY / TileHeight + (mX / TileWidth / 2 - maxY / 2) * TileHeight


where
mX is the mouse X position, offsetted by the X drawing offset
mX is the mouse Y position, offsetted by the Y drawing offset
maxX is the number of virtual tiles on X axis
maxY is the number of virtual tiles on Y axis

what virtual tiles means is, the bounding rectangle of a drawn tile, divided by 2, by an axis is in the non-rotated top view, not the isometric view.

Ofcourse, it should work only for that virtual tile and then I could use a color map to determine which exact tile is selected, but even without a color map, I can see this does not work.

Yeah, I moved some stuff around and my determinant had the wrong sign. I edited my original post and fixed the equations

I've also posted some code which demonstrates the math. It will visualize an isometric grid and show you which tile is under the mouse (just pass it a graphics context to draw and a mouse position to update the mouseover). Tile sizes are in pixels, so you will want to make them larger!