I am trying to decide what is the best way to do picking in OpenGL with glOrtho2D mode on a tile based game. Would using the color id method be better or would doing a ray test against each tile and sprite having an AABB around it?

Please advise and if possible give an example of code to do the picking based on your choice.

Thanks!

For a simple 2D game it could be as simple as getting the mouse's current position on-screen, and from there go through your game objects and see which one matches that location.

How is that going to work when I can move the map off screen?

e.g.

map size is 2048x2048 screen is 1024x768

I move the screen to pan up/down/left/right to see more of the map. The screen coordinates aren't going to match the absolute position of the object on the map...

So far I am thinking the color method is the best and simplest way to go about this... with rgb and 256 colors per channel you have what... ~16.7million colors available for units?

How is that going to work when I can move the map off screen?

Because you need to calculate what portion of the map to display after all that panning (usually called the view transformation), you can do the inverse calculation to relate the screen origin back to the reference for the map. Its just a generic answer, I know, but you haven't provided enough details for a more detailed answer.

So far I am thinking the color method is the best and simplest way to go about this... with rgb and 256 colors per channel you have what... ~16.7million colors available for units?

Yes, 24 bits give you 2^24 possible values. However, you need to use another rendering pass. You have to ensure pixel perfect masks, e.g. no AA that blends your color masks. You have to ensure that texels with alpha below 0.5 (i.e. more transparent than opaque) will be discarded, so that transparent texels do not disturb the color mask. And you need to read back the render target to get access on the CPU side. So, after all, the algorithm is probably still less sophisticated, but it does not come for free, and whether testing bounding volumes or looking at pixels is more performant in the end would need to be investigated further.