I apologize if this is kind of all over the place. I have a 3 month old so I had to write this in bits and pieces. So, I've got a terrain that I'm happy with getting displayed okay. It has no issues drawing so far, but once I populate the map with objects and characters, I imagine that I'll have issues. Plus, I would really like to use a QuadTree for picking purposes.
I have a few questions. But first, a bit about my map. It's a tile map essentially, just placing them in strategic positions so they look good after I do my MVP transformations. However, the tiles themselves exist entirely in the xy-plane. I put walls in the z-plane, but I don't care about being able to pick them (and they'll get drawn with their tile, no matter what). Currently, I have a giant VBO and giant Index array that stores all of these. I rotate the model matrix, then setup my lookat(view) and projection matrix and I'm good. I have a specific camera matrix that goes between the world matrix and projection*view matrix that I translate in order to give me camera control. I plan on keeping the one VBO, and the leaf nodes will contain the indexes for the tiles that it contains, and once we decide which ones to not cull, we put together a master IBO with only those indexes.
So, the first question is, what should I use as the coordinates to sort my quadtree around? My first thought is to just use the xy coordinates that I initially laid them all out on. However, that means any time I want to do picking or culling, I have to translate from screen space all the way to model space. This seems pretty inefficient as I'd like to be able to highlight the tile that the mouse is currently over, which would require a WHOLE LOT of translation calculations. I can't think of a better way to do it however, would this work?
Second question, how do I re-evaluate what to cull. My thinking is, I'll decide what is on the screen currently, then draw a bit more than that (maybe another quarter of that size in each direction) so I don't have to change it all every frame if they're moving in 2 directions at once. However, what's a good method for deciding when to do that evaluation? How do I figure out what my screen is currently looking? Do I just look at the model coordinates of each corner and if my tiles are in between those coordinates, I draw them (plus a few in each direction?). This seems to have the same problem as the mouse issue above.
Third question, when picking, if a leaf node of the quadtree contains multiple tiles, how do I determine which tile was clicked on? It's pretty obvious if they all contain a single tile, otherwise not so much. Anyway, I'm tired and want to get back to working on this stuff. Any help would be greatly appreciated. Thanks.