For transparency rendering you're supposed to sort your objects by depth, however I'm having trouble finding a good method of choosing each object's depth. Just using the position doesn't work for objects that aren't centered, getting the center of the bounding box doesn't work much better. I also tied getting the smallest depth from all six corners of the bounding box, but even that has problems. Has anyone come up with a good way to do this?

If your target hardware is powerful enough, you can get all fancy with order-independent transparency.

If not, then you are stuck with finding a best estimate. What are you trying to render here? Techniques will vary depending on whether you are drawing camera-aligned particles, cut-away models, or something else...

As mrjones says, sorting individual polygons by their centre is a possibility, although likely not feasible due to the sheer number of polygons, and the resulting lack of batching.

I'm doing a high end shader based engine and also a software renderer. Just trying to render regular models, no particles or anything fancy, and not worried about transparency within a single object yet.

Despite every 3D game having to solve this problem, it's still an unsolved problem

Using the centre of the object, it's farthest point, or it's closest point are common solutions.

Sometimes you'll use different methods for different objects -- e.g. if the player is inside a 1km glass dome, you want to use it's farthest point (because that's a better representation of where it's surfaces actually are), but maybe for other objects you want to use their nearest point.

If its a deferred for transparent objects you can use interlace, deinterlace method (Used for toy story). But it also have some limitation yet no needs for sorting.