So the sorting function can be as simple as this:
struct RenderInstance
{
u64 sortKey;
Command* pCommands;
};
bool compare(RenderInstance i, RenderInstance j)
{
return (i.sortKey < j.sortKey);
}
void sortInstances(std::vector<RenderInstance> instances)
{
sort(instances.begin(), instances.end(), compare);
}
The order you put the bits inside the key will change how it'll be sorted.
Example:
You'll most likely want sort entity by translucency (so you render opaque objects first and then transparent) and then only then sort the two groups by depth.
So since the translucency is more important you store it in most significant bits.
The sort key would look like this:
| translucency (1 bit) | depth (24 bits) |
So whatever the depth of each entity is you can be sure that opaque entities will be separated from transparent entities because most significant bits have more influence in the sort key.
Long story short, think carefully what is more important and build the sort key accordingly.
P.S. Keep in mind that depth is usually a float value so you can't store it directly in 24 bits without any conversion, or without having a more complex sorting function.