Trying to cram everything into a 64-bit value solves nothing and creates problems. You are limited in your bits and comparisons are 3 times slower.
So perhaps you should explain why you feel the need to go this route and then perhaps we can give you a better alternative.
I totally see that it creates a lot of problems storing everything in a 64 bit value.
Woah I didn't know that it's 3 times slower to compare a 64 bit value. So I thought that it was as fast 32-bit compare on 64 bit computers.
The thing is that I don't feel I need to go this route cramming everything into 64-bit it was just that it was the first idea that came to me and I tried to stick to that idea. Im quiet open to any idea. Initially I wanted some input so that I could avoid the worst bottlenecks and design decisions.
I really like the idea that Hodgman gave me to just treat the sorting key as a binary blob that is only used for sorting. That's something I would never have thought of if it wasn't for this thread.
Coming up with a "perfect key" is impossible to do I know it now.
It's almost like an AI that needed to create these keys. Has someone tried building something like a neural network to create keys?
The inputs are shaders, uniforms, textures, vbos, ibos, z-depth, transparent and some other stuff. Output is a key. Training the network would be the draw calls or frame time.
I think it would be slow but will it be slower than the all the unnecessary render state changes?
Typically, my items will have state-groups such as per-camera (view/proj matrix), per-object (world matrix), per material (textures, shader), etc...
Wouldn't this create a bunch a of memory fragmentation? Or create a lot of dynamic dispatch?
I know writing branch free code is really hard so I guess that's ok.