Got it solved, appearantly I wasn't setting the bits right, here's the solution:
//makeKey = ((0xFF & tr.MatGroup) << 40) | ((0xFF & tr.Material) << 32) | ((0xFF & tr.Mesh) << 24) | ((0xFF & tr.Instance) << 8) | (0xFF & tr.DistToCamConv);
makeKey |= (UINT64)tr.MatGroup << 40;
makeKey |= (UINT64)tr.Material << 32;
makeKey |= (UINT64)tr.Mesh << 24;
makeKey |= (UINT64)tr.DistToCamConv << 16;
makeKey |= (UINT64)tr.Instance;