Got it solved, appearantly I wasn't setting the bits right, here's the solution:

		//makeKey = ((0xFF & tr.MatGroup) << 40)	| ((0xFF & tr.Material) << 32)	| ((0xFF & tr.Mesh) << 24)			| ((0xFF & tr.Instance) << 8)		| (0xFF & tr.DistToCamConv);
		makeKey |= (UINT64)tr.MatGroup	<< 40;
		makeKey |= (UINT64)tr.Material	<< 32;
		makeKey |= (UINT64)tr.Mesh		<< 24;
		makeKey |= (UINT64)tr.DistToCamConv << 16;
		makeKey |= (UINT64)tr.Instance;