Assuming you are using this as a means of encryption you have a few problems. The biggest one being that integers are not closed under division. Overall you are just mapping vectors to vectors. So basically (x,y) maps to ((x+y)/2,(y-x)/2). I don't believe there is any general rounding rule that will cause (1,1), (1,2), (2,1) and (2,2) to map to four distinct pairs. The problem is that division and the rounding it requires. You could map (x,y) to (x+y,x-y) instead and not have that problem. Your inverse function is then (x',y') maps to ((x'+y')/2,(x'-y')/2). There you don't have rounding because x' and y' are either always both even or both odd. Specifically if x and y are both odd or both even then x+y is even and x-y is even. If one is odd and one is even then x+y is odd and x-y is odd. Another problem in your example is that you are actually using (x,y) maps to ((x+y)/2,|x-y|/2). That makes (x,y) and (y,x) map to the same pair because of the absolute value taken, i.e. x+y=y+x and |x-y|=|y-x|.

I don't understand at all why you are having trouble finding the coefficents. The only thing I can think is that because you don't have a bijection and thus no inverse you are having trouble reconstructing the original stream when you test. The problem isn't how you calculate your coefficients, but rather the definition of the coefficients itself.