Well, I basically sat down and considered carefully the four possible cases (in each of the four quadrants) and figured out that if the player was on the right half (π/2 < θ < -π/2) of the field, rx would be -rcosθ and ry would be -rsinθ, where r is the player's resultant speed (playerspeed for horizontal/vertical movement, and sqrt2*playerspeed for diagonal). Then, on the left side, rx = rcos(θ+π), ry = rsin(θ+π). What I was doing wrong before was, as you can see from the bit of code in my first post, not computing r at all, and using tan, as well as getting Newton's Third Law wrong. Now that I got my maths(and physics) figured out, it works exactly as I wanted it to.
Also, the jagged movement was due to me forgetting to add in the code to restrict the player's position. So the player would move outside the boundary momentarily before being pushed back in by the restoring force.
In conclusion, I was not thinking straight at all before, and adopted a totally illogical approach =\ I still feel kinda dumb even now D: