One alternative I can think of:
- Consider D to be a hinge.
- Place a new rect A'B'C'D' and point H' directly adjacent to DEFG and treat this as being a rotated version of ABCD.
- You now know the positions of both H' and H. Calculate the angle formed between them using D as the common vertex.
- Use that angle to rotate points A' through C' about D, back to their expected positions.
You could probably do this entirely with matrices and never actually measure an angle (I suspect), but I'm not good enough with linear algebra to verify that.