One possible issue I see might be the order of operations of your relative vector calculation. If the center of your hit box is calculated as (rect.position + rect.size/2) then what you are actually doing in your call to atan2 is not correct. Consider the case of a hit box at (1,1) and sized (2,6). The center of the hitbox will be at (2,4). Now, consider a mouse location at, say, (4,4). Logically, the vector (mouse.position-hitbox.center) would be (2,0). But the vector you are calculating with (mouseY-hitBox.y+(hitbox.h/2), mouseX-hitBox.x+(hitbox.w/2) is not equal to (2,0). It's equal to (4-1+1, 4-1+3), or (4,6). You see? It's the order of operations. What you want is (A-(B+C)) but what you are doing is (A-B+C), which is not the same.

So try parenthesizing to get the correct order of operations atan2(mouseY-(hitbox.y+hitbox.h/2), mouseX-(hitbox.x+hitbox.w/2)) and see if that makes any difference.