One of the easiest ways is to store the hit regions of your icons as 1-bit (i.e. black and white) bitmap images. Then you can supply arbitrary shapes as images, and the hit test is as simple as "is the pixel under the mouse white?".

This is the route I'd take, but I'd use the button's alpha channel and check for transparency instead of having a separate bit mask (or, if the texture is in GPU memory, generate the non-GPU bitmask using that channel).

You can even use a 3D object as a button and use it's mesh to check the bounds.

yes i understand that its easier to implement as a rectangle for easy functionality with the mouse coordinates,but its is possible to make a star button and been able to click it without it being a rectangle point checking.

Here's my experience of that.

I tried to implement an arrow-shaped 2D button, to scroll the camera over a side-perspective landscape. I wanted the button to be an arrow to visually indicate right or left. And the detection of whether the user clicked the arrow-shaped button was programmed as a triangle.
- But when I playtested this, it turned out to be difficult as a user to click on the arrow, because it becomes very thin near its points/tips/vertices. When testing, I moved the mouse cursor towards the arrow clicked, and realized I clicked a bit outside, it felt frustrating.
- So I simplified the implementation, making it look like an arrow visually, but detect the user click by programming the arrow as a rectangle. That was a win-win solution, it became easier for the user to click, and it became easier to program (rectangle detection instead of triangle/polygonal)

The conclusion is, keep in mind that we (= or i) move the mouse right/left, or up down, maybe because the screen is rectangular and we see the mouse cursor move in relation to the screen edges (rectangle). So it's slightly harder to move the mouse diagonally with precision. That's how it feels for me at least, when using a computer mouse.