Is this for a special effect or are you trying to gain performance? In the latter case, it is likely not worth the hassle, as changing the relevant render states is relatively slow and you might better off just antialiasing everything. This depends on the target system bottlenecks, though.

If you want to generally mask some rendering by using arbitrary shapes, read up on stencil buffers. Using them, you would draw the star shape to the stencil buffer (to form the mask), and then setting the render states for the subsequent rendering to either pass or reject pixels based on the stencil. In this case, you'd do the antialiased rendering with the stencil configuration that would pass only the pixels within the mask. It is also possible to have more than one bit of stencil data per pixel, for multi-layered pass/reject logics.

If I remember correctly, you can control antialiasing during a frame by using glEnable, glDisable and GL_MULTISAMPLE. Enabling it requires that you have a rendering context that supports antialiasing to begin with.