It is possible to do this per pixel, but it may not be generally feasible.

Since a pixel shader strictly does not allow writing of values to pixels other than the current one, you have to invert the necessary scatter algorithm to a gather algorithm.

The sampling logic of the pixel shader would therefore need to perform the inverse of the non-linear weighted rotation operation in order to arrive at the correct sampling point (source pixel) given the current pixel position (destination). This is very difficult to derive to begin with, and it wouldn't probably be very performant on most GPU:s. And the more general the transformation would be, the more difficult it would be to derive the sample points.

The common approach to drawing this type of stuff is to tessellate the original rectangle (split it to many smaller pieces) and perform the weighted rotation operation on the smaller primitives. This way, you can apply the weighted rotation directly to the drawn primitives without inverting the operation for sampling.

(EDIT: "pixel shader" (d3d term) and "fragment shader" (ogl term) mean the same thing)

Are you sure you want to use geometry shaders? Perhaps not all of your customers have GPU that support it.

Tessellation, by itself, does not imply 3D geometry or even the usage of GPU to perform it. It is simply the act of dividing any geometry to smaller primitives, usually done in order to introduce more geometric fidelity as compared to the original primitives. It is necessary here because you can't really bend straight edges of a single triangle (or the triangle itself) - but you can approximate the bending by dividing the edges (and therefore triangles) to smaller segments.

You could construct the tessellated geometry on the CPU and just run the rotation against the resulting grid as a vertex shader.

The rotation logic is relatively simple to implement as a VS program: assign the desired weight of rotation for each vertex, and interpolate between the unrotated and rotated position by that weight. The unweighted rotation itself is a simple linear transformation.

Should you have problems in writing the vertex shader, first try to make a vertex shader that unconditionally rotates all vertices given an angle. After that works, it should be easy to see how to apply the per vertex weighting to the equation.

---

Geometry shaders are not very suitable for large-scale tessellation, because the amount of data that can be emitted from a single GS invocation is relatively limited. Newer hardware units have a dedicated, highly configurable tessellator engine which is a little bit less flexible than GS but can output much greater amounts of geometry. But as said, using GPU is not mandatory for implementing tessellation; it merely makes the operation more efficient.