Looks like a cut down version of stencil shadow volumes. Where it draws the shadows quads to the stencil buffer, and then the lit things are drawn except for the pixels marked by the stencil buffer. Pretty neat idea, looks like it would work for any N sided polygon.

You can't do it in a vertex shader since you can only access one vertex at a time, where as that algorithm needs access to two. Also I think vertex shaders can spit out only one vertex at a time, where that spits out 4. If you had geometry shaders you could do it there but that's opengl 3.3+.

Edit:

Actually you could pass more than one vertex to the shader by doing something like

for verts in quad or triangle {

glVertex4f(cur.x,cur.y,next.x,next.y);

}

But I don't know if you are able to emit more than 1 vertex from the shader. If you could, it might be possible...

Edit2:

Actually I just realised another idea, where you send a quad for each edge,

e.g.

for each edge {

glBegin(GL_QUAD);

glNormal(edgeNor.x,edgeNor.y,0);

glVertex3f(cur.x,cur.y,0);//0 represents the vertex on the edge to not extrude

glNormal(edgeNor.x,edgeNor.y,0);

glVertex3f(next.x,next.y,0); //0 represents the vertex on the edge to not extrude

glNormal(edgeNor.x,edgeNor.y,0);

glVertex3f(cur.x,cur.y,1); //1 represents the extruded vertex

glNormal(edgeNor.x,edgeNor.y,0);

glVertex3f(next.x,next.y,1);//1 represents the extruded vertex

glEnd();

}

Basically in the vertex shader you:

1. redundently do the dot product check to see if a shadow should be extruded. Pass a varying float to there fragment shader, where if shadow pass 1, else pass 0

2. If the vertex.z ==1, extrude the vertex along the light direction.

Fragment shader:

1. Check the varying float passed from the vertex shader, if == 0 then call discard to abort drawing the shadow

Kind of convoluted, but it should/could work.