I'm working on a DOOM-esque game in opengl for 7dfps and I need to make the billboarded pickups and enemies. Is there somewhere which documents the 'correct' way to rotate the quads and to choose which sprite of the enemy to show? If not, can someone tell me if there's anything wrong with the algorithms I've come up with?

For billboards:

I have made the billboard perpendicular to the player's view angle. I also tried making it perpendicular to a line pointing at the player's position, but it looked worse.

For enemies:

radiansToSide=PI*2/8 (8 is the number of sprites I have for enemy)

angle=atan2(enemy.y-player.y,enemy.x-player.x)

angle+=radiansToside/2

angle/=PI*2

angle*=8

angle=floor(angle)

//angle is now an int 0-7

Close but you are not taking into account the direction the enemy is facing. You will need to transform the players position into the enemy's local coordinate system and then apply your algorithm.