float angle = std::acos(glm::dot(direction, tocam));

A dot product can be used to find the angle between 2 vectors BUT it will not give you direction(clockwise/anti-clockwise). For example, if something is 45 degrees to the right of me and another thing is 45 degrees to the left of me then using a dot product to work out the angle will give me 45 degrees for both objects (rather than 45 and -45(or 315). I suspect this is causing the problems you describe.

I don't know what the usual 'done' thing is for this but instead of rotating you could create an orthogonal basis. You know the direction to the camera, that will be your 'forward' vector, cross that with your world 'up' to get a 'right' vector. Cross the 'right' and the 'forward' to get the object's new 'up' vector. Now you have 3 orthogonal vectors you can create your basis (which is, in effect the rotation matrix that you need). Those 3 vectors will be rows in your rotation matrix. Assuming 'y' is up then row0 will be 'right', row1 will be 'up' and row2 will be 'forward'. Column 3 and row 3 should be all 0 except for R[3][3] which should be 1 (load it as an identity then replace the top 3x3 values with your 3 vectors).

Do you want these things leaning over towards the player? Good for some things but not so good for trees etc.

You could extract modelview matix of your camera amd get screen coord up and right vectors in notime

I saw something similar in the other billboard thread that's floating around. This certainly sounds like a far more efficient solution. With this approach I'm guessing all the billboards/particles will then be perpendicular to the screen (rather than facing the camera position)? This may or may not be an issue but it sounds like it's worth trying and seeing what it looks like as it is far more efficient than all those cross products per particle.

So like the following?

What you have there is transforming your world axis into view space which isn't quite what you need. I'm not 100% sure about this myself as I've never had to do it but what you want is the first 3 rows of that view matrix (the right, up and forward vectors). I'm not sure what the syntax is to get that with glm so I can't show an example. Once you have those they will be the opposite of what you want, you should be able to flip the 'right' and 'forward' vectors which should work the same as rotating about the 'up' axis by 180 degrees.

Another option might be to do similar to what you are currently doing and define those vectors in view space (so they are looking at the camera, right(-1, 0, 0), up (0, 1, 0), forward (0, 0, -1)) and then transform them back into world space by multiplying them by the inverse of your view matrix.

@Nanoha, do u mean flip the forward and right vectors or flip what the vector is (.x becomes .z)

Sorry for being unclear, I mean (x, y, z) becomes (-x, -y, -z). I guess I should have said mirror the vector. But only for the right and forward vectors.

	glm::mat4 viewInv = glm::inverse(view);
        
        // You might actually be able to rip this stuff right out of the matrix rather than multiplying
	glm::vec4 right = viewInv * glm::vec4(-1, 0, 0, 0);
	glm::vec4 up = viewInv * glm::vec4(0, 1, 0, 0);
	glm::vec4 forward = viewInv * glm::vec4(0, 0, -1, 0);

	glm::mat4 mytest = glm::mat4(right, up, forward, glm::vec4(0, 0, 0, 1));

The reason that 'right' is mirrored is because the particles right is your left. Forward is mirrored for similar reasons, particles forward is my backwards. You will have to check forward though because with OpenGL I think forward might actually be (0, 0, -1). It depends on the coordinate system. Up remains the same because the particle's up is also the viewer's up.

Your best bet is to look for a tutorial that is dealing with this stuff.