You have 3 choices to make this a bit more simple
1) build the matrix straight from the camera. This will always generally towards the camera, but for anything large you'll notice that it rotates relative to other objects as you rotate the camera, however it is good for things like text that you want to show up "ortho normal" to the camera anyway so they are legible. They won't rotate as the camera translates, which can also be a nice property.
[ Camera.left, Camera.up, -Camera.at, obj.pos], as that will always have the same orientation as your camera and always be facing the opposite direction.
2) build the matrix using what is normally done with a library's "look at" matrix function. This will make your billboard behave like it is on a sphere centered at your camera, always facing the center. They won't rotate when you rotate the camera, and won't be as good for text but better for particles. They will however rotate as you translate your camera, so moving by one will make it seem to rotate around you.
at = normalize( camera.pos - obj.pos );
left = cross(camera.up, at)
up = cross(at, left)
"look at" matrix = [left, at, up, obj.pos]
3) if you want the object pinned on an axis (like a tree, that rotates on the Y axis only) then use technique 2, but use a constant for the "up" axis instead of calculating it (then ortho-normalize based on that axis instead of the 'at' like I showed).