I think you need a system for directly moving workers from x to y, while allowing multiple x or multiple y to be selected (so you can lets say select to take from "idlers" and move to all the others)

If you only drag a single bar, it takes from all the others (or puts to all the others)

I would make it so that you can select the "from bars" by clicking them, then dragging to each "to" bar from any of the selected "from" bars.

So if you want to move from "idlers" and "slackers" to "slaves" and "minions", you will click the first 2, then drag from either idlers or slackers to both minions and slaves.

There would be a "master" slider which you can then use to evenly move workers between the above groups. You should also be able to still use the individual sliders without deselecting your "from" and "to" selections. eg. if you were to drag ONLY idlers, it would go to both slaves and minions while slackers would stay unchanged.