In case anyone is interested, I solved this problem. I use linear programming to optimize for the minimum sum of the absolute values of the difference between "attained force / torque this frame (per dimension)", and "required force / toque this frame", where required force this frame is the force required to get the ship to a set target velocity / rotational velocity. The variables are the percentage of each thruster; the coefficients are the force and torque the thruster has in each dimension. The thrusters are constrained between 0 and 1.

This page has a really great explanation on how to minimize absolute value even though it itself is not linear: http://lpsolve.sourceforge.net/5.5/absolute.htm

Compiling LPSolve library with VC++14 worked fine.

If anyone has questions about the details of this feel free to ask.