I think I will go for general competence (no specializations) and individual loyalty.

But maybe add a sort of virtues/beliefs to each courtier? Like Military, Science, Finances, Security, Intelligence, Diplomacy, Justice. This would represent what this person beliefs is the most important thing in the empire. Each courtier would add +1 of his belief to the empire stats, ministers add +3 (or +5?). It is not connected to ministry at all, minister of Fincances could be pro military and minister of War pro Justice. They all add to the overall empire stats.

This would mean, sometimes you might not appoint your best people (comptent & loyal) because for example you need them all to be pro military since you are preparing for a big war.