While boxing/unboxing and all sort of interoperations between int and Integer, or between Java types and native code types (IntBuffer, FloatBuffer, etc) are covered, they have their quirks here and there (specially dealing with collections, sometimes you have to be aware of when the boxing/unboxing is happening), and they still work pretty fine nevertheless.
Thing is, like String, Integer, Float, etc are immutable objects. So one of the worst offenders (performance wise) you could make, which is appending massive quantity of String objects (instead of mutable StringBuilders), could start happening with Integers. Every time you operate on an integer (addition, subtraction, everything) you'd be creating massive amount of new objects.
Probably JVM devs have some way to deal with that stuff with good performance anyway, the thing is that like String, it will probably end in the list of "Things to watch out for" that a programmer has to have in mind.
"Why my array averaging method is so slow!?"
"Well, you're using an Integer instead of IntegerBuilder. Use an IntegerBuilder and refactor those 30 methods that indirectly depend on Integer's immutability so it doesn't crash everything"