^This. In Java is very, very easy allocate thousands of new objects without noticing (a common example is making tons of String objects). Allocation time by itself wont be noticeable (unless you're allocating many complex objects, it those cases it is recommended that you reuse such objects where makes sense), but the allocation time + garbage collection time can build up and be a problem eventually.
Though I'm not sure if he meant a wrapper coded by him or the boxed Double
By looking around I'm inclined to say that Java has earned its reputation of being "slow" because is very easy and convenient to allocate new objects around at every single line of code, inside loops (worst), at the "return" of all methods (not so bad), etc.