That's for sure. Cube-based, axis aligned worlds are pretty in vogue nowadays but bullet has no way to exploit this very strong property. The result might be suboptimal. Overall, I'd still consider physics APIs anyway.
Time to learn Bullet (or another physics API) may take few hours more than to code a simple box-sphere collision check, but at the end you'll have access to lots of features (which you can't code in the same time).
For the performance part, it would be an interesting to see how many percents faster a home-grown solution would be. If I had to take a guess, the difference between solutions would be in few percents ranges (either way).