My biggest downsides are crappy 3D asset importing, and the whole way that project files are handled. Unity wants me to work out of a folder they manage, and it becomes a pain in the ass to manage files that are constantly changing, because I can't risk saving the only copy in the Unity folder in case it gets accidentally nuked.
Oh, I know precisely what you mean... Not mentioning that sometimes when you upgrade your Unity version it breaks your project.
Most of the time I work on Unity is because the company I work for currently uses it. Other than that it´s because of the cross-platform features that once you know how they work, they can really save some serious time.