As for drawbacks, the main problem is that a circle is not necessarily a close enough representation of your object. For a long thin object the bounding sphere will contain a huge amount of empty space. That will lead to a lot of false positives in intersection tests. An axis aligned or oriented box is a tighter fit in many cases.

Whether circles are good enough for your circumstances, well, that of course has more to do with your circumstances than the circles.

it also depends on whether or not you need to deal with objects which may have an arbitrary rotation.

with a circle or sphere, no matter how the object is rotated, it will fit within the existing space.

if you have to make an axis-aligned bounding box (AABB) over every possible rotation, generally it will involve more empty space than the sphere, though bounding box is generally more accurate if bounding a particular rotation.

though, for oriented bounding boxes (OBB), these are often a tighter fit, but collision checks are more expensive.

likewise for convex polyhedra (where the object is bounded by a collection of bounding planes, or the 2D equivalent being a polygon).

so, generally, if using a more expensive solid-type (like an OBB or polyhedra), it may still make sense to use sphere checks first to eliminate impossible collisions early.

likewise, they may have other uses, like a sphere traveling through space is simpler to bound accurately than an AABB, and may also have advantages for certain other cases, like when trying to find the initial collision point for fast-moving objects, ...