Private is very important in that it lets you provide a clean API to end users that allow you to modify the internals with minimal recompilation and preserve class invariants. Protected has a few useful applications (for holding an interface's destructor for example if you don't want someone destroying an object via the interface).
Except we're talking about C++, which immediately and catastrophically fucks this up by making the private internals part of the header anyway. So then we end up with tricks ranging from vanilla polymorphic interfaces to pImpl idiom variations, rendering the whole exercise pointless.
Like others have said, access semantics, like other things in the language, are there to protect the programmer from making mistakes and to help enforce good code design
And the moment you inflict friend, we're not talking about good code design anymore, we're talking about continuing hacks where the core design has already gone awry. It's worth noting that a number of modern languages get by just fine without access semantics. My argument is they may get along better without them and that the very use of access semantics represents total failure of encapsulation in C++. Why? Because even if I can't access the internal details of a class, I can see them just fine due to other encapsulation-puncturing issues in the language. That serves to bind clients to implementation details anyway, in the absence of pImpl type techniques. Lacking the relevant semantics forces you to deal with the interface/implementation divide directly. This is not the case in Java or C#, which implement sane module systems and let you alter implementations without impacting clients.
If you consider C++ private/protected semantics relevant to your code design, it's time for a redesign.