It is possible in to handle and recover from a failed memory allocation attempt and continue running, however

You see mobile games handling this much more often, as memory is often quite constrained on mobile devices, and the OS tends to be much more aggressive about closing applications that use too much memory.

Although it is worth noting that the allocation call itself rarely actually fails on mobile. Instead, both Android and iOS have high-level events which will (hopefully) be delivered to your applications before the system runs out of memory, and if you continue to allocate more memory in such a situation, the OS will shortly terminate your process.

It's not a simple question actually.

Virtual address is not the same as virtual memory.

32-bit processes cannot address more than 2^32 bytes of memory. However, they can access more than that considering they only work on sections of it at a time and remap different chunks of virtual memory into virtual addresses as they need.

With this trick, on 32-bit OSes, up to 64GB of memory can be accessed (assuming PAE is enabled). On 64-bit OSes I don't know if the 64GB limit still applies, or if they can use more.

Here's an example on how to do that: https://blogs.msdn.microsoft.com/oldnewthing/20040810-00/?p=38203

You will notice that you need OS specific calls to do this. Regular malloc won't do.

Going back to the original question, what will happen? Well it depends of how the program is coded:

1. Apps using malloc that run out of virtual addressing ranges, will get a null pointer. If the app checks for this condition, it can successfully try to recover (i.e. start to free memory and try again, show an error and quit, silently quit, etc). If the app doesn't check for this condition, it will continue to execute normally until the null pointer is used which will lead to a crash. Usually malloc'ed memory is used immediately afterwards, so the crash will happen close to the malloc call; but who knows how much the crash could be delayed if the null pointer isn't accessed soon. It's also likely many subsequent malloc calls will fail, which means a crash is imminent unless a lot of memory suddenly became available again.
2. Apps using C++'s new operator will throw an exception when it runs out of memory; so it will crash immediately. However the app can catch this exception and try to recover, like in the malloc case. Poorly written C++ code may catch all exceptions and not notice it's catching an out of memory exception; in which case they'll continue to execute except now they have an uninitialized pointer (which worse than a null pointer). Uninitialized pointers normally crash immediately when they're used, or may corrupt the process' memory and continue to run until it's so broken it can't work anymore or crashes (think Gameboy's Pokemon glitches)
3. Apps using other languages (i.e. Python, Ruby, Java) depend on how their interpreter manages out of memory conditions, but most of them handle it by just crashing or raising an exception (also depends on whether the interpreter couldn't honour the amount of memory the executed code is requesting; or if the interpreter itself ran out of memory while inside of its internal routines).
4. Apps manually managing memory (the CreateFileMapping method in the blog) are advanced enough to assume they're handling this condition. When out of memory, CreateFileMapping and/or MapViewOfFile will fail. The rest is as the malloc case.

Edit: What I wrote is Windows/Linux/OS X specific. As described above by swiftcoder, iOS and Android are much more strict about memory (they just immediately terminate a process that starts consuming too much memory)

Basically true, although you can specifically override that behavior with std::nothrow. However while it should be mentioned for completeness sake I have never actually seen it in the wild.

It's worth noting that ~90% of PCs have a 64bit OS these days (src: steam hardware survey)

Unless you're stuck with a 32bit code base for legacy reasons, there's really no reason not to go 64bit.