Depending on how you allocate that memory Windows will let you do that (use more memory than is physically available).
Back in the old days of yore (Win 3.11) I could load and scroll on a 8 MB bitmap on a 4 MB machine. No, it was not particularely fluid, but it worked.
Checking the state is good, however having a forced abort is not. Even if you deem memory not sufficient, let it run (maybe with a warning).