Jump to content
  • Advertisement
Sign in to follow this  
  • entries
  • comments
  • views

Stack corruption tale

Sign in to follow this  


Stack corruption tale

While writing the next tutorial, and I came back to the new FAT12 file loading routines, I was reminded of some of the very odd bugs I had with it when writing the initil code. The routines simply had stack corruption (Which was fixed some time ago). The results from this corruption was weird though. I personally never seen anything like it.

While trying to fix the code, I have ran into three problems. The first two are pretty simple ones, but the third problem is interesting.

As you know, each routine pushes all general registers on the stack (And the call instruction pushes IP). Because of this, the routine ended with:


Pretty standard and simple. Because of stack corruption, however, the ret instruction popped an incorrect value into CS:IP. This of course, usually causes the processor to execute whatever garbage code or data that is at that location, and will continue to do so until either a triple fault or the end of memory is reached (again, triple fault.)

The second thing that happened was an infinity loop, where the program kept executing itself over and over again. Because the program is loaded at 0x500 (0x50:0), and the stack was corrupt, the ret instruction popped a 0 into IP. Because CS was still equal to 0x50, this effectively made the ret instruction jump back to this new CS:IP location, which is now 0x50:0 (0x500 absolute address), causing it to execute itself. Ugh.

Again, THAT was fixed.

The third problem was the strangest. Somehow, the ret instruction caused the video mode itself to change, yet everything was still running fine. Mind you, this was a graphics mode, not a text mode, so some random pixels here and there, but nothing much.

I am not entirely sure how this one happened :/ Perhaps the ret instruction jumped to that location in the IVT? Perhaps it jumped to some code that just happened to have an VGA INT 0x10 call? Perhaps it jumped to a location that just happened to write via VGA controller ports? No clue.

Everything is working fine now, of course, but still. Ugh... OS Programming is a pain sometimes. Thank you Bochs Debugger!

Now that the tale is over... Back to the tutorials...
Sign in to follow this  


Recommended Comments

There are no comments to display.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Advertisement

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

GameDev.net is your game development community. Create an account for your GameDev Portfolio and participate in the largest developer community in the games industry.

Sign me up!