Back to General and Gameplay Programming

stack overflows: why not turn the stack upside down?

DracoLacertae · 2014-02-14T10:18:02

We all know about stack overflows. You overwrite your buffer on the stack, and you are now trashing another buffer, or even worse, the return address. Then when 'ret' (I'm using x86 lingo here), runs, the cpu jumps to a bad place. This is because on most platforms the stack grows downward. When you exceed your stack frame's bouundary, you start tracking the previous stack frame. This downward growth is mostly hisorical. On older computers, before virtual memory, your stack started at the top of RAM, and grew downwards. The heap started somewhere else and grew up. You ran out of memory when the two pointer met. With virtual memory, you can put the stack 'anywhere' and even grow it on a page fault in some cases. And I think it's time to turn the stack around: Suppose f() calls g() and that calls h(): classic stack: 'R' is where the return address goes: low addr [empty-----------------R][--f--] high addr low addr [empty------------[--g-R][--f--] high addr low addr [empty-----[--h-R][--g-R][--f--] high addr How H overflows: low addr [empty-----[--h--*R*****][--f--] high addr we crash, or worse, run an explot from h inserting an return address to who-knows-where upside-down stack: low addr [--f--][-empty------------------] high addr low addr [--f--][R-g--][empty-----------] high addr low addr [--f--][R-g--][R-h--][empty----] high addr Now H overflows: low addr [--f--][R-g--][R-h***********--] We've overflowed into empty space. Return address and previous stack frames are safe This wouldn't be too hard to do. Most stack access during a function is done pointer arithmetic: mov eax, [esp-4] mov eax, [esp+4] I know there are some hardware platforms that already have stacks that grow upwards. Redefining the ABI for x86 would break using standard libraries, but for inside your own application, this might enhance security. I suppose its even possible to use an 'upwards' stack for your application, and then when you call a 3rd party library, switch the stack pointer to a seperate area where you have a standard downwards stack defined. I imagine this would involve a lot of hacking around in gcc or llvm to make it work. In an open OS like linux, maybe you could recompile the whole system to use upwards stacks. Just a thought. Downvote if it stinks!

General and Gameplay Programming Programming

Started by DracoLacertae February 13, 2014 11:41 PM

9 comments, last by Paradigm Shifter 10 years, 2 months ago

Paradigm Shifter

5,832

February 14, 2014 10:18 AM

(Singing) "Whichever way the stack grows doesn't really matter to me!"

The heap grows the other way though... stack at one end, heap at other, of addressable memory. When they meet in the middle it's Bummed In The Gob Time (R)

This doesn't apply to many modern systems. Consider that each thread in a multithreaded system needs its own stack.

Good point! Drunk again ;)

"Most people think, great God will come from the sky, take away everything, and make everybody feel high" - Bob Marley

stack overflows: why not turn the stack upside down?

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

stack overflows: why not turn the stack upside down?

This topic is closed to new replies.

Popular Topics

Recommended Tutorials

Reticulating splines