Sign in to follow this  

Aligning ESP stack pointer

This topic is 3583 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

In a recent post I found a problem where a DLL compiled with GCC expects the stack to ALREADY be aligned to 16 byte boundaries, wherefore misalignment causes a crash in some SSE2 routines in the DLL. This happens when calling the GCC DLL function from a VC++ application, where the stack pointer is only guaranteed to be 4 byte aligned. I need to call this function in the DLL: int _cdecl ffmpeg_main(char* params); Can anyone here provide me with some Visual C++ (Express 2005) inline ASM (or otherwise), make sure ESP is 16 byte aligned once in the DLL function? It'd be greatly appreciated.

Share this post


Link to post
Share on other sites

__asm
{
mov eax, esp
and eax, 15
add eax, 4
sub esp, eax

push eax
push params
call ffmpeg_main

add esp, 4
pop eax
add esp, eax
}



ESP will be 16-byte aligned on entry to ffmpeg_main, params is your char* argument. Hope that works.

Share this post


Link to post
Share on other sites
Thanks alot! It worked after a slight modification ;) Rate up for you. Correction and capturing of return value:


__asm
{
mov eax, esp
and eax, 15
add eax, 8
sub esp, eax

push eax
push cmdLine
call ffmpeg_main
mov retVal, eax

add esp, 4
pop eax
add esp, eax
};


Share this post


Link to post
Share on other sites
You're welcome.

I don't understand your correction however, why are you pushing an extra 4 bytes? If you're stepping through it with a debugger note that the call instruction will also push the return address onto the stack, so your ESP should be 4 off just before and after the call, but on entry to the function it will be aligned.

Share this post


Link to post
Share on other sites
Ehm.. well, I have no idea to be honest. All I know is that the SSE2 stuff doesn't crash any more when the low ESP nibble is zero before the "call". When GCC expects the stack to be 16 byte aligned, perhaps that's what's meant?

Share this post


Link to post
Share on other sites
Just thinking about a DLL having that kindof requisites makes me sick. This is something that should be handled in the functions themselves, not be forced upon the consumer of the module.

Share this post


Link to post
Share on other sites
Hehe.. well fortunately my application will be the only consumer of this particular DLL. However it's likely it might always be an issue when using the original FFMPEG DLLs from VC++. It's the price we pay when there's no standard for these kind of things. Using a newer unofficial realease of GCC there is automatic stack alignment in critical functions. I just rather fix it in VC++ than in FFMPEG, as it's nice to just get the SVN without having to change the code. That and I've always thought GCC's inline ASM looked scarier than VC++'s ;)

Share this post


Link to post
Share on other sites
I am also having the same problem,
calling avcodec_decode_video() from microsoft visual studio 2005 crashes the application. I dont have knowledge in inline assemply so please provide me Inline assembly( with ESP 16 byte aligned ) to call this function .

int avcodec_decode_video(AVCodecContext *avctx, AVFrame *picture,
int *got_picture_ptr,
uint8_t *buf, int buf_size);

where AVCodecContext and AVFrame are structures.


thanks in advance.

Share this post


Link to post
Share on other sites

This topic is 3583 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this