Aligning ESP stack pointer
Author
In a recent post I found a problem where a DLL compiled with GCC expects the stack to ALREADY be aligned to 16 byte boundaries, wherefore misalignment causes a crash in some SSE2 routines in the DLL. This happens when calling the GCC DLL function from a VC++ application, where the stack pointer is only guaranteed to be 4 byte aligned.
I need to call this function in the DLL:
int _cdecl ffmpeg_main(char* params);
Can anyone here provide me with some Visual C++ (Express 2005) inline ASM (or otherwise), make sure ESP is 16 byte aligned once in the DLL function? It'd be greatly appreciated.
__asm{ mov eax, esp and eax, 15 add eax, 4 sub esp, eax push eax push params call ffmpeg_main add esp, 4 pop eax add esp, eax}ESP will be 16-byte aligned on entry to ffmpeg_main, params is your char* argument. Hope that works.
Author
Thanks alot! It worked after a slight modification ;) Rate up for you. Correction and capturing of return value:
__asm { mov eax, esp and eax, 15 add eax, 8 sub esp, eax push eax push cmdLine call ffmpeg_main mov retVal, eax add esp, 4 pop eax add esp, eax };
You're welcome.
I don't understand your correction however, why are you pushing an extra 4 bytes? If you're stepping through it with a debugger note that the call instruction will also push the return address onto the stack, so your ESP should be 4 off just before and after the call, but on entry to the function it will be aligned.
I don't understand your correction however, why are you pushing an extra 4 bytes? If you're stepping through it with a debugger note that the call instruction will also push the return address onto the stack, so your ESP should be 4 off just before and after the call, but on entry to the function it will be aligned.
Author
Ehm.. well, I have no idea to be honest. All I know is that the SSE2 stuff doesn't crash any more when the low ESP nibble is zero before the "call". When GCC expects the stack to be 16 byte aligned, perhaps that's what's meant?
Just thinking about a DLL having that kindof requisites makes me sick. This is something that should be handled in the functions themselves, not be forced upon the consumer of the module.
Author
Hehe.. well fortunately my application will be the only consumer of this particular DLL. However it's likely it might always be an issue when using the original FFMPEG DLLs from VC++. It's the price we pay when there's no standard for these kind of things. Using a newer unofficial realease of GCC there is automatic stack alignment in critical functions. I just rather fix it in VC++ than in FFMPEG, as it's nice to just get the SVN without having to change the code. That and I've always thought GCC's inline ASM looked scarier than VC++'s ;)
I am also having the same problem,
calling avcodec_decode_video() from microsoft visual studio 2005 crashes the application. I dont have knowledge in inline assemply so please provide me Inline assembly( with ESP 16 byte aligned ) to call this function .
int avcodec_decode_video(AVCodecContext *avctx, AVFrame *picture,
int *got_picture_ptr,
uint8_t *buf, int buf_size);
where AVCodecContext and AVFrame are structures.
thanks in advance.
calling avcodec_decode_video() from microsoft visual studio 2005 crashes the application. I dont have knowledge in inline assemply so please provide me Inline assembly( with ESP 16 byte aligned ) to call this function .
int avcodec_decode_video(AVCodecContext *avctx, AVFrame *picture,
int *got_picture_ptr,
uint8_t *buf, int buf_size);
where AVCodecContext and AVFrame are structures.
thanks in advance.
This topic is closed to new replies.
Advertisement
Popular Topics
Advertisement
