I've currently got libjpg playing ball now and I can successfully read all the scan lines from an image and copy them into a local buffer without it crashing or otherwise over running memory, however its not pretty and there is a slight problem.
For some reason the decoded data is only showing up as a green line down one side of the image and the rest is grey. Which says to me i've a slighy copying issue when it comes to reading in the data (although you reading a 4096 bytes at a time and read a scan line at a time into a local buffer the final consumer is only reading 128bytes at a time, thus there is some copying which has to go on).
I'm considering reworking the code so that it reads ALL the day into memory first and then supplies it as series of bytes at a time to the calling function, basically treating the reading as a statefull operation.
It might get around a few issues i've been having and allow me to get things done a bit cleaner, I'll investigate tomorrow I think.