While I'll be Fu**ed. Decompressing PNGs using a 3rd party library turns out to be slightly faster than reading in uncompressed TGA/BMPs. I had no idea that FileIO was such a bottleneck. Especially, I had assumed the difference betweeen reading 500bytes and 3000 was negligable. Stupid me.

I expect to get a larger gain when I write my own PNG reader that only considers the PNG formats I'm using in my engine.