Exactly, the whole point of adding compression is that sometimes reading uncompressed data takes longer than reading it compressed into memory and uncompress it on the fly.
Well that was a stupid question. I totally forgot that you can load files into a MemoryStream. So you would just load the zip file contents into a MemoryStream without having to decompress it to the hard drive.
Although ZIP is kinda heavy weight for decompression AFAIK, you could try with other faster methods like LZ4 (that dont give such good compression rations though). As with anything, you'd need to test it out.