I got SFML to linked and compiled. I noticed I downloaded the wrong zlib, I didn't get the dll but I do now. For zlib that sounds too easy. Isn't there supposed to be additional dependencies in the Project > Properties > Linker > Input > additional dependencies. Like sfml had the sfml-graphics.lib files I had to add.
I'm not really looking for the easiest way. Probably somewhere between cleanest setup and easy. The way Servant of the Lord described setting it up sounds messy. I was hoping for a process similar to setting up SFML where the only files I need to move around are dll or something.