Just make sure to program your game to save data correctly! It doesn't go in the program folder, it goes into the user's proper folder for saving program save data and settings.
Mhm... why would this be "correct" way to save data?
I know allot games do save "save files" in "my documents" folder or "my documens/Games".
Googling provided little info on this matter.
Many people don't want to remove their savegames when they uninstall a game. Keeping them separate from the actual game folder provides an easy way to allow uninstall/reinstall without losing any savegames.
Edit: Also, I would highly suggest using NSIS as Daaark suggested. I've created multiple installers using it and the scripting is very powerful.