It's best to collect crash dumps from executables you distribute. You can then load these crash dumps with the PDBs generated _at the same time you built the executable_ into Visual Studio quite easily (meaning that you must keep your PDBs around for any executable - or DLL - that you distribute).
You could generate minidumps yourself using code like that illustrated in
CodeProject: Post-Mortem Debugging Your Application with Minidumps and Visual Studio .NET.
Another option is
Google Breakpad. It not only generates the dumps but also helps provide a way of automatically collecting those crash dumps with minimal (or no) user interaction, which will be helpful once you're distributing to builds to more than a couple friends.
With these options, you can collect crashes from any user without any runtime overhead. The executable does not need to be built in Debug mode though it does need to have symbols enabled (which has no runtime overhead).
The problem with adding logging is that you very probably didn't add all the logging you need to debug the problem. Then you're having to continuously deliver updated executables to your long-suffering users, who are continously sending you slightly different log outputs in the hope that the problem is fixed.
With a crash dump, you can immediately inspect the back trace and see exactly where the crash occurred, inspect the stack (to a degree, anyway, since it _is_ a Release mode executable) and even some parts of the program's memory. Basically, you can load up the crash right into your debugger just as if the crash had happened on your own machine, with a few limitations.
--
On top of all this, you can split up your logging, asserts, and so on a bit further. You don't need just Debug mode with everything turned on and Release with all the logging turned off. You can and having some logging statements enabled even in your final production builds. e.g. you might have macros like DEBUG_LOG_IF and RELEASE_LOG_IF or ASSERT and ALWAYS_ASSERT or whatever suits your fancy. A number of high-performance, flexible logging libraries already provide APIs of that sort.
It's not uncommon for games to have a Debug build (no optimizations, all logging enabled, etc.), an Optimized build (full optimizations, all logging enabled) and a Release build (all optimizations, limited logging). You might even have more intermediate build types that e.g. enable "excessive" logging for some modules of the engine to help pinpoint bugs there by turning off all optimizations in that module (so debugging is easy) while keeping optimizations on everywhere else so the game runs well.
Yet another option (that can be combined with all of the previous ones; none of these are exclusionary choices) is to allow efficient runtime selection of logging. A macro like:
#define LOG_IF(predicate, message, ...) do{ if (!!(predicate)) ::LogFunctionPrintf(__FILE__, __LINE__, __FUNCTION__, (message), __VA_ARGS__); }while(false)
Would call the LogFunctionPrintf with location information only if predicate is set. This predicate can just check for a simple global boolean (among other things) so that graphics logging (for example) is only enabled if gLogGraphics is set. Fancier macros and support systems can automate the enable/disable support by modules, vastly improve on the awfulness of the C printf syntax and feature set, allow a "log only once" feature, allow a "break first time this is hit if in a debugger" feature, and so on. Various existing logging libraries support these features, or you can add them yourself if you're into that.