First of all, let me express my congratulations that you managed to stay away from optimizations up till this point, since now you have something up&running and just need to clean it up a bit.
Is this C# ? What platform are you getting the slowdowns on ? I am asking since under .NET, the GC runs in a separate thread on PC and you can experience the GC-caused slowdowns only on XBOX.
Also, note that you don't want to write to a log file 10 times during a single frame. You want to collect the data in RAM and only dump the data to physical file after you are done with profiling.
If it is C#, there are countless threads on XNA forums (of MS) about what does and what does not leak memory under C#. That list is pretty long and some of the stuff you find there is really suprising.
I will, however, mention No1 suspect and that is the String class. Whichever way you use it, leaks incredible amount of memory (so just use StringBuilder).
On XBOX, even if all you do is this single line every frame : String s1 = s2+s3; it will cause your game to halt for half a second every ~5 seconds.
There are .NET profilers, that will run alongside your game and will tell you exactly, each frame, how much memory has been leaked and you can see the moment when the GC runs. The name of that .NET profiler , unfortunately, escapes me at the moment.