if(SDL_PollEvent(&event))

Should be:

while ( SDL_PollEvent( &event ) )

It's likely that your event queue accumulates more and and more windows events until your game slows down to a crawl -- you need to process all of the window messages between each frame, not just one per frame. (Not sure if this will fix your problem, but it will certainly help...)

if(SDL_PollEvent(&event))

Should be:

while ( SDL_PollEvent( &event ) )

It's likely that your event queue accumulates more and and more windows events until your game slows down to a crawl -- you need to process all of the window messages between each frame, not just one per frame.

if(SDL_PollEvent(&event))

Should be:

while ( SDL_PollEvent( &event ) )