Is it also expected to have the frame sometimes take 5ms and sometimes .3ms? I am checking now if triple buffering is enabled, but I do know that double is, as that is the SDL_SwapBuffers() command.
Absolutely. There might be a context switch and scheduler time slices are often quite long. Also, as tonemgub points out, you should really only be calling now() once per frame. (directly after swap is usually a good choice)