At 500hz, your frame time is just 2ms, so small jitter values will have a large impact. Windows thread scheduling granularity defaults to 15ms.
If you've got other programs running, it could be that your thread called Sleep, and another program got the CPU for 15ms, resulting in one frame that's 7 times longer than the surrounding ones.
Do you have a virus scanner running?

Also, at 500hz, assuming you've got a regular 60hz monitor, then your monitor is only displaying every 8(and a 3rd)th frame. If your rotation speed is fixed per frame, this means that you'll have three frames where you see "rotation*8" movement, but the fourth frame you'll see "rotation*9" movement.

If there's some jitter in your frame time (which you should expect a small amount of), and for some reason your frame times are between 1.5 and 2.5ms, instead of a solid 2ms, then:
1.5ms == 666hz == every 11.1th frame displayed == movement per refresh: 11,11,11,11,11,11,11,11,11,12,...
2ms == 500hz == every 8.3th frame displayed == movement per refresh: 8,8,8,9,8,8,8,9,8...
2.5ms == 400hz == every 6.6th frame displayed == movement per refresh: 6,7,6,8,7...

So with such low frame-times, frame-rates higher than the display rate, and with non frame-rate independent movement, you should expect some jitter.
If you vsync'ed to 60, you probably wouldn't be noticing these issues right now

I would start by implementing frame-rate independent movement - where your rotations take into account the actual time passed.

At 500hz, your frame time is just 2ms, so small jitter values will have a large impact. Windows thread scheduling granularity is 15ms.
If you've got other programs running, it could be that your thread called Sleep, and another program got the CPU for 15ms, resulting in one frame that's 7 times longer than the surrounding ones.
Do you have a virus scanner running?

Also, at 500hz, assuming you've got a regular 60hz monitor, then your monitor is only displaying every 8(and a 3rd)th frame. If your rotation speed is fixed per frame, this means that you'll have three frames where you see "rotation*8" movement, but the fourth frame you'll see "rotation*9" movement.

If there's some jitter in your frame time (which you should expect a small amount of), and for some reason your frame times are between 1.5 and 2.5ms, instead of a solid 2ms, then:
1.5ms == 666hz == every 11.1th frame displayed == movement per refresh: 11,11,11,11,11,11,11,11,11,12,...
2ms == 500hz == every 8.3th frame displayed == movement per refresh: 8,8,8,9,8,8,8,9,8...
2.5ms == 400hz == every 6.6th frame displayed == movement per refresh: 6,7,6,8,7...

So with such low frame-times, frame-rates higher than the display rate, and with non frame-rate independent movement, you should expect some jitter.
If you vsync'ed to 60, you probably wouldn't be noticing these issues right now

I would start by implementing frame-rate independent movement - where your rotations take into account the actual time passed.

At 500hz, your frame time is just 2ms, so small jitter values will have a large impact. Windows thread scheduling granularity is 15ms.
If you've got other programs running, it could be that your thread called Sleep, and another program got the CPU for 15ms, resulting in one frame that's 7 times longer than the surrounding ones.
Do you have a virus scanner running?

Also, at 500hz, assuming you've got a regular 60hz monitor, then your monitor is only displaying every 8(and a 3rd)th frame. If your rotation speed is fixed per frame, this means that you'll have three frames where you see "rotation*8" movement, but the fourth frame you'll see "rotation*9" movement.

If there's some jitter in your frame time, and for some reason your frame times are between 1.5 and 2.5ms, instead of a solid 2ms, then:
1.5ms == 666hz == every 11.1th frame displayed == movement per frame: 11,11,11,11,11,11,11,11,11,12,...
2ms == 500hz == every 8.3th frame displayed == movement per frame: 8,8,8,9,8,8,8,9,8...
2.5ms == 400hz == every 6.6th frame displayed == movement per frame: 6,7,6,8,7...

So with such low frame-times, frame-rates higher than the display rate, and with non frame-rate independent movement, you should expect some jitter.
If you vsync'ed to 60, you probably wouldn't be noticing these issues right now

I would start by implementing frame-rate independent movement - where your rotations take into account the actual time passed.

At 500hz, your frame time is just 2ms, so small jitter values will have a large impact. Windows thread scheduling granularity is 15ms.
If you've got other programs running, it could be that your thread called Sleep, and another program got the CPU for 15ms, resulting in one frame that's 7 times longer than the surrounding ones.
Do you have a virus scanner running?

Also, at 500hz, assuming you've got a regular 60hz monitor, then your monitor is only displaying every 8(and a 3rd)th frame. If your rotation speed is fixed per frame, this means that you'll have three frames where you see "rotation*8" movement, and every fourth frame you'll see "rotation*9" movement.

If there's some jitter in your frame time, and for some reason your frame times are between 1.5 and 2.5ms, instead of a solid 2ms, then:
1.5ms == 666hz == every 11.1th frame displayed == movement per frame: 11,11,11,11,11,11,11,11,11,12,...
2ms == 500hz == every 8.3th frame displayed == movement per frame: 8,8,8,9,8,8,8,9,8...
2.5ms == 400hz == every 6.6th frame displayed == movement per frame: 6,7,6,8,7...

So with such high frame-rates, and non frame-rate independent movement, you should expect a lot of jitter. If you vsync'ed to 60, you probably wouldn't be noticing these issues.