I managed to "fix" the issue by reducing my max vertex buffer size. I noticed that the size of the input layout didn't really matter. It was more to do with the stride size (size of my CPU vertex struct).
But this is very much still magic for me as I don't understand how the maximum vertex buffer size is related to stride size. This also doesn't explain why I had still high frame rates before a device reset with the original vertex buffer sizes and only lost fps after the device reset. (I don't change vertex buffer sizes during device resets).
It's no longer a pressing issue but I'm curious as to what's going on here.