And just to show that RenderDoc was trying to warn me of my downfall, here's some shots with the unset and set sampler. They've even gone and highlighted it in red for me
I had noticed this earlier in my debugging but I dismissed it too easily. I guess I didn't trust my tools. Lesson learned.