The initial assumption is that you have arrived at the last chamber, and there is a 50% chance that the referee has put in a bullet.
If there is a bullet, the bullet must be in this last chamber (there is no other way!), and the referee made his decision whether or not to put one in before anyone touched a trigger. The 50% chance that there is a bullet in one chamber doesn't change after the referee has tossed his coin, merely because nobody has died during the first five rounds. The only thing that changes during the game is the number of chambers that are left (and people dying, but that is outside the frame conditions of the experiment).
This is where your misunderstanding lies. There is a 50% chance that the referee has put a bullet in this gun. Not in this chamber. You don't know which gun you have. You might have the empty gun.
Think of it like this: You have a total of 12 available barrels. 1 barrel has the bullet.
Initial chance: 1/12
second chance 1/11
third chance 1/10
4th chance 1/9
5th chance 1/8
6th chance: 1/7.
It doesn't matter what random subset of barrels you take the first 5 samples from. If there wasn't a bullet yet, it has to be in the next 7 barrels.