PIX works by running the shaders in software (reference device) when you break (pause) the execution.

Vendor-specific debuggers could break execution in actual hardware, and copy the shader stack and registers to CPU-accessible memory for the debugger to access.

I think that if i created a virtual machine which is support asm shader code, may be i can replay the pixel shader process

Yes, that is the gist of it, but its a whole project in its own right. You might be able to piggy-back on an existing software renderer like Mesa , Direct3D WARP or the DirectX Ref device. There are limitations though -- for one, your debugging is only as accurate as your VM's adherence to the spec -- this can get very, very detailed, down to the level of where exactly a texel is sampled during boundary conditions, how floating-point values behave, and implementing 16-bit half-floats in software, along with their conversions to and from more common types. Secondly, it's debugging WRT to a reference, so it can help you make sure that *you* are doing the right thing, but if the driver or device does the wrong thing it won't help you (its still a great benefit to know the problem belongs to someone else).