Thank you everyone,
I didn't post my shader because I don't want to scare people - stage3d shaders are assembler like code. I double checked it and it seems correct - I even don't have divisions. And the main thing - without antialiasing I don't have artifacts. So MJP's answer is the explanation I searched for. Unfortunatelly, stage3d doesn't allow to setup sampling or anything like that.
@bwhiiting, for render target AA levels are 0, 1, 2, 3, 4.
@baldurk, I have wrong results in fragment shader. For example in vertex shader I have a varing (vi0) and I can see it's value, but in fragment shader I suddenly have vi1 (not vi0) value filled with zeroes only. So I can't step through the code and check intermediate results (and output color actually is wrong too, although on the screen it's ok). But as you say it can be problem with render target - flash internally draws to some texture and internally draws it to back buffer in the end. And I tried to turn on every option.
Upd.: I checked on different machine and I can't see this artifacts, so I assume that this is my hardware's fault.