when you say the pixel shader has no effect, do you mean that nothing is rendered? have you tried pix to see what the vertices look like after the vertex shader has passed?

EDIT: IIRC SetFVF will not work with the shader pipeline, only the FFP. For shaders, vertices need to be bound with VertexDeclarations, which WILL work with the FFP.

http://msdn.microsoft.com/en-us/library/windows/desktop/bb206335%28v=vs.85%29.aspx

EDIT2: I was wrong, FVF vertex buffers can be used with vertex shaders.