So do I understand you correctly that you want to draw a filled shape (circle in this case) and use Bresenham's algorithm (which is used mostly for drawing circle contour) ?
The difference between D3DFVF_XYZRHW and D3DFVF_XYZ is that the RHW version skips vertex shader / projection part ie. it assumes that the data is already transformed. D3DFVF_XYZ doesn't skip transformation and projection so you'll need to provide a correct world, view and projection matrix in order to draw correctly.
If you are drawing 2d shapes on the screen then the RHW version should do the trick. However, you may accomplish the same with D3DFVD_XYZ but you'll need correct matrices for it to work correctly. Do you need to draw circles in 3d space?
yeah, that's what i mean, in 3d space
even i increase the radius and adjust the parameter of D3DXMatrixLookAtLH to suit the radius, the effect is awful