Here's a quick attempt to help you define the Earth's initial position and initial velocity vector if it were to have a circular orbit path, where the Sun is fixed in position and has no velocity. Try using these parameters first so you'll know if it works as it should, before you start monkeying around with the value of G.
G = 6.67384e11 // Newton's gravitational constant
M = 1.9891e30 // Mass of Sun in kilograms
r = 149597887500 // Earth's semi-major axis -- its average orbit distance in metres
v = sqrt(GM/r) = 29789 // Earth's average orbit speed in metres per second -- NOTE: THIS DIFFERS FROM THE EQUATION THAT YOU'RE USING IN YOUR INITIAL POST.
Sun's fixed position = {0,0,0}
Sun's fixed velocity vector = {0,0,0}
Earth's initial position = {0, 149597887500, 0}
Earth's initial velocity vector = {29789, 0, 0} // orthogonal to the position vector
It's that simple.
If you want to, you can rotate the Earth's initial position and velocity (make sure you rotate both using the same angle) along the z axis before the simulation begins: http://en.wikipedia.org/wiki/Rotation_matrix If you need help with this, let us know. Hint: you can just use the 2D rotation since the orbit path will lie along the xy plane http://en.wikipedia.org/wiki/Rotation_matrix#In_two_dimensions
If you're still stuck, then you're probably doing something wrong when it comes to how you're using a time step (assuming you're calculating acceleration correctly). Check out gaffer's Fix Your Timestep if you haven't already: http://gafferongames.com/game-physics/fix-your-timestep/ and also see the integration basics tutorital http://gafferongames.com/game-physics/integration-basics/ If you're still stuck after reading those two gaffer articles then upload your code into a zip file and post it here.
- Shawn
P.S. For more information, the following link discusses elliptical orbits:
http://www.gamedev.net/topic/447624-calculating-an-initial-velocity-for-desired-planetary-orbit/#entry3961958
So we're clear, the eccentricity parameter is set to 0 for circular orbits.