What's transform.position? If it's not the camera position or ray origin, then the direction won't point to where you clicked. And by the way, I think it would be better to give an impulse to your bullet by setting its velocity directly instead of using AddForce.

The script is attached to the Player character so it's the center of the player capsule. While it's not the camera transfrom, camera is also attached to this player and it's position is maybe 0.1 units up in y direction from player capsule, but let me try to set it to camera explicitly.

Edit: Yes that was it! The ball flies in the right direction, and I'll look into setting the impulse, thanks Faelnor!