In 3d space where

(1,0,0) - right

(0,1,0) - up

(0,0,1) - forward

0 degree means that vector from two points is (0,0,1)

90 degree means that vector from two points is (-1,0,0)

180 degree means that vector from two points is (0,0,-1)

270 degree means that vector from two points is (1,0,0)

My algorithm for calculation angle between two points is as below (in fact I calculate angle between camera location position and camera look-at position:

float angle = calculateAngle(cam->GetCamPos(), cam->GetCamLook()) - d3d::DEGREE_90;

float calculateAngle(D3DXVECTOR3* baseVec, D3DXVECTOR3* targetVec)
{
D3DXVECTOR2 vDiff;

D3DXVECTOR2 baseVec2(baseVec->x,baseVec->z);
D3DXVECTOR2 targetVec2(targetVec->x,targetVec->z);

D3DXVec2Subtract(&vDiff, &targetVec2, &baseVec2);
D3DXVec2Normalize(&vDiff, &vDiff);
//the angle is calculated from +X axis where the center of coordinate system XZ is baseVec
if(!vDiff.y)
{
    if(vDiff.x>0)
       return 0;
    else
       return D3DX_PI;
}
else if( vDiff.y > 0.f )
    return acosf(vDiff.x);
else
     return (acosf(-vDiff.x) + D3DX_PI);
}

The results is as in the attached image:

[attachment=21083:issue0.JPG]

[attachment=21084:issue1.JPG]

Instead of angle 45-60 I have angle 4 degree, so something is wrong.

My input data GetCamPos and GetCamLook are from View Matrix. Their coordinates are also seen at the attached pictures (camera pos, camera target).

The aim is that when I press "D" ball is kicked straigt forward according to the view.

I suppose that algorithm is OK, however instead GetCamLook I should take another input data, however I don't know which one. Maybe GetCamPos + GetCamLook or something like that.

First some clarifications: You seem me to want to compute the angle from the principal z direction to the look direction vector, in the x-z plane, and in the counter-clockwise manner. Notice that this is not the same as computing the angle between the difference from the origin to the camera position and the difference from the origin to the look-at position (as you do in the OP). Saying "the angle between" is not direction dependent, i.e. gives the same result if the look vector is e.g. 45° left or 45° right to the axis. In such a case the dot product can be used, as phil_t has mentioned. However, in the OP you make a distinction between (+1,0,0) to result in 90° and (-1,0,0) to result in 270°, so the following method may be better suited...

The look vector is a direction vector. You can extract it from the camera matrix (notice that this is the inverse of the view matrix). Or you can compute it as difference from the camera position to the view at position.

The atan2 function can be used to compute the angle (in radian) from the x and z components of the said look vector relative to a principal z axis. The question to answer is whether to use atan2(x, z) or atan2(z, x), and whether one of the components need to be negated. Without having proven it, I assume atan2(-x, z) would do the job (please check twice).

The result of atan2 is in [+pi,-pi), so you need to add 2pi if the atan2 itself is negative if you actually need [0,2pi) for some reason. Since you want to have the angle in degrees, you further have to transform it, accordingly, of course.

However, as phil_t has mentioned, you perhaps (I tend to say "probably" yet) do not need to compute the angle at all. It is often better to use direction vectors directly.

First some clarifications: You seem me to want to compute the angle from the principal z direction to the look direction vector, in the x-z plane, and in the counter-clockwise manner. Notice that this is not the same as computing the angle between the difference from the origin to the camera position and the difference from the origin to the look-at position (as you do in the OP). Saying "the angle between" is not direction dependent, i.e. gives the same result if the look vector is e.g. 45° left or 45° right to the axis. In such a case the dot product can be used, as phil_t has mentioned. However, in the OP you make a distinction between (+1,0,0) to result in 90° and (-1,0,0) to result in 270°, so the following method may be better suited...

The look vector is a direction vector. You can extract it from the camera matrix (notice that this is the inverse of the view matrix). Or you can compute it as difference from the camera position to the view at position.

The atan2 function can be used to compute the angle (in radian) from the x and z components of the said look vector relative to a principal z axis. The question to answer is whether to use atan2(x, z) or atan2(z, x), and whether one of the components need to be negated. Without having proven it, I assume atan2(-x, z) would do the job (please check twice).

The result of atan2 is in [+pi,-pi), so you need to add 2pi if the atan2 itself is negative if you actually need [0,2pi) for some reason. Since you want to have the angle in degrees, you further have to transform it, accordingly, of course.

Correction: you would need to add PI, not 2pi, to transform an interval ranging from from -pi..pi, to the range of 0..2pi. In general to convert from a signed -n..n, range to an unsigned 0..2n range add 1/2 the range, 1n in this case, or 1pi in the situation above.

Correction: you would need to add PI, not 2pi, to transform an interval ranging from from -pi..pi, to the range of 0..2pi. In general to convert from a signed -n..n, range to an unsigned 0..2n range add 1/2 the range, 1n in this case, or 1pi in the situation above.

The transform I mentioned is not linear: With atan2 giving you angles in the range [+pi,-pi) where [+pi,0] is as desired, but (0,-pi) should be (2pi,pi) instead, you have to add 2pi for all resulting angles less than 0. As pseudo code:

result := angle < 0 ? 2pi+angle : angle w/ angle := atan2(-x, z)

Correction: you would need to add PI, not 2pi, to transform an interval ranging from from -pi..pi, to the range of 0..2pi. In general to convert from a signed -n..n, range to an unsigned 0..2n range add 1/2 the range, 1n in this case, or 1pi in the situation above.

The transform I mentioned is not linear: With atan2 giving you angles in the range [+pi,-pi) where [+pi,0] is as desired, but (0,-pi) should be (2pi,pi) instead, you have to add 2pi for all resulting angles less than 0. As pseudo code:

result := angle < 0 ? 2pi+angle : angle w/ angle := atan2(-x, z)

Some day I'll learn not to spout my mouth off before coffee time :) Thanks for the correction on my correction.