I have a character and want to check if there is a certain model is in front of the character (cone visibility check)

I'm not sure how do I get cosMaxAngle, I think it has something to do with field of view, if so how do I get the character field of view? (I have the character transformation matrix)

BOOL CheckVisibility()
{
   D3DXVECTOR3 view = target->GetPosition() - character->GetPosition();
   float distance = GetDistance(character->Position(), target->Position());

   if(distance > 100.0f)
      return false; //not visible, too far

   D3DXVec3Normalize(&view, &view); //view = normalize(view);
   D3DXVECTOR3 enemyLookDir;

   D3DXMATRIX ZUnitVec;
   D3DXMATRIX matRot;
   D3DXMatrixTranslation(&ZUnitVec, 0.0f, 0.0f, 1.0f); // matrix holding position of vector pointing to positive z
   D3DXMatrixRotationYawPitchRoll(&matRot, D3DXToRadian(character->rotationX), D3DXToRadian(character->rotationY), D3DXToRadian(character->rotationZ));
   D3DXMATRIX m = ZUnitVec * matRot;
   enemyLookDir = D3DXVECTOR3(m._41,m._42,m._43); // Get vector from the matrix
    D3DXVec3Normalize(&enemyLookDir, &enemyLookDir);

   // Cosine of the angle between the view vector and the direction the enemy is looking
   float cosAngle = D3DXVec3Dot(&enemyLookDir, &view);

   float cosMaxAngle = ??? // I think I should use the field of view here? how? I want the character vision cone to be limited to 90 degrees

   // Check if player is inside the cone
   if(cosAngle < cosMaxAngle)
      return false; // Not visible

   return true; // Visible
}

When you dot two (normalized) direction vectors you get value in range 1 to -1, so 90 degrees would be 0.5.

Your code is a bit unclear to me, so i will try to explain a bit different.

Say you have character1 position, its target position (look at position) and character2 position, so your function would be something like this:

bool CheckVisibility( const D3DXVECTOR3& char1Pos, const D3DXVECTOR3& char1Target, const D3DXVECTOR3& char2Pos, float cosMaxAngle = 0.5f, float maxDistance = 100.0f)
{
    float maxDistSq = maxDistance * maxDistance;
    float dist = D3DXVec3LengthSq(&(char2Pos - char1Pos));
    if(dist > maxDistSq)
        return false;
 
    D3DXVECTOR3 view = char1Target - char1Pos;
    D3DXVec3Normalize(&view, &view);
   
    D3DXVECTOR3 toChar2 = char2Pos - char1Pos;
    D3DXVec3Normalize(&toChar2, &toChar2);
 
    float cosAngle = D3DXVec3Dot(&toChar2, &view);
    if(cosAngle < cosMaxAngle)
        return false;
    return true;
}

@belfegor : I think I'm getting close, but I see there is something wrong, the character is seeing from his side and when I rotate him he is still seeing.

float maxDistSq = maxDistance * maxDistance;
    float dist = D3DXVec3LengthSq(&(target->Position() - character->GetPosition()));
    if(dist > maxDistSq)
        return false;

D3DXVECTOR3 modelDir;
D3DXMATRIX ZUnitVec;
D3DXMATRIX matRot;
D3DXMatrixTranslation(&ZUnitVec, 0.0f, 0.0f, 1.0f); // matrix holding position of vector pointing to positive z
D3DXMatrixRotationYawPitchRoll(&matRot, D3DXToRadian(character->rotationX), D3DXToRadian(character->rotationY), D3DXToRadian(character->rotationZ));
D3DXMATRIX m = ZUnitVec * matRot;
modelDir = D3DXVECTOR3(m._41,m._42,m._43); // Get vector from the matrix
    D3DXVec3Normalize(&modelDir, &modelDir);

    D3DXVECTOR3 view = modelDir - model->Position();
    D3DXVec3Normalize(&view, &view);
   
    D3DXVECTOR3 toChar2 = target->Position() - character->Position();
    D3DXVec3Normalize(&toChar2, &toChar2);

    float cosAngle = D3DXVec3Dot(&toChar2, &view);
float cosMaxAngle = 0.5f;
    if(cosAngle < cosMaxAngle)
        return false;
    return true;

I gave you the working code, all you have to do is to fill it with correct params, but you mix (screw) it up with your code and expect it to work.

Your variable naming does not make sense.

1. Why do you name it modelDir (direction) when you extract position from matrix?

2. Why do you normalize position vector?

modelDir = D3DXVECTOR3(m._41,m._42,m._43); // Get vector from the matrix
    D3DXVec3Normalize(&modelDir, &modelDir);

3. Why do you use ZUintVec matrix and matRot for?

4. Why do you reverse the order of matrix multiplication?

D3DXMATRIX m = ZUnitVec * matRot; // m = translation * rotation should be m = rotation * translation if you want to rotate about mesh origin

5. What represent target and what is character?

I think there is something confusing here.

target = The model which I want to check if the character see or not, target->Position = the model position which the character might be looking at.

modelDir = I assume that's the direction the character is looking it? This calculation should be based on the character ROTATION, since if the character rotated, he won't be looking at the same direction and therefore might not be looking at the target model.

character = This is the character itself that want to see if he is seeing "target", so character->Position is the position of the character in 3D world.

That is a bit clearer, then this should work (assuming that you have created your model facing positive Z, positive Y is up and positive X is right):

float maxDistSq = maxDistance * maxDistance;
    float dist = D3DXVec3LengthSq(&(target->GetPosition() - character->GetPosition()));
    if(dist > maxDistSq)
        return false;
 
    D3DXMATRIX temp;
    D3DXMatrixRotationYawPitchRoll(&temp, D3DXToRadian(character->rotationX), D3DXToRadian(character->rotationY), D3DXToRadian(character->rotationZ));
    D3DXVECTOR3 view( temp.m[2][0], temp.m[2][1], temp.m[2][2] );
    D3DXVec3Normalize(&view, &view);
   
    D3DXVECTOR3 toTarget = target->GetPosition() - character->GetPosition();
    D3DXVec3Normalize(&toTarget, &toTarget);

    float cosAngle = D3DXVec3Dot(&toChar2, &view);
float cosMaxAngle = 0.5f;
    if(cosAngle < cosMaxAngle)
        return false;
    return true;

Okay that works, except that the character see from his back instead of his front, I have negative Z for moving forward, hmmm should I make it postivie Z or keep it as it is?

Anyway, I tried:

D3DXVECTOR3 view( -temp.m[2][0], -temp.m[2][1], -temp.m[2][2] );

And now it's working as expected with negative Z for Forward.

Since I have cosMaxAngle = 0.5f is the same as 90 degree, I want to give the number in degrees instead, so I want to be:

float cosMaxAngle = GetMaxCosAngle(90); // Should return 0.5f

I am really really sorry, looks like i gave you cos for 60 not 90 degree. Need to learn more about unit circle.

This should be your function to get cos from degree angle for view cone:

float GetMaxCosAngle(float angleDegree)
{
    return std::cos( D3DXToRadian(angleDegree) * 0.5f );
}

Great

One last thing, the character should never see the model if it's too high, for example: the model is above him.

Can you give me some idea on how I should do restriction to this?