In FPS game, I'm trying to make the enemies detect vision on the player.

What is the best way to detect vision by a character using ray-casting?

Here's some ideas:

1. First check to see if the player is within the cone of vision from the enemy (i.e. is the player close enough and in front). If not then there's no need for a ray cast.

2. All ray casts should probably start from the enemies eye. It's the target on the player that's harder to choose.

3. To minimize the number of ray casts one option is to remember what happened last time you did a ray cast. If the ray cast hit the player then repeat a ray cast to the same target next time. If not then cycle through targets like head, hands, feet, torso, weapon, etc. A few frames of latency for detection of a partly obscured target shouldn't be an issue, you'll probably want to add some artificial latency anyway to simulate the reaction time of a human.

4. You may want to let enemies that are actively attacking the player do more ray casts than others.

Hi Adam,

Can I create a cone shape starting from the enemy eyes to detect collision and once the collision is detected I do raycasting from the enemy eyes towards the player to check if the enemy got direct sight on the player?

Can I create a cone shape starting from the enemy eyes to detect collision and once the collision is detected I do raycasting from the enemy eyes towards the player to check if the enemy got direct sight on the player?

Simple math can probably be enough to do the vision cone check.

int fastVisibilityCheck(float4 playerPos, float4 enemyPos, float4 enemyLookDir,
                        float maxViewDistance, float cosHalfMaxAngle)
{
     //Vector from enemyPos to playerPos
     float4 view = playerPos-enemyPos;
     float distance = sqrt(view.x^2 + view.y^2 + view.z^2);

     if(distance > maxViewDistance)
          return 0; //not visible

     view = normalize(view);
    
     //cosine of the angle between the view vector and the direction the enemy is looking
     float cosAngle = enemyLookDir dot view;

     //check if player is inside the cone
     if(cosAngle < cosMaxAngle)
          return 0; //not visible

     return 1; //visible
}

Use the previous function like this:

for(int i = 0; i < enemies.size(); i++)
{
     if(fastVisibilityCheck(player.pos, enemies[i].pos, enemies[i].lookDir, enemies[i].maxViewD, cos(enemies[i].fieldOfView/2)))
     {
          //use raycasting to check if enemy can really see the player
     }
}

@TiagoCosta: Need a little bit of more clarification.

playerPos is the beginning of the cone (enemy eyes position)? Is that true?

someNumber is how far the enemy can see.

someOtherNumber: How do I get this value?

@TiagoCosta: Need a little bit of more clarification.

playerPos is the beginning of the cone (enemy eyes position)? Is that true?
someNumber is how far the enemy can see.
someOtherNumber: How do I get this value?

How do I calculate enemyLookDir from the rotation? I mean the enemy look direction depends on its own rotation.

Another thing is: Does this work well without performance issues? I have to iterate through all the models to check for vision on every one, in performance manner, is that way better or using a cone shape and detecting collision?

Enemy1 -> iterate through all the models every frame to detect vision

Enemy2 -> iterate through all the models every frame to detect vision

Enemy3 -> iterate through all the models every frame to detect vision

......

Enemy100 -> iterate through all the models every frame to detect vision

Etc...