Hello,

I'm trying to get view and projection matrices for directional light for shadow map. I'm confused about getting rotation part for rotating/multiplying corners. Currently this seems work:

GkCamera*
gkCameraForLight(GkScene *scene, GkLight *light) {
  mat4      view, proj;
  GkCamera *cam;

  cam = scene->camera;

  switch (light->type) {
    case GK_LIGHT_TYPE_DIRECTIONAL: {
      vec4   *c;
      vec3    box[2], he, v;
      int32_t i;

      memset(box, 0, sizeof(box));

      gkLightRotation(scene, light, view[0], view[1], view[2]);
      
      glm_vec_broadcast(0.0f, view[3]);
      view[0][3] = view[1][3] = view[2][3] = 0.0f;
      view[3][3] = 1.0f;

      c = cam->frustum.corners;
      for (i = 0; i < 8; i++) {
        glm_vec_rotate_m4(view, c[i], v);

        box[0][0] = glm_min(box[0][0], v[0]);
        box[0][1] = glm_min(box[0][1], v[1]);
        box[0][2] = glm_min(box[0][2], v[2]);

        box[1][0] = glm_max(box[1][0], v[0]);
        box[1][1] = glm_max(box[1][1], v[1]);
        box[1][2] = glm_max(box[1][2], v[2]);
      }

      glm_vec_sub(box[1], box[0], he);
      glm_vec_scale(he, 0.5f, he);

      glm_mat4_copy(GLM_MAT4_IDENTITY, proj);
      proj[0][0] = 1.0f / he[0];
      proj[1][1] = 1.0f / he[1];
      proj[2][2] = 1.0f / he[2];

      /* create view matrix: there is no position */
      glm_mat4_transpose(view);
      break;
    }
    case GK_LIGHT_TYPE_POINT:
    case GK_LIGHT_TYPE_SPOT:

    default:
      break;
  }

  return gkMakeCamera(proj, view);
}

/* ------ */

void
gkLightRotation(GkScene *scene,
                GkLight *light,
                vec3     right,
                vec3     up,
                vec3     fwd) {
  GkTransform *trans;

  glm_vec_copy(light->direction, fwd);
  glm_vec_copy(light->up,        up);

  if ((trans = gkLightTransform(light))) {
    glm_vec_rotate_m4(trans->world, fwd, fwd);
    glm_vec_rotate_m4(trans->world, up,  up);
  }

  glm_vec_cross(fwd, up, right);

  glm_vec_normalize(right);
  glm_vec_normalize(up);
  glm_vec_normalize(fwd);
}

I was getting rotation matrix (for getting bbox for light) with this by following question / answers:

gkLightDirWorld(scene, light, view[2]); /* forward vector */
glm_vec_normalize(view[2]);

glm_vec_cross(view[2], GLM_ZUP, view[0]); /* right vector */
glm_vec_normalize(view[0]);

glm_vec_cross(view[2], rot[0], view[1]); /* up vector */
glm_vec_normalize(view[1]);

this also works. Since I don't know how it works because we need least two vector for cross product, right? Z_UP (0, 0, 1) seems arbitrary vector :-S

So I stored default UP (0, 1, 0) vector in light struct as I did for default light direction (0, 0 -1), now every light has direction and up vector, right vector is computed as cross product. They will be rotated when every transform changed.  Is it good idea to store extra UP vector since we can get like above? How do you get UP vector if you do similar? if not what is best way to get projection/view matrix for directional light? 

My TODOS:
 - I'll iterate all objects and create bbox which surrounds all objects instead of taking camera's frustum corners

Up vector is usually a bad idea with gimbal lock issues. For an orientation, please use a quaternion !

light or cameras are the same, a product of linear transformation and a projection, stored in matrix in game engines as it is convenient. to travel one reference frame to an other, all you do is multiply a sequence of matrices holding the transformations.

10 hours ago, galop1n said:

Up vector is usually a bad idea with gimbal lock issues. For an orientation, please use a quaternion !

Thank you for your comments, but I don't understand how can I use quat here? The only think I have is light direction to build rotation matrix :-S Is it possible to create a quaternion from light direction? If light direction is axis then what is the angle to pass quat?

@galop1n I'm reconsidering to use quaternion here, did you mean replace light direction with quaternion? Sounds interesting, light would only store orientation instead of storing direction, up and right vectors

First thing I need is converting default light direction [0, 0, -1] to quaternion I think:

/* this assumes light direction is [0, 1, 0] and applies rotation */

versor defaultLightOri;
glm_quatv(defaultLightOri, -M_PI_2, GLM_XUP);

then I have to update this by light node's transform matrix if it changes. Then the only thing I need is rotating frustum corners by this updated quaternion, cool ? We're talking about the same thing, right? 

1 hour ago, Mike2343 said:

I assume you're using pure C for whatever you're working on?

Yes I'm using pure C to build graphics engine: http://github.com/recp/gk then C++ wrappers / helpers can be written on top of it later

Also
       cglm (math): https://github.com/recp/cglm
        AssetKit (importer): https://github.com/recp/assetkit
        todo: physics, ai ...

READMEs may not be up to date, unlike cglm they are still in progress

1 hour ago, Mike2343 said:

And yes, pretty sure they were referring to using a quaternion here to eliminate the up/right vector requirements.

thank you for clarification, I think I need to keep store light direction (or forward vector) because it needs in shaders, I will try to use quaternion to represent light orientation instead of storing individual vectors ? I may come back later with extra questions

You do not have to keep the direction separately ( and duplicated data is a source of bug as you have to deal with keeping them in sync ), a vector rotation transform by a quaternion is only a few mul and adds. In your case, even less work as what you want is to rotate (0,0,1) to get back the direction and it nullify many terms.

You did not need quaternion to solve your initial problem, they are just an extra tool to your arsenal, and you are likely to need them at some point. What you are aiming for is good understanding of the euclidean space and trigonometry, nothing more in order to get your mind over the geometry problems you encounter in a 3D engine

If you want a few reason why quaternion are useful :

* Solve the gimbal lock issue on most camera system ( camera loosing a degree of freedom )

* Small memory footprint ( 4 values ). A full 4x4 world matrix (rot/trans/scale) is 64B, while a q+trans+scale is 32/40B ( homogenous scale or not )

* Premium for animation using slerp, an interpolation method with a constant speed ( soon in DXIL 6.1 with custom interpolator ).

* Easy to negate, multiply, …

1 hour ago, galop1n said:

You do not have to keep the direction separately ( and duplicated data is a source of bug as you have to deal with keeping them in sync ), a vector rotation transform by a quaternion is only a few mul and adds. In your case, even less work as what you want is to rotate (0,0,1) to get back the direction and it nullify many terms.

What is default quaternion value for light direction if I shouldn't store direction it self? Default light direction is [0, 0, -1] (OpenGL RH) so in previous post I tried to get this by selecting [0, 1, 0] and rotate around X by 90deg, so it gives [0, 0, -1]. I thought that if all nodes have IDENTITY matrix then I can use this default rotation to rotate frustum corners to get bbox to build directional light's frustum

If you say use IDENTITY quat for default direction then how can I rotate frustum corners then? I would need to default light direction and ... 

I don't understand what [0, 0, 1] means, sounds like [0, 0, -1] in DirectX, LH? Rotate around what? Which axis, which angle.. 
Light (actually light->node) has 4x4 matrix it is not quat. I can't convert it to quat totally because non-uniform scaling must be supported

1 hour ago, galop1n said:

You did not need quaternion to solve your initial problem, they are just an extra tool to your arsenal, and you are likely to need them at some point. What you are aiming for is good understanding of the euclidean space and trigonometry, nothing more in order to get your mind over the geometry problems you encounter in a 3D engine

If you want a few reason why quaternion are useful :

* Solve the gimbal lock issue on most camera system ( camera loosing a degree of freedom )

* Small memory footprint ( 4 values ). A full 4x4 world matrix (rot/trans/scale) is 64B, while a q+trans+scale is 32/40B ( homogenous scale or not )

* Premium for animation using slerp, an interpolation method with a constant speed ( soon in DXIL 6.1 with custom interpolator ).

* Easy to negate, multiply, …

I think my actual problem is shadow map  I like math and I'm trying understand, learn, invent ... as possible as I can. I know about quaternions, if I need learn / re-learn... then I will, sure  

Also it is not fear to compare 4x4 with quat I think, actual rotation is 3x3 (quat is winner here ok) and it can contain more information than quat e.g. non-uniform scaling, skewing

When I'm confused my questions may get weird, thanks for your patience

24 minutes ago, recp said:

I don't understand what [0, 0, 1] means, sounds like [0, 0, -1] in DirectX, LH? Rotate around what? Which axis, which angle.. 
Light (actually light->node) has 4x4 matrix it is not quat. I can't convert it to quat totally because non-uniform scaling must be supported

It is a unit vector, your light orientation is a quaternion ( it encodes an axis + angle ), so if you transform the unit vectors (1,0,0), (0,1,0), and (0,0,+/-1), you get back a 3x3 rotation matrix, and likely the last row is your light direction ( in 99% engine, idTech view vector is X for historical reasons… ). So why i said it is not necessary to store it separately. 

In my size comparison, that was fair btw, i grouped the quaternion with a translation scales, and i even did not mention that you do not need full precision floats for a quaternion if you really want to push it ( i pack the TBN of my meshes in a single HALF4 + sign for example, other even encode it in 32bits total ). And no, you do not want skewing, you do not even want non uniform scale in an engine as it leads to malformed tangent space and incorrect lighting