I figured them how to rotate camera matrix to horizon frame when view on planet surface. I studied rotation matrices on LearnOpenGL and Wikipedia website. I created code for horizon reference frame.

R = zRotate(lat) * yRotate(lng);

I wrote my own function calls from one of websites.

// rotation matrices for right-handed rule
template <typename T>
inline glm::dmat3 xRotate(T radians)
{
    double sang = sin(radians), cang = cos(radians);
    return glm::dmat3(
        { 1.0,   0.0,   0.0  },
        { 0.0,   cang, -sang },
        { 0.0,   sang,  cang }
    );
}

template <typename T>
inline glm::dmat3 yRotate(T radians)
{
    double sang = sin(radians), cang = cos(radians);
    return glm::dmat3(
        { cang,  0.0,   sang  },
        { 0.0,   1.0,   0.0   },
        {-sang,  0.0,   cang  }
    );
}

template <typename T>
inline glm::dmat3 zRotate(T radians)
{
    double sang = sin(radians), cang = cos(radians);
    return glm::dmat3(
        { cang, -sang,  0.0   },
        { sang,  cang,  0.0   },
        { 0.0 ,  0.0,   1.0   }
    );
}

It worked but view is tilted 90 degrees counterclockwise. Surface is on left-side screen. I tried different coordinates like (0,0), (40,0), (40,40) etc., and it resulted the same but different surface. It looks good but view always was tilted 90 degrees counterclockwise. Does anyone know formula to rotate camera matrix to horizon level from north pole instead of equator? My code rotates camera view origin from (0, 0) equator on UTC line.

Also, I tried to rotate camera around surface, but it did not follow horizon frame but still follow global frame regardless of any rotation matrix.

double cphi =   cos(go.phi),   sphi = -sin(go.phi);
double ctheta = cos(go.theta), stheta = sin(go.theta);

rrot = { cphi, sphi*stheta, -sphi*ctheta,
         0.0, ctheta, stheta,
         sphi, -cphi*stheta, cphi*ctheta };

grot = R * rrot;

It did not work. I tried to rotate camera around surface but it still follows global frame instead of horizon frame (or reference frame). Does anyone know any code to rotate camera matrix to reference frame (or horizon frame)?

My advice: don't use Euler angles for this. Use pure vector math to determine a 3x3 basis on the sphere surface, then if you need normal camera control multiply that matrix with another that rotates relative to the sphere surface.

Ok, I will use quaternions instead.

Ok, I tried quaternion but faced some problems.

go.Q = zqRotate(go.lat) * yqRotate(go.lng);

I wrote my quaternion rotate function calls.

template <typename T>
inline glm::dquat xqRotate(T radians)
{
    T ang = radians * T(0.5); // half angle
    return glm::dquat(cos(ang), sin(ang), 0, 0);
}

template <typename T>
inline glm::dquat yqRotate(T radians)
{
    T ang = radians * T(0.5); // half angle
    return glm::dquat(cos(ang), 0, sin(ang), 0);
}

template <typename T>
inline glm::dquat zqRotate(T radians)
{
    T ang = radians * T(0.5); // half angle
    return glm::dquat(cos(ang), 0, 0, sin(ang));
}

gqrot = glm::conjugate(go.Q) * glm::conjugate(cam.rqrot);
grot  = glm::mat3_cast(gqrot);

I tried (0, 0), (20, 0), (40, 0), etc. It worked but I still rotate camera in global frame.

I tried (20, 20), (40, 40), etc. It did not work because it tilted clockwise and down. Quaternion behaves weird comparing with rotation matrix above. Higher degrees and weirder behave (higher tilted clockwise and down).

That's why I want to rotate camera in local frame, not global frame. I tried to remove glm::conjugate but it behaves weirder.

I have some books to figure them out… Does anyone have any information about local horizon frame with quaternions?

Aressera said:
My advice: don't use Euler angles for this. Use pure vector math to determine a 3x3 basis on the sphere surface, then if you need normal camera control multiply that matrix with another that rotates relative to the sphere surface.

I second this advice.
A typical application is to construct a ‘look at’ matrix, with related tutorials easy to find.
The concept of 3D rotation is not needed, since we construct orientation directly.
That's very nice, because 3D rotations tend to be difficult.

So i assume you have a spherical world, and you want to know the orientation a player or spaceship has at any point, so it is oriented like walking on the planets surface.
This means it should stand upright, aligned to the direction of gravity.

Mat4x4 CalcReferenceFrameOnSphere (vec3 sphCenter, Mat4x4 currentObjectTransfrom, vec3 newObjectPosition)
{
	vec3 up = (newObjectPosition - sphCenter).Normalized(); // we have our up vector
	
	vec3 front = currentObjectTransfrom[2]; // assuming z is forwards direction of our object, which we want to keep the same
	
	vec3 side = front.Cross(up).Normalized(); // we have our left (or right) vector (i never momorize which we get, so i need to do trial and error eventually, by swapping the operands for the cross product)
	
	front = up.Cross(side); // upaditng the front direction so it is perpendicular to up and side (again not sure if i accidentally get the backwards direction - swapping if so)
	
	Mat4x4 result = Mat4x4::Identity();
	result[0] = side;
	result[1] = front;
	result[2] = up; // column indices are a matter of convention, and i might need to negate directions as well
	result[3] = newObjectPosition;
	
	return result;
}

We can then use our regular first / third person camera controls within the given reference frame as usual.
The player does not need to adjust forwards direction while walking over the sphere, it happens automatically.

I've written this out of my mind, so there may be bugs.
You should be able to understand how it works geometrically.
It helps to visualize up, front and side directions to do so, which also helps with the uncertainty regarding flipped directions mentioned in the comments.

The current transform is required to define a second direction, because otherwise we would know only up, and only one direction is not enough to define orientation.
Alternatively, we could calculate the inittial front direction like so:

front = newObjectPos - currentObjectPos;

But this fails if the object does not move, giving a zero vector. So it's not very robust.

However, my example also fails if any of the direction vectors become zero (player exactly at the center of the planet, player lays flat on the floor, so front and up become the same direction, etc.
Those cases are rare, but need to be handled to ensure robustness.
I did not add related code to keep it simple.

Well, I finally realized that I ended up in global frame instead, not local frame. I had to swap two rotations and all problems went away.

grot = rrot * R;

I have to rotate camera around first then applying it to horizon reference frame as final.

I tried different coordinates like (0, 0), (40, 0), (40, 40), etc. Camera rotation now aligns with horizon reference frame.

    double clat = cos(lat), slat = sin(lat);
    double clng = cos(lng), slng = sin(lng);
    R = { slat*clng,  clat*clng, slng,
         -clat,       slat,      0,
         -slat*slng, -clat*slng, clng };

I tried that horizon reference frame. My camera rotation now aligns with horizon frame, not tilted 90 degrees counterclockwise. I am now figuring out about two simple Z-axis and Y-axis rotation from that matrix.

I now learned about 3D rotation matrix a lot.

All problems had been resolved.

JoeJ said:

I second this advice.
A typical application is to construct a ‘look at’ matrix, with related tutorials easy to find.
The concept of 3D rotation is not needed, since we construct orientation directly.
That's very nice, because 3D rotations tend to be difficult.

So i assume you have a spherical world, and you want to know the orientation a player or spaceship has at any point, so it is oriented like walking on the planets surface.
This means it should stand upright, aligned to the direction of gravity.

Mat4x4 CalcReferenceFrameOnSphere (vec3 sphCenter, Mat4x4 currentObjectTransfrom, vec3 newObjectPosition)
{
	vec3 up = (newObjectPosition - sphCenter).Normalized(); // we have our up vector
	
	vec3 front = currentObjectTransfrom[2]; // assuming z is forwards direction of our object, which we want to keep the same
	
	vec3 side = front.Cross(up).Normalized(); // we have our left (or right) vector (i never momorize which we get, so i need to do trial and error eventually, by swapping the operands for the cross product)
	
	front = up.Cross(side); // upaditng the front direction so it is perpendicular to up and side (again not sure if i accidentally get the backwards direction - swapping if so)
	
	Mat4x4 result = Mat4x4::Identity();
	result[0] = side;
	result[1] = front;
	result[2] = up; // column indices are a matter of convention, and i might need to negate directions as well
	result[3] = newObjectPosition;
	
	return result;
}

Yeah. I already tried that, and it worked for me. I now learned about local/global frame ordering. That is another method for horizon reference frame. To rotate camera, person, etc in local frame.

<global position> = <local position> * <horizon frame>;

Well, I tested many things, and everything worked so well. That is final update.

// Set rotation matrix for local horizon frame
// for right-handed rule (OpenGL). Points
// to east as origin at (0, 0).
//
//     |  slat  clat   0  | |  clng   0   slng |
// R = | -clat  slat   0  | |   0     1    0   |
//     |   0     0     1  | | -slng   0   clng |
//
double clat = cos(lat), slat = sin(lat);
double clng = cos(lng), slng = sin(lng);
R = { slat*clng,  clat*clng, slng,
     -clat,       slat,      0,
     -slat*slng, -clat*slng, clng };

That formed the horizon-level reference frame for spherical world.

// Rotate camera in local frame. Negate theta value for 
// clockwise rotation. Points to east as default origin
// so that adding pi/2.0 to theta value for pointing to 
// north with zero heading.  
rrot = xRotate(phi) * yRotate(-theta + pi/2.0);

Rotate camera by using theta (horizontal) and phi (vertical) in local frame.

grot = rrot * R * cbody->getuOrientation();

Applying rrot (camera view rotation in local frame) to horizon-level reference frame for world space.

It worked so well. I added some code for moving camera around spherical world.

// Rotate movement at the direction of camera view.
dm = rrot * dm;

// Updating ground observer
lat += dm.z;
lng += dm.x;

    
// Set rotation matrix for local horizon frame
// for right-handed rule (OpenGL). Points
// to east as origin at (0, 0).
double clat = cos(lat), slat = sin(lat);
double clng = cos(lng), slng = sin(lng);
R = { slat*clng,  clat*clng, slng,
     -clat,       slat,      0,
     -slat*slng, -clat*slng, clng };

Processing inputs from user each frame. Updating dm (x and z axis) for movement. Applying dm to a direction of camera view matrix for moving forward. Then applying lat and lng with dm vector. Updating R each time.

It worked so well that I was able move around spherical world and look at starry night. You will see curvature of world. Sorry, earth flatters!