I have a Camera class like this:

class Camera
{
    public:
        Matrix4x4& GetMatrix(float& interpolation); 
    private:
        Vector3 _cameraPosition;
        Vector3 _cameraTarget;
        Vector3 _cameraOrientation;

        Vector3 _cameraForward;
        Vector3 _cameraUp;
        Vector3 _cameraRight;

        Matrix4x4 _cameraMatrix;
};

And GetMatrix(interpolation) does this:

Matrix4x4& Camera::GetMatrix(float& interpolation)
{
    Vector3 currentPosition;
    currentPosition.x = _cameraPosition.x + ((cameraPosition.x - _cameraPosition.x) * interpolation);
    currentPosition.y = _cameraPosition.y + ((cameraPosition.y - _cameraPosition.y) * interpolation);
    currentPosition.z = _cameraPosition.z + ((cameraPosition.z - _cameraPosition.z) * interpolation);

    Vector3 currentTarget;
    currentTarget.x = _cameraTarget.x + ((cameraTarget.x - _cameraTarget.x) * interpolation);
    currentTarget.y = _cameraTarget.y + ((cameraTarget.y - _cameraTarget.y) * interpolation);
    currentTarget.z = _cameraTarget.z + ((cameraTarget.z - _cameraTarget.z) * interpolation);

    _cameraMatrix[0][0] = 1.0f;
    _cameraMatrix[1][1] = 1.0f;
    _cameraMatrix[2][2] = 1.0f;
    _cameraMatrix[3][3] = 1.0f;

    _cameraForward = Math::Normalize(currentTarget - currentPosition);
    _cameraUp = Math::Normalize(cameraOrientation);
    _cameraRight = Math::Normalize(Engine::Math::Cross(_cameraForward, _cameraUp));
    _cameraUp = Math::Cross(_cameraRight, _cameraForward);

    _cameraMatrix[0][0] = _cameraRight.x;
    _cameraMatrix[1][0] = _cameraRight.y;
    _cameraMatrix[2][0] = _cameraRight.z;

    _cameraMatrix[0][1] = _cameraUp.x;
    _cameraMatrix[1][1] = _cameraUp.y;
    _cameraMatrix[2][1] = _cameraUp.z;

    _cameraMatrix[0][2] = -_cameraForward.x;
    _cameraMatrix[1][2] = -_cameraForward.y;
    _cameraMatrix[2][2] = -_cameraForward.z;

    _cameraMatrix[3][0] = -Math::Dot(_cameraRight, currentPosition);
    _cameraMatrix[3][1] = -Math::Dot(_cameraUp, currentPosition);
    _cameraMatrix[3][2] = Math::Dot(_cameraForward, currentPosition);

    return _cameraMatrix;
}

The GetMatrix() function is from GLM, with the interpolation part that I added.

It works perfectly for tests and such, by setting the _cameraTarget and _cameraPosition.

I'm adding camera controls and this is where I'm getting problems. I'm adding rotation to the camera but it's not working as expected... what I tried:

void Camera::RotateAroundY(float angleDegrees)
{
    float angleRadians = angleDegrees * (PI / 180);

    float cAngle = cosf(angleRadians/2);
    float sAngle = sinf(angleRadians/2);
		
    Quaternion tRotation;
    tRotation.w = cAngle;
    tRotation.x = 0;
    tRotation.y = sAngle;
    tRotation.z = 0;

    //Finding the rotated direction of my camera
    _cameraDirection = (cameraTarget - cameraPosition) * tRotation; //Vector3 * Quaternion

    //Setting it back to cameraTarget as well
    cameraTarget = cameraPosition + _cameraDirection;
}

With also a RotateAroundX function.

Standing still at 0,0,0, these functions work well alone. If I use only RotateAroundY I can navegate around and I didn't noticed anything wrong. Same for X.

The problem starts when I try to combine both, the movement gets really weird... If I'm looking to the right from the Origin (1, 0, 0), the Look Up/Down doesn't move at all. When I'm looking to the back (0, 0, 1), the Up/Down rotation is inversed.

I was calculating the rotation not when the functions are called but after, so I just stored them in a Vector3 rotation, and then calculated a single Quaternion with the GLM Quaternion from Vector3 (euler angles), but I started getting this result, so I separated them in two function to try to pinpoint the issue but I still can't figure why.

I managed to get around this by creating a new Vector3 called "direction" which I calcule every time the rotation changes with:

_camera.directionY = sin(_camera.rotationX * 3.1415/180);		
_camera.directionX = sin(_camera.rotationY * 3.1415/180) * cos(_camera.rotationX * 3.1414/180);
_camera.directionZ = cos(_camera.rotationX * 3.1415/180) * cos(_camera.rotationY * 3.1415/180); 

But I have to constantly check if rotationX/Y >= 360, and I'd like to keep using the Quaternion and LookAt functions from GLM...

Any ideas on how I can solve this?