Hi there!

I'm really struggling here trying to wrap my brain around these different kinds of math involving Quaternions, Matrices, Vectors and whatnot. But right now, I'm just trying to create an FPS style camera using Quaternions. I'm using GLM as my math library with OpenGL. Don't let the naming in the code I show fool you, as I just can't stand the abbreviated vec3, quat, mat4, and whatnot, so I'm typedeffing them.

Right now I seem to have mouse rotation mostly working, but the more I move the mouse, the more it seems like the triangle on the screen, rolls. I am normalizing the Quaternion, so I'm not sure what's going on here. All the places I've looked about mouse input seemed to have a lot of spaghetti looking code, so I don't know if I'm doing this quite right. Translating mouse delta values to rotations isn't quite clear to me.

This is the triangle before I move the mouse:

And here it is after I move the mouse around a little bit:

Now to dive in the code. This is currently my Camera's update function

void Camera::Update()
{
	if ( g_pInput->IsButtonPressed( "Forward" ) )
	{
		SetPosition( GetPosition() + m_vForward * 0.01f * g_pEngine->GetDeltaTime() );
	}
	else if ( g_pInput->IsButtonPressed( "Backward" ) )
	{
		SetPosition( GetPosition() + -m_vForward * 0.01f * g_pEngine->GetDeltaTime() );
	}
	if ( g_pInput->IsButtonPressed( "Left" ) )
	{
		Vector3f m_vLeft = glm::cross( m_vUp, m_vForward );
		SetPosition( GetPosition() + m_vLeft * 0.01f * g_pEngine->GetDeltaTime() );
	}
	else if ( g_pInput->IsButtonPressed( "Right" ) )
	{
		Vector3f m_vRight = glm::cross( m_vForward, m_vUp );
		SetPosition( GetPosition() + m_vRight * 0.01f * g_pEngine->GetDeltaTime() );
	}

	m_flFrameYaw = 0.02f * g_pInput->GetMouseDeltaX() * g_pEngine->GetDeltaTime();
	m_flFramePitch = 0.02f * g_pInput->GetMouseDeltaY() * g_pEngine->GetDeltaTime();

	Quaternion qFrame = Quaternion( Vector3f( m_flFramePitch, m_flFrameYaw, m_flFrameRoll ) );

	m_flFramePitch = 0.0f;
	m_flFrameYaw = 0.0f;
	m_flFrameRoll = 0.0f;

	m_qRotation = qFrame * m_qRotation;
	m_qRotation = glm::normalize( m_qRotation );

	Matrix4f mat4Rotate = glm::toMat4( m_qRotation );

	Matrix4f mat4Translate = glm::translate( GetPosition() );

	m_mat4ViewMatrix = mat4Rotate * mat4Translate;
}

My input code at the start of the function is a bit old as I was just using glm's lookat function before, but now I want update my position based on the Quaternion rotation. So one of my questions is, how do I translate the Quaternion rotation into a Vector representing a direction? I'm guessing that's what I want to do so when I press the "forward" button I move some amount + a forward direction Vector.

My other question is am I doing the mouse rotation wrong?

For the sake of transparency, here's my Input class definitions, the relevant part being the Update()

Input::Input()
{
	m_pCurrentKeyboardState = nullptr;
	m_pOldKeyboardState = nullptr;
}

Input::~Input()
{
	if ( m_pCurrentKeyboardState )
		delete m_pOldKeyboardState;
}

bool Input::Init()
{
	m_pCurrentKeyboardState = SDL_GetKeyboardState( &m_iNumKeys );
	m_pOldKeyboardState = new Uint8[ m_iNumKeys ];

	// TODO: Create config file with dynamic button mapping
	m_mapButtons[ "Forward" ] = SDL_SCANCODE_W;
	m_mapButtons[ "Backward" ] = SDL_SCANCODE_S;
	m_mapButtons[ "Left" ] = SDL_SCANCODE_A;
	m_mapButtons[ "Right" ] = SDL_SCANCODE_D;
	m_mapButtons[ "Jump" ] = SDL_SCANCODE_SPACE;
	m_mapButtons[ "Quit" ] = SDL_SCANCODE_ESCAPE;
	m_mapButtons[ "TurnLeft" ] = SDL_SCANCODE_LEFT;
	m_mapButtons[ "TurnRight" ] = SDL_SCANCODE_RIGHT;

	return true;
}

// May want to do checks to see if a button name is valid
bool Input::IsButtonPressed( const std::string &strButtonName )
{
	return ( m_pCurrentKeyboardState[ m_mapButtons[ strButtonName ] ] == 1 );
}

bool Input::IsButtonReleased( const std::string &strButtonName )
{
	return !IsButtonPressed( strButtonName );
}

bool Input::IsButtonJustPressed( const std::string &strButtonName )
{
	return ( m_pCurrentKeyboardState[ m_mapButtons[ strButtonName ] ] == 1 && m_pOldKeyboardState[ m_mapButtons[ strButtonName ] ] == 0 );
}

bool Input::IsButtonJustReleased( const std::string &strButtonName )
{
	return ( m_pCurrentKeyboardState[ m_mapButtons[ strButtonName ] ] == 0 && m_pOldKeyboardState[ m_mapButtons[ strButtonName ] ] == 1 );
}

void Input::Update()
{
	m_flMouseDeltaX = 0.0f;
	m_flMouseDeltaY = 0.0f;

	// Note: We allocate the old keyboard state in the Init function
	// Also no need to call SDL_GetKeyboardState as the state ponter is updated when events are processed, hence SDL_PumpEvents()
	std::memcpy( m_pOldKeyboardState, m_pCurrentKeyboardState, sizeof( Uint8 ) * m_iNumKeys );
	SDL_PumpEvents();
	
	while ( SDL_PollEvent( &m_event ) )
	{
		switch ( m_event.type )
		{
		case SDL_MOUSEMOTION:
			m_flMouseDeltaX = ( float ) m_event.motion.xrel;
			m_flMouseDeltaY = ( float ) m_event.motion.yrel;
			break;
		}
	}
}

As you can see I'm using SDL, SDL 2.0.5 to be exact. Ideally I would like to calculate the mouse delta values without polling events like that, but whatever works I guess.

Lastly, I think it's worth mentioning how I compute the model matrix. This is currently done using Vectors, but I would like to do it with Quaternions for consistency sake.

inline Matrix4f GetModelMatrix() const 
{
  Matrix4f posMatrix = glm::translate( m_vPosition );
  Matrix4f rotXMatrix = glm::rotate( m_vRotation.x, Vector3f( 1.0f, 0.0f, 0.0f ) );
  Matrix4f rotYMatrix = glm::rotate( m_vRotation.y, Vector3f( 0.0f, 1.0f, 0.0f ) );
  Matrix4f rotZMatrix = glm::rotate( m_vRotation.z, Vector3f( 0.0f, 0.0f, 1.0f ) );
  Matrix4f scaleMatrix = glm::scale( m_vScale );

  Matrix4f rotMatrix = rotZMatrix * rotYMatrix * rotXMatrix;

  return posMatrix * rotMatrix * scaleMatrix;
}

I've already thrown a lot of code up, so I guess I'll leave it at that for now. If there's nothing that anybody finds particularly wrong, I can show stuff like the shader code and whatnot. However I'm sure that's okay.

Any help is greatly appreciated though.

3D rotation control with a mouse can be tricky.  I wouldn't use quats.  It looks like you are just using Euler angles derived from the mouse movement, which is probably the best way to do it actually.  You just need some tweaks.  Here is the simple 2-step rotation I use:

Step 1: Use the X mouse movement to rotate the camera matrix around the WORLD Y axis.

Step 2: Then use the Y mouse movement to rotate the camera matrix around its newly rotated LOCAL X axis.

And that's it.

As for your model matrix, maybe it's just me, but it seems odd that you are calculating each axis of the rotation matrix individually and then assembling them into a Voltron matrix, rather than just rotating the whole matrix.  And I think you were over-complicating it with all of the matrix multiplication.  

Anyway, I'm not super familiar with GLM, but I think the camera code would look roughly like this:

  Matrix4f camera = glm::mat4();

  m_flFrameYaw = 0.02f * g_pInput->GetMouseDeltaX() * g_pEngine->GetDeltaTime();
  m_flFramePitch = 0.02f * g_pInput->GetMouseDeltaY() * g_pEngine->GetDeltaTime();

  // do the Y rotation first, in world-space
  camera = glm::rotate( camera, m_flFrameYaw, Vector3f( 0.0f, 1.0f, 0.0f ) );

  // next do the X rotation in local-space
  camera = glm::rotate( camera, m_flFramePitch, Vector3f( camera[0][0], camera[0][1], camera[0][2] ) );

  // do translation last
  camera = glm::translate( camera, GetPosition() );

I had to modify that a little bit to get it to somewhat work, but even that is wonky. This is the current code:

m_mat4ViewMatrix = glm::mat4();

m_flFrameYaw += 0.02f * g_pInput->GetMouseDeltaX() * g_pEngine->GetDeltaTime();
m_flFramePitch += 0.02f * g_pInput->GetMouseDeltaY() * g_pEngine->GetDeltaTime();

m_mat4ViewMatrix = glm::rotate( m_mat4ViewMatrix, m_flFrameYaw, Vector3f( 0.0f, 1.0f, 0.0f ) );

m_mat4ViewMatrix = glm::rotate( m_mat4ViewMatrix, m_flFramePitch, Vector3f( m_mat4ViewMatrix[ 0 ][ 0 ] ,  m_mat4ViewMatrix[ 0 ][ 1 ],  m_mat4ViewMatrix[ 0 ][ 2 ] ) );

m_mat4ViewMatrix = glm::translate( m_mat4ViewMatrix, GetPosition() );

Aside from the fact the mouse input is still weird, this still doesn't account for computing a forward direction based on the camera's current rotation or yaw. This method also continually increases\decreases the pitch and yaw values until I guess it would hit the maximum float value and wrap around. So I would need to clamp this somehow. It would seem like using a Quaternion would make this more manageable.

However, with this code, the triangle isn't continually rotating when I put the mouse back towards the normal view. However, when I move my mouse away from the triangle, it gets really wonky. As per my model matrix code, yes I would like to change that, however I just changed it to return a glm::translate value so I can rule out my model matrix being the culprit.

This is my scene before I move the mouse (I added a floor to make it more apparent):

And this is after:

If I try to very carefully just look across ONE axis, either up and down, or left and right, it does look somewhat normal. But if I move the mouse in both directions, it gets skewed like this.

Again if I try to move my mouse back towards the center, it looks normal again, whereas before the triangle was staying rotated.

Anything else that doesn't look right?

I can show some of my other code, for example this is my shader's update function:

void GenericShader::Update( const Transform &transform )
{
	Matrix4f modelViewProjection = g_pRenderer->GetViewPort()->GetPerspectiveProjection() * transform.GetModelMatrix();
	
	glUniformMatrix4fv( m_mapUniforms[ "transform" ], 1, GL_FALSE, &modelViewProjection[ 0 ][ 0 ] );
	glUniform4f( m_mapUniforms[ "color" ], m_vColor.r, m_vColor.g, m_vColor.b, m_vColor.a );
}

This is my viewport code:

bool ViewPort::InitPerspectiveViewPort( float flFOV, float flAspect, float flZNear, float flZFar, ICamera *pCamera )
{
	if ( pCamera == nullptr )
		return false;

	m_pCamera = pCamera;

	// TODO: Add window resize handling
	m_mat4PerspectiveMatrix = glm::perspective( flFOV, flAspect, flZNear, flZFar );

	return true;
}

void ViewPort::UpdateViewPort()
{
	m_pCamera->Update();
}

Matrix4f ViewPort::GetPerspectiveProjection()
{
	return m_mat4PerspectiveMatrix * m_pCamera->GetViewMatrix();
}

And lastly, here's my actual vertex shader code:

#version 330

layout (location = 0) in vec3 position;

uniform vec4 color;

uniform mat4 transform;
out vec4 color0;

void main ()
{	
	gl_Position = transform * vec4( position, 1.0 );
	color0 = color;
}

*Phew, I'm sorry I'm throwing so much code out there, but it's possible I'm making some other mistake I'm not aware of. But I really feel like this is just an issue with the mouse movement\rotation.

Quote

 So one of my questions is, how do I translate the Quaternion rotation into a Vector representing a direction? I'm guessing that's what I want to do so when I press the "forward" button I move some amount + a forward direction Vector.

Let's consider a camera which has identity matrix (or quaternion). If you move the camera along z axis (forward or backward in your case) you will see same scene with different scales (assuming projection is perspective). Actually I used this to implement zooming instead of scale whole scene, because in this way you only change view matrix not every node transform in scene graph

But if you rotate the camera (not around z) then your forward axis will not be Z axis anymore, you must find new axis for forward to move camera. To do that we can do somthing like this:

The defeult camera orientation is (0, 0, -1) in RH, we know total amount rotation of camera which is that top-left matrix (3x3). So if we rotate the vec3(0.0f, 0.0f, -0.1f) by the camera rotation and adding it to camera position will give you new camera direction.

More or less:

vec3 dir    = {0.0f, 0.0f, -1.0f};                  // consider this as unit vector of forward
vec3 amount = rotate(dir,  camera(Not View)Matrix); // use appropriate rotate func, this is just an example 
cameraMatrix[3] += amount; 

viewMatrix = inverse(cameraMatrix);

you must do it with every movement. Here my implementation about zoom, it uses cglm (C99): https://github.com/recp/libgk/blob/master/src/gk_camera.c#L117-L119

Probably there many ways to do that but this is my approach

If you store total amount rotation a quaternion you can rotate it by quaternion if you want. I do it with matrix because I'm storing camera (world) and view matrix

15 hours ago, MichaelBarth said:

If I try to very carefully just look across ONE axis, either up and down, or left and right, it does look somewhat normal. But if I move the mouse in both directions, it gets skewed like this.

This sounds like the X rotation isn't happening in local space.  Again, I'm not real familiar with GLM.  It looks like GLM uses column-major ordering, so you need to swap the array indices for the vector in the X rotation part.  Like this :

m_mat4ViewMatrix = glm::rotate( m_mat4ViewMatrix, m_flFramePitch, Vector3f( m_mat4ViewMatrix[ 0 ][ 0 ] ,  m_mat4ViewMatrix[ 1 ][ 0 ],  m_mat4ViewMatrix[ 2 ][ 0 ] ) );

15 hours ago, MichaelBarth said:

Aside from the fact the mouse input is still weird, this still doesn't account for computing a forward direction based on the camera's current rotation or yaw.

The camera's forward direction is already baked into the matrix.

The top left 3x3 set of numbers in the matrix is the rotation part.  Which isn't nearly as complicated as it sounds.  It just defines the X-Y-Z directions.  The first row is the X-axis (left-right), the second row is the Y-axis (up-down), and the third row is the Z-axis (forward-backward).  So at any time, you can just grab the third row of a transform matrix to get its "forward" direction.  Something like : vec3( mat[0][2], mat[1][2], mat[2][2] )

Ah, this is so cool once you understand where things are.

It does fully work now. However, I still feel like my method of clamping the yaw and pitch values is a bit silly. I'm wondering if I should refer to them in degrees instead of radians, then translate them to radians when passing it onto glm::rotate.

Hm, well anyway, this is what I ended up with:

void Camera::Update()
{
	m_mat4ViewMatrix = glm::mat4();

	m_flYaw += 0.02f * g_pInput->GetMouseDeltaX() * g_pEngine->GetDeltaTime();
	m_flPitch += 0.02f * g_pInput->GetMouseDeltaY() * g_pEngine->GetDeltaTime();

	float flYawDegrees = glm::degrees( m_flYaw );
	float flPitchDegrees = glm::degrees( m_flPitch );

	if ( flYawDegrees < 0.0f )
		m_flYaw = glm::radians( flYawDegrees + 360.0f );
	else if ( flYawDegrees > 360.0f )
		m_flYaw = glm::radians( flYawDegrees - 360.0f );

	if ( flPitchDegrees < 0.0f )
		m_flPitch = glm::radians( flPitchDegrees + 360.0f );
	else if ( flPitchDegrees > 360.0f )
		m_flPitch = glm::radians( flPitchDegrees - 360.0f );

	m_mat4ViewMatrix = glm::rotate( m_mat4ViewMatrix, m_flYaw, Vector3f( 0.0f, 1.0f, 0.0f ) );

	m_mat4ViewMatrix = glm::rotate( m_mat4ViewMatrix, m_flPitch, Vector3f( m_mat4ViewMatrix[ 0 ][ 0 ] ,  m_mat4ViewMatrix[ 1 ][ 0 ],  m_mat4ViewMatrix[ 2 ][ 0 ] ) );

	m_mat4ViewMatrix = glm::translate( m_mat4ViewMatrix, GetPosition() );

	m_vForward = Vector3f( m_mat4ViewMatrix[ 0 ][ 2 ], m_mat4ViewMatrix[ 1 ][ 2 ], m_mat4ViewMatrix[ 2 ][ 2 ] );
	m_vUp = Vector3f( m_mat4ViewMatrix[ 0 ][ 1 ], m_mat4ViewMatrix[ 1 ][ 1 ], m_mat4ViewMatrix[ 2 ][ 1 ] );
	m_vLeft = Vector3f( m_mat4ViewMatrix[ 0 ][ 0 ], m_mat4ViewMatrix[ 1 ][ 0 ], m_mat4ViewMatrix[ 2 ][ 0 ] );

	m_vBackward = -m_vForward;
	m_vRight = -m_vLeft;

	if ( g_pInput->IsButtonPressed( "Forward" ) )
	{
		SetPosition( GetPosition() + m_vForward * 0.01f * g_pEngine->GetDeltaTime() );
	}
	else if ( g_pInput->IsButtonPressed( "Backward" ) )
	{
		SetPosition( GetPosition() + m_vBackward * 0.01f * g_pEngine->GetDeltaTime() );
	}
	if ( g_pInput->IsButtonPressed( "Left" ) )
	{
		SetPosition( GetPosition() + m_vLeft * 0.01f * g_pEngine->GetDeltaTime() );
	}
	else if ( g_pInput->IsButtonPressed( "Right" ) )
	{
		SetPosition( GetPosition() + m_vRight * 0.01f * g_pEngine->GetDeltaTime() );
	}
}

Well I'm happy this works. Anything I should look out for?

4 hours ago, missionctrl said:

The camera's forward direction is already baked into the matrix.

The top left 3x3 set of numbers in the matrix is the rotation part.  Which isn't nearly as complicated as it sounds.  It just defines the X-Y-Z directions.  The first row is the X-axis (left-right), the second row is the Y-axis (up-down), and the third row is the Z-axis (forward-backward).  So at any time, you can just grab the third row of a transform matrix to get its "forward" direction.  Something like : vec3( mat[0][2], mat[1][2], mat[2][2] )

Good answer, I updated my implementation too, since we know camera direction (in matrix) already, there is no need to keep default direction manually. 

Glad it works now!

9 hours ago, MichaelBarth said:

However, I still feel like my method of clamping the yaw and pitch values is a bit silly.

No, this isn't silly at all.  I do it all the time.  But there are a couple other options that might work better for different situations (although I think your current method is the best for most situations).

First, you could just stop clamping it.  A rotation of 20,000 degrees is still valid and should still work.  The only downside is that it's less readable (i have no idea what direction 20,000 is pointing).  Eventually, in a really long session, numbers might also accumulate to the point where you start losing floating-point precision.

The other option is to stop accumulating the rotation and only incrementally rotate the matrix.  Right now your function resets the matrix every frame and then re-rotates it back to the desired rotation.  Instead, you could stop resetting it and only rotate it by the delta yaw and pitch.  Then you don't need to track the rotation separately at all.  The only problem is that it now becomes more difficult if you need to reference the player's heading for some other purpose (like an overhead map view for example)

Alright cool. What I'm getting from this is that the view matrix is essentially the same as a model matrix, just here we're handling it through some kind of input.

So I guess the next thing is gimbal lock. Is it really such a scary thing that people make it out to be? Because that's probably the only reason I wanted to use Quaternions. Is this something I really need to worry about?

No, Gimbal lock isn't an issue here.  Gimbal lock is really only an issue when you are interpolating between keyframes.  But you are explicitly setting a rotation every frame, so Gimbal lock never happens.