Hi, I'm attempting to implement a first person, mouse-controlled camera for variable orientation situations. This means basically I need a regular mouse look camera to behave normally with any "up" vector. This will be used for moving around the entire surface of a spherical planet. So as you walk along, the view direction stays the same relative to the ground, so the view doesn't stay fixed on one point as you change gravity vectors. Pretty basic right?

Well I've got the necessary code down, and it works perfectly in most situations. However, there is a big problem with it. Near the south pole (basically near gravity=(0, -1, 0)), the direction vector seems to be transformed toward the south pole as the gravity/orientation changes. So if you're standing still, a few units away from the south pole, the direction is fine. Try and move in any direction, and the view direction is shifted toward the south pole. It's extremely odd.

It seems to me that the transformation matrix for transforming the world-axis-based mouse movements is somehow affecting the direction vector to shift it to point toward the south pole. Basically, I think the transformation I'm using (a rotation from the original up vector(0, 1, 0) and the current orientation) is forcing the direction to be "in line" with that, hence pointing it toward the south pole. But the transformation seems to work fine for most other orientations.

I would upload a video, but the recording really doesn't capture the problem, because it looks like the mouse is just being moved toward the south pole. Also, only the horizontal component gets messed up, the vertical component stays level.

Anyway, here's the code. I would really appreciate some guidance from someone who's got a properly working system.

Here's the implementation with just using sines and cosines to calculate direction vector from mouse angles.

        glm::mat4 trans;
	float factor = 1.0f;
	m_horizontal += horizontal;
	m_vertical += vertical;

	while (m_horizontal > TWO_PI) {
		m_horizontal -= TWO_PI;
	}

	while (m_horizontal < -TWO_PI) {
		m_horizontal += TWO_PI;
	}

	if (m_vertical > MAX_VERTICAL) {
		vertical = MAX_VERTICAL - (m_vertical - vertical);
		m_vertical = MAX_VERTICAL;
	}
	else if (m_vertical < -MAX_VERTICAL) {
		vertical = -MAX_VERTICAL - (m_vertical - vertical);
		m_vertical = -MAX_VERTICAL;

        glm::vec3 tmp = m_orientation;
	tmp.y = fabs(tmp.y);

/* this check is to prevent glm abort on cross product of parallel vectors */
/* factor=-1.0f only extremely close to south pole. Problem occurs much outside of that region */
	if (glm_sq_distance(tmp, glm::vec3(0.0f, 1.0f, 0.0f)) > 0.001f) {
		glm::vec3 rot = glm::normalize(glm::cross(glm::vec3(0.0f, 1.0f, 0.0f), m_orientation));
		float angle = (acosf(m_orientation.y) * 180.0f) * PI_RECIPROCAL;
		glm::quat t = glm::angleAxis(angle, rot);
		trans = glm::mat4_cast(t);
	}
	else if (m_orientation.y < 0.0f) {
		factor = -1.0f;
	}

	tmp = glm::vec3(cos(m_vertical) * sin(m_horizontal),
			sin(m_vertical),
			cos(m_vertical) * cos(m_horizontal)) * factor;

	m_up = m_orientation;

	m_direction = glm::vec3(trans * glm::vec4(tmp.x, tmp.y, tmp.z, 0.0f));

	m_view = glm::lookAt(m_position, m_position + m_direction, m_up);
	m_vp = m_perspective * m_view;

I also have a quaternion implementation, but it's a little more prone to glm aborts (anyone have an elegant solution for those, btw?). Both the quaternion and regular angle one behave identically.

        glm::mat4 trans;
	float factor = 1.0f;
	m_horizontal += horizontal;
	m_vertical += vertical;

	while (m_horizontal > TWO_PI) {
		m_horizontal -= TWO_PI;
	}

	while (m_horizontal < -TWO_PI) {
		m_horizontal += TWO_PI;
	}

	if (m_vertical > MAX_VERTICAL) {
		vertical = MAX_VERTICAL - (m_vertical - vertical);
		m_vertical = MAX_VERTICAL;
	}
	else if (m_vertical < -MAX_VERTICAL) {
		vertical = -MAX_VERTICAL - (m_vertical - vertical);
		m_vertical = -MAX_VERTICAL;
	}

        glm::quat t, quat;

	glm::vec3 tmp = m_orientation;
	tmp.y = fabs(tmp.y);

	if (glm_sq_distance(tmp, glm::vec3(0.0f, 1.0f, 0.0f)) > 0.001f) {
		glm::vec3 axis = glm::normalize(glm::cross(glm::vec3(0.0f, 1.0f, 0.0f), m_orientation));
		float angle = (acosf(m_orientation.y) * 180.0f) * PI_RECIPROCAL;
		t = glm::angleAxis(angle, axis);
	}
	else if (m_orientation.y < 0.0f) {
		factor = -1.0f;
	}

	glm::quat rot = glm::angleAxis(m_horizontal * ONEEIGHTY_PI, glm::vec3(0.0f, 1.0f, 0.0f));
	quat = rot * quat;

	rot = glm::angleAxis(m_vertical * -ONEEIGHTY_PI, glm::vec3(1.0f, 0.0f, 0.0f));
	quat = quat * rot;

	t = t * quat;

	trans = glm::mat4_cast(t);
	m_direction = glm::vec3(trans[2]);

Thanks in advance for the help.

Thanks for the suggestions. I did attempt a matrix-only implementation at one point, but was unsuccessful. It's entirely possible that I had some mistakes in there, and your solution is pretty clean, so I will give that a try. I'll report back after work.

One question I have is: how OK is it to use the previous frame's right vector? I'm sure it works, but I would prefer an up-to-date right vector, is there a good way to calculate that? I ended up completely avoiding a right vector in my code. And in your pitch and yaw angle method, you're using a right vector of vec3(1,0,0), which surely isn't correct for different orientations.

Well I tried this out. And it did fix the original problem I had, so it was clearly the weird transformation I was using. However, there are two other problems. The first is that the look direction is not automatically adjusted as the orientation changes. So the direction stays fixed in place as you move over the planets surface, which is definitely awkward. The direction needs to stay the same relative to the ground as the orientation changes. The second problem is that attempting to cap the vertical angle at which you can look seems to cap to the same section despite the orientation. Well, it's weird, if you move along and don't move the mouse much, then the look direction is capped between -90 and 90 of the original orientation (0, 1, 0), which is obviously a problem. However, putting the vertical angle at max and moving horizontally a lot seems to fix the issue, and then the cap is reset to the current orientation. Very peculiar. I can't quite imagine why that would be happening...

Here's the code I used, maybe someone can spot the issues causing the two problems I mentioned.

void Camera::set_angles_advanced(float horizontal, float vertical) {
	glm::mat4 trans;
	float factor = 1.0f;
	m_horizontal += horizontal;
	m_vertical += vertical;

	while (m_horizontal > TWO_PI) {
		m_horizontal -= TWO_PI;
	}

	while (m_horizontal < -TWO_PI) {
		m_horizontal += TWO_PI;
	}

	if (m_vertical > MAX_VERTICAL) {
		vertical -= m_vertical - MAX_VERTICAL;

		if (vertical < 0) {
			vertical = 0;
		}

		m_vertical = MAX_VERTICAL;
	}
	else if (m_vertical < -MAX_VERTICAL) {
		vertical -= m_vertical - MAX_VERTICAL;

		if (vertical > 0) {
			vertical = 0;
		}

		m_vertical = -MAX_VERTICAL;
	}

	glm::mat4 pitch = glm::rotate(glm::mat4(), vertical * ONEEIGHTY_PI, glm::normalize(m_right));
	glm::mat4 yaw = glm::rotate(glm::mat4(), horizontal * ONEEIGHTY_PI, m_orientation);

	m_camera = pitch * yaw * m_camera;

	m_direction = glm::vec3(m_camera[2]);

	m_view = glm::lookAt(m_position, m_position + m_direction, m_orientation);
	m_vp = m_perspective * m_view;


	m_dir_norm = glm::normalize(m_direction);
	m_dir_horizontal_norm = glm::normalize(m_direction - glm_project(m_direction, m_orientation));

	m_right = glm::cross(m_direction, m_orientation);
	m_right_horizontal_norm = glm::normalize(m_right);
}

Thanks in advance everyone!

I'm going to bump this with my most recent attemps. I still haven't solved this issue.

I'll post all the code I've tried. There are comments above each block that indicate the issue with that particular implementation. Surely someone has implemented this type of camera at some point? I would be happy to basically copy someone else's implementation for this rather than attempt to fix my own work, if necessary.

void Camera::set_angles_advanced(float horizontal, float vertical) {

glm::mat4 trans;
float factor = 1.0f;
float real_vertical = vertical;
m_horizontal += horizontal;
m_vertical += vertical;

while (m_horizontal > TWO_PI) {
    m_horizontal -= TWO_PI;
}

while (m_horizontal < -TWO_PI) {
    m_horizontal += TWO_PI;
}

if (m_vertical > MAX_VERTICAL) {
    vertical -= m_vertical - MAX_VERTICAL;

    if (vertical < 0) {
        vertical = 0;
    }

    m_vertical = MAX_VERTICAL;
}
else if (m_vertical < -MAX_VERTICAL) {
    vertical -= m_vertical - MAX_VERTICAL;

    if (vertical > 0) {
        vertical = 0;
    }

    m_vertical = -MAX_VERTICAL;
}

// -------------------- south pole rotation
/*glm::quat rotation;

if (m_orientation != glm::vec3(0.0f, 1.0f, 0.0f)) {
    glm::vec3 axis = glm::normalize(glm::cross(glm::vec3(0.0f, 1.0f, 0.0f), m_orientation));
    rotation = glm::rotate(rotation, acosf(m_orientation.y) * ONEEIGHTY_PI, axis);
}

rotation = glm::rotate(rotation, m_horizontal * ONEEIGHTY_PI, glm::vec3(0.0f, 1.0f, 0.0f));
rotation = glm::rotate(rotation, m_vertical * ONEEIGHTY_PI, glm::vec3(1.0f, 0.0f, 0.0f));

m_direction = glm::vec3(rotation * glm::vec4(0.0f, 0.0f, -1.0f, 0.0f));*/



// --------------------- south pole rotation
/*glm::vec3 tmp = m_orientation;
float look_factor = 1.0f;
float addition = 0.0f;

if (tmp.y < 0.0f) {
    tmp.y *= -1.0f;
    look_factor = -1.0f;
    addition = 180.0f;
}

glm::mat4 yaw = glm::rotate(glm::mat4(), m_horizontal * ONEEIGHTY_PI, m_orientation);
glm::mat4 pitch = glm::rotate(glm::mat4(), m_vertical * -ONEEIGHTY_PI, glm::vec3(1.0f, 0.0f, 0.0f));

if (tmp != glm::vec3(0.0f, 1.0f, 0.0f)) {
    glm::vec3 axis = glm::normalize(glm::cross(glm::vec3(0.0f, 1.0f, 0.0f), tmp));
    pitch = glm::rotate(glm::mat4(), acosf(tmp.y) * ONEEIGHTY_PI * look_factor + addition, axis) * pitch;
}

glm::mat4 cam = yaw * pitch;

m_direction = glm::vec3(cam[2]);*/

// -------------------- oscillation when looking close to vertical, vertical range capped
/*glm::mat4 yaw_matrix = glm::rotate(glm::mat4(), m_horizontal * ONEEIGHTY_PI, m_orientation);

m_right = glm::cross(m_direction, m_orientation);

glm::mat4 pitch_matrix = glm::rotate(glm::mat4(), m_vertical * -ONEEIGHTY_PI, glm::normalize(m_right));

glm::mat4 camera_matrix = pitch_matrix * yaw_matrix;
m_direction = glm::vec3(camera_matrix[2]);*/


// --------------------- oscillation when looking close to vertical, vertical range always capped to -90,90
/*glm::mat4 yaw = glm::rotate(glm::mat4(), m_horizontal * ONEEIGHTY_PI, m_orientation);
glm::mat4 pitch = glm::rotate(glm::mat4(), m_vertical * -ONEEIGHTY_PI, m_right);

glm::mat4 cam = pitch * yaw;

m_right = glm::vec3(cam[0]);
m_up = glm::vec3(cam[1]);
m_direction = glm::vec3(cam[2]);*/



// ----------------------- south pole rotation
/*glm::dvec3 dir = glm::dvec3(cos(m_vertical) * sin(m_horizontal),
    sin(m_vertical),
    cos(m_vertical) * cos(m_horizontal));

glm::vec3 tmp = m_orientation;
tmp.y = fabs(tmp.y);

glm::dmat4 dtrans;
float angle;

if (glm_sq_distance(tmp, glm::vec3(0.0f, 1.0f, 0.0f)) > 0.001f) {
    glm::vec3 axis = glm::normalize(glm::cross(glm::vec3(0.0f, 1.0, 0.0f), m_orientation));
    angle = acos(m_orientation.y) * ONEEIGHTY_PI;
    dtrans = glm::rotate(glm::mat4(), angle, axis);
}
else if (m_orientation.y < 0.0f) {
    factor = -1.0f;
}

dir = glm::dvec3(dtrans * glm::dvec4(dir.x, dir.y, dir.z, 0.0f));
m_direction = glm::vec3(dir);*/


m_dir_horizontal_norm = glm::normalize(m_direction - glm_project(m_direction, m_orientation));

m_view = glm::lookAt(m_position, m_position + m_direction, m_orientation);
m_vp = m_perspective * m_view;
}

Solved it. See http://gamedev.stackexchange.com/questions/73588/how-do-i-fix-my-planet-facing-camera for a pretty good explanation. I'll post the code here.

glm::mat4 trans;
float factor = 1.0f;
float real_vertical = vertical;
m_horizontal += horizontal;
m_vertical += vertical;

while (m_horizontal > TWO_PI) {
    m_horizontal -= TWO_PI;
}

while (m_horizontal < -TWO_PI) {
    m_horizontal += TWO_PI;
}

if (m_vertical > MAX_VERTICAL) {
    m_vertical = MAX_VERTICAL;
}
else if (m_vertical < -MAX_VERTICAL) {
    m_vertical = -MAX_VERTICAL;
}

glm::quat world_axes_rotation = glm::angleAxis(m_horizontal * ONEEIGHTY_PI, glm::vec3(0.0f, 1.0f, 0.0f));
world_axes_rotation = glm::normalize(world_axes_rotation);
world_axes_rotation = glm::rotate(world_axes_rotation, m_vertical * ONEEIGHTY_PI, glm::vec3(1.0f, 0.0f, 0.0f));

m_pole = glm::normalize(m_pole - glm::dot(m_orientation, m_pole) * m_orientation);

glm::mat4 local_transform;

local_transform[0] = glm::vec4(m_pole.x, m_pole.y, m_pole.z, 0.0f);
local_transform[1] = glm::vec4(m_orientation.x, m_orientation.y, m_orientation.z, 0.0f);
glm::vec3 tmp = glm::cross(m_pole, m_orientation);
local_transform[2] = glm::vec4(tmp.x, tmp.y, tmp.z, 0.0f);
local_transform[3] = glm::vec4(m_position.x, m_position.y, m_position.z, 1.0f);

world_axes_rotation = glm::normalize(world_axes_rotation);
m_view = local_transform * glm::mat4_cast(world_axes_rotation);
m_direction = -1.0f * glm::vec3(m_view[2]);
m_up = glm::vec3(m_view[1]);
m_right = glm::vec3(m_view[0]);

m_view = glm::inverse(m_view);