static float xrot{},yrot{};
int x{},y{};

SDL_GetRelativeMouseState(&x,&y);
xrot+=x/100.0f;
yrot+=y/100.0f;

if(xrot>360.0f)
	xrot-=360.0f;

if(yrot>360.0f)
	yrot-=360.0f;


if(xrot<-360.0f)
	xrot+=360.0f;

if(yrot<-360.0f)
	yrot+=360.0f;

std::stack<mat4> modelview;
modelview.push(mat4(1.0f));

glm::vec3 front=vec3{1.0f,0.0f,0.0f};
modelview.top()*=lookAt(vec3{0.0f,0.0f,0.0f},front,vec3{0.0f,1.0f,0.0f});
modelview.top()*=rotate(xrot,vec3{0.0f,1.0f,0.0f});
modelview.top()*=rotate(yrot,vec3{1.0f,0.0f,0.0f}); //if I comment this line then x rotation works.

//Upload projection matrix
program->projectionMatrix(projection);

for(auto entity: entities) {
	modelview.push(modelview.top());
	modelview.top()*=translate(entity.position);

	//Upload modelview matrix
	program->modelviewMatrix(modelview.top());

	for(auto m: models)
		if(m->getName()==entity.modelName)
			m->render(program->getId());

	modelview.pop();
}

I have gone through several tutorials but none seem to show how to do the front direction calculation compatible with GLM. I have really tried to get the problem here... xrot and yrot are in 360 degrees space.

why does x rotation work when I comment the y rotation line? how do I calculate direction for use with lookat?

I know this might come off as a stupid thread for a first post from a user when I should probably read the tutorials again. But I just cannot seem to adapt it to my code.

Thanks for reading!

The way i'm doing lookat for my camera:

void Camera::update()
{
	double sinAlfa = glm::sin(m_alfa * DAG2RAD);
	double sinBeta = glm::sin(m_beta * DAG2RAD);
	double cosAlfa = glm::cos(m_alfa * DAG2RAD);
	double cosBeta = glm::cos(m_beta * DAG2RAD);

	m_viewMatrix = glm::lookAt( m_position, glm::vec3(m_position.x + sinAlfa*cosBeta, m_position.y - sinBeta, m_position.z - cosAlfa*cosBeta), glm::vec3(0.0f, 1.0f, 0.0f));
}

just counting sin and cos function for each of angles of your camera, and with little math you can count point your camera is looking at. DAG2RAD is just pi/180 to convert my angle to radiants. I dont know if it solves your problem, but i hope it will help.

The way i'm doing lookat for my camera:

void Camera::update()
{
	double sinAlfa = glm::sin(m_alfa * DAG2RAD);
	double sinBeta = glm::sin(m_beta * DAG2RAD);
	double cosAlfa = glm::cos(m_alfa * DAG2RAD);
	double cosBeta = glm::cos(m_beta * DAG2RAD);

	m_viewMatrix = glm::lookAt( m_position, glm::vec3(m_position.x + sinAlfa*cosBeta, m_position.y - sinBeta, m_position.z - cosAlfa*cosBeta), glm::vec3(0.0f, 1.0f, 0.0f));
}

just counting sin and cos function for each of angles of your camera, and with little math you can count point your camera is looking at. DAG2RAD is just pi/180 to convert my angle to radiants. I dont know if it solves your problem, but i hope it will help.

Thank you!

There was also a bug in my second code snippet on line 14. I have edited it for people wanting to use the code. it used to read modelview.top()*=translate(modelview.top(),entity.position);