With a little bit of clean-up, what's about the hopefully-now-considering-all-caveats-solution:

Matrix4x4 createView( Vector3* pos, Vector3* target, Vector3* up )
{
   Matrix4x4 cameraMat;
   Matrix4x4 view;

   Vector3 vX;
   Vector3 vY;
   Vector3 vZ;

   // computing vZ as the unit vector to look along
   vZ = sub(pos, target);
   normalise(&vZ);

   // computing vX being orthogonal on the plane of up and vZ; considering
   // its place in the ortho-normal basis later, this means up x vZ
   vX = crossProd(up, &vZ);
   normalise(&vX);

   // computing vY being orthogonal on the plane of vZ and vX; considering
   // its place in the ortho-normal basis later, this means vZ x vX
   vY = crossProd(&vZ, &vX);
   normalise(&vY);

   // building basis ...
   setColv(&cameraMat, 0, &vX, 0.0f);
   setColv(&cameraMat, 1, &vY, 0.0f);
   setColv(&cameraMat, 2, &vZ, 0.0f);
   // ... and injecting position
   setColv(&cameraMat, 3, pos, 1.0f);

   // inverting camera matrix to yield in the view matrix
   view = inverseMat4(&cameraMat);

   return view;
}

Well, err ... forgotten that OpenGL uses a camera that looks along the negative z axis. So although the (unnormalized) forward vector is

forward := target - pos

it doesn't point in the same direction as rowZ from the matrix, because rowZ is (per definition) along the positive z axis.

Also

rowY = crossProd(&rowX, &rowZ); // new rowY

should probably better be

rowY = crossProd(&rowZ, &rowX); // new rowY

to match the other cross-product.

With a little bit of clean-up, what's about the hopefully-now-considering-all-caveats-solution:

Matrix4x4 createView( Vector3* pos, Vector3* target, Vector3* up )
{
   Matrix4x4 cameraMat;
   Matrix4x4 view;

   Vector3 vX;
   Vector3 vY;
   Vector3 vZ;

   // computing vZ as the unit vector to look along
   vZ = sub(pos, target);
   normalise(&vZ);

   // computing vX being orthogonal on the plane of up and vZ; considering
   // its place in the ortho-normal basis later, this means up x vZ
   vX = crossProd(up, &vZ);
   normalise(&vX);

   // computing vY being orthogonal on the plane of vZ and vX; considering
   // its place in the ortho-normal basis later, this means vZ x vX
   vY = crossProd(&vZ, &vX);
   normalise(&vY);

   // building basis ...
   setColv(&cameraMat, 0, &vX, 0.0f);
   setColv(&cameraMat, 1, &vY, 0.0f);
   setColv(&cameraMat, 2, &vZ, 0.0f);
   // ... and injecting position
   setColv(&cameraMat, 3, pos, 1.0f);

   // inverting camera matrix to yield in the view matrix
   view = inverseMat4(&cameraMat);

   return view;
}

Please notice that looking along the negative axis also has an implication on the code when pressing W a.k.a. "go forward". In fact, you need to go in direction of the negative 3 column vector then.

Hey, thank you for your reply. Sorry it took me a couple days to get back to you.

I managed to get it working, but not exactly 100% how you've shown me.

Matrix4x4 createView(Camera* cam)
{
	Matrix4x4 cameraMat;
	Matrix4x4 view;

	Vector3 rowZ;
	Vector3 row4 = { cam->pos.x, cam->pos.y, cam->pos.z };
	Vector3 rowX;
	Vector3 rowY;
	Vector3 target;

	target = addVec(&cam->pos, &cam->dir);

	rowZ = sub(&target, &cam->pos);
	normalise(&rowZ);

	setEqual(&rowY, &cam->up);
	rowX = crossProd(&rowY, &rowZ); 
	normalise(&rowX);

	rowY = crossProd(&rowZ, &rowX);
	normalise(&rowY);

	setRowv(&cameraMat, 0, &rowX, 0.0f);
	setRowv(&cameraMat, 1, &rowY, 0.0f);
	setRowv(&cameraMat, 2, &rowZ, 0.0f);
	setRowv(&cameraMat, 3, &row4, 1.0f);	

	view = inverseMat4(&cameraMat);	// has to be inverse for camera transformations

	return view;
}

I've changed a couple things such as passing in the camera and accessing the data from that, just easier. I was still getting weird results using setColv function, so I fixed up the setRowv function and that is what has made it work for me (not entirely sure why because setColv should be working anyway).

void setRowv( Matrix4x4* m, int rowNum, Vector3* v, float w )
{
	int index1 = (0 * 4) + rowNum;
	int index2 = (1 * 4) + rowNum;
	int index3 = (2 * 4) + rowNum;
	int index4 = (3 * 4) + rowNum;
	m->m.m1[index1] = v->x;
	m->m.m1[index2] = v->y;
	m->m.m1[index3] = v->z;
	m->m.m1[index4] = w;
}

Also I found out that changing the cam->up vector was actually causing weird rotations. But as long as I calculate the rowX as the cross product of the up vector (0, 1, 0) it works. If the up vector is set to the new rowY it gets messed up and I seem to experience some roll when moving the camera around.

Thank you for your help and explanations, I've think this should be ok from here, feel free to chime in if you notice why its working like this