I have a function that will orbit a ship around a planet-- that part works fine. I'm having trouble keeping the ship looking at the same point relative to the planet. If I turn the ship to look at the horizon, it should always be looking at the horizon. If the "Up" of the ship and the orbit are the same, then it works fine, but as they become more "off" the ship starts to face in some other direction. And I have noticed something else: sometimes the lookAt will lock to a direction (I think this happens when the "Ups" are perpendicular to each other)..... Here is the code I am using:

		D3DXQUATERNION rot;
		D3DXMATRIX tmp;
		//set up the rotation quaternion for orbit speed
		D3DXQuaternionRotationYawPitchRoll(&rot, OrbitData.OrbitRate*float(GE->MyCamera.fTBF)/100.0f, 0, 0);//1000.0f
		D3DXQuaternionMultiply(&OrbitData.OrbitPosition, &OrbitData.OrbitPosition, &rot);
		D3DXMatrixRotationQuaternion(&tmp,&OrbitData.OrbitPosition);
		tmp(3,0)=tmp(3,1)=tmp(3,2)=0;
		D3DXMatrixInverse(&tmp,0,&tmp);
		D3DXVECTOR3 pos(0,0,1);
		D3DXVec3TransformCoord(&pos,&pos,&tmp);
		Ship->V_Location=pos;
		Ship->V_Location*=OrbitData.Radius;
		Ship->V_Location+=OrbitData.Center;


		//rotate the view so it's locked relative to the planet as the ship orbits
/*
//*********************************** this doesn't work........
		//get the Up of the m_orientation quat
	D3DXMatrixRotationQuaternion(&tmp,&GE->MyCamera.m_orientation);
	D3DXVECTOR3 Up(0,1,0);
	D3DXVec3TransformCoord(&Up,&Up,&tmp);
	Up-=OrbitData.Up;
	if (D3DXVec3Length(&Up)>=1.41421356237f) D3DXQuaternionInverse(&rot,&rot);
*/
		D3DXQuaternionMultiply(&GE->MyCamera.m_orientation, &GE->MyCamera.m_orientation, &rot);

The rem'd out section is an attempt to correct this. It only fixes the direction issue (if the ship is orbiting the planet and the planet is on its left, it works fine. If the planet is on the right, the ship curves away from the planet instead of staying locked)...... It doesn't fix it completely though.....

rotating it should be trivial.

Explain with some code.

I'm not sure what you are trying to accomplish. Do you want to make a satellite like object with a fixed position relative to the planet? If this is the case you may simply define the ship position relative to the planet and then apply the planet transformation to it. But I think you should try to explains your problem better.

I have a ship in orbit around a planet. The orbit can be at any angle. With no other adjustments, the ship's "view" is a fixed direction as it orbits the planet. i.e. if my ship's LookAt is (0,0,1) when the orbit starts, it remains at (0,0,1) throughout the orbit. I have input so that the player can turn the ship in any direction while orbiting. This means that if the player turns the ship to (0,0.707,0.707), then it will continue to look in that direction throughout the orbit. What I want is: If I turn the ship to face the horizon of the planet, then the ship will always face the horizon at that angle. This should be true for ANY angle the player chooses. I have tried SO MANY different methods to make this work with no luck. I have done so many that I don't even want to discuss them! I am going to try (when I get off work): When orbit starts, create a fixed point around the planet using the ship's position and the look vector. Make that point orbit with the ship and point the ship at it. Capture the input for ship rotation. Rotate the point around the ship. I hope this works.....

I figured out a partial solution. It's not perfect, but when facing near the orbit path, it works well. anything too far off the "orbit plane" will cause a drifting effect. This is not a big issue, so for now it will do. Here is the code I am using:

void AUTOPILOT::OrbitPlanet(GAMEENGINE *GE,SHIP *Ship){
	if (OrbitData.OrbitEstablished){
		HUGEVECTOR3 hv=Ship->V_Location;//store this value for velocity calc later
		D3DXQUATERNION rot;
		D3DXMATRIX tmp;

		//set up the rotation quaternion for orbit speed
		D3DXQuaternionRotationYawPitchRoll(&rot, OrbitData.OrbitRate*float(GE->MyCamera.fTBF)/10.0f, 0, 0);//1000.0f
		D3DXQuaternionMultiply(&OrbitData.OrbitPosition, &OrbitData.OrbitPosition, &rot);
		D3DXMatrixRotationQuaternion(&tmp,&OrbitData.OrbitPosition);
		tmp(3,0)=tmp(3,1)=tmp(3,2)=0;
		D3DXMatrixInverse(&tmp,0,&tmp);
		D3DXVECTOR3 pos(0,0,1);
		D3DXVec3TransformCoord(&pos,&pos,&tmp);
		Ship->V_Location=pos;
		Ship->V_Location*=OrbitData.Radius;
		Ship->V_Location+=OrbitData.Center;

		//capture the yaw/pitch input
		if (MyInput.mousestate.rgbButtons[1]>0){
			D3DXMATRIX la;
			HUGEVECTOR3 hv=Ship->V_Location+GE->MyCamera.m_viewDir*1000000.0f;
			hv-=OrbitData.Center;
			OrbitData.CameraLookAt=hv.Normalize();
			OrbitData.CameraLookAt*=float(hv.Length);
			OrbitData.LookAtLength=float(hv.Length);
			D3DXMatrixLookAtLH(&la,&D3DXVECTOR3(0,0,0),&OrbitData.CameraLookAt,&OrbitData.Up);
			D3DXQuaternionRotationMatrix(&OrbitData.LookAtQuat,&la);
			D3DXQuaternionNormalize(&OrbitData.LookAtQuat,&OrbitData.LookAtQuat);
		}


		D3DXQuaternionMultiply(&OrbitData.LookAtQuat, &OrbitData.LookAtQuat, &rot);
		D3DXMatrixRotationQuaternion(&tmp,&OrbitData.LookAtQuat);
		tmp(3,0)=tmp(3,1)=tmp(3,2)=0;
		D3DXMatrixInverse(&tmp,0,&tmp);
		pos=D3DXVECTOR3(0,0,1);
		D3DXVec3TransformCoord(&pos,&pos,&tmp);
		HUGEVECTOR3 hv2=pos*OrbitData.LookAtLength;
		hv2+=OrbitData.Center;
		hv2-=Ship->V_Location;
		GE->MyCamera.m_viewDir=hv2.Normalize();
		D3DXMatrixLookAtLH(&tmp,&D3DXVECTOR3(0,0,0),&GE->MyCamera.m_viewDir,&OrbitData.CameraUp);
		D3DXQuaternionRotationMatrix(&GE->MyCamera.m_orientation,&tmp);

		//capture the roll input
		if (abs(GE->MyCamera.RateRoll)>0.001f){
			D3DXMATRIX la;
			OrbitData.CameraUp=GE->MyCamera.m_yAxis;
			HUGEVECTOR3 hv=Ship->V_Location+GE->MyCamera.m_viewDir*1000000.0f;
			hv-=OrbitData.Center;
			OrbitData.CameraLookAt=hv.Normalize();
			OrbitData.CameraLookAt*=float(hv.Length);
			OrbitData.LookAtLength=float(hv.Length);
			D3DXMatrixLookAtLH(&la,&D3DXVECTOR3(0,0,0),&OrbitData.CameraLookAt,&OrbitData.Up);
			D3DXQuaternionRotationMatrix(&OrbitData.LookAtQuat,&la);
			D3DXQuaternionNormalize(&OrbitData.LookAtQuat,&OrbitData.LookAtQuat);

		}
		
		//calculate the velocity
		hv-=Ship->V_Location;
		D3DXVECTOR3 veldir=-hv.Normalize();
		Ship->V_velocity=veldir;
		Ship->V_velocity*=hv.Length *1000.0/GE->MyCamera.fTBF;
	}
	else{//not in orbit yet
		UINT wpSize=WP.size();
		if (wpSize>0){

			//continue to WPs until the next to last one is reached. then delete the last 2 WPs and make wpSize=0;
		}

		if (wpSize==0){//at the end of the wp list. create orbit trajectory
			OrbitData.OrbitEstablished=true;
			//D3DXVec3Cross(&tmpv,&(Range0[i].vert[px-1][py-1].v-Range0[i].vert[px][py].v),&(Range0[i].vert[px][py-1].v-Range0[i].vert[px][py].v));
			D3DXVECTOR3 ShipLoc1,ShipLoc2,Heading;
			ShipLoc1=(Ship->V_Location-OrbitData.Center).Normalize();
			OrbitData.Radius=float((Ship->V_Location-OrbitData.Center).GetLength());
			ShipLoc1*=OrbitData.Radius;//these values are only to get the "Up" vector
			Heading=Ship->V_velocity.Normalize();
			OrbitData.OrbitSpeed=Ship->V_velocity.Length;
			ShipLoc2=ShipLoc1+Heading*float(OrbitData.OrbitSpeed);

			D3DXVec3Cross(&OrbitData.Up,&ShipLoc2,&ShipLoc1);//********** i think the direction is correct....?????

			D3DXMATRIX la;
			D3DXVec3Normalize(&ShipLoc1,&ShipLoc1);
			D3DXVec3Normalize(&OrbitData.Up,&OrbitData.Up);
			D3DXMatrixLookAtLH(&la,&D3DXVECTOR3(0,0,0),&ShipLoc1,&OrbitData.Up);
			D3DXQuaternionRotationMatrix(&OrbitData.OrbitPosition,&la);
			D3DXQuaternionNormalize(&OrbitData.OrbitPosition,&OrbitData.OrbitPosition);

			OrbitData.OrbitRate=atan2(float(OrbitData.OrbitSpeed),OrbitData.Radius);

			OrbitData.CameraUp=GE->MyCamera.m_yAxis;
			HUGEVECTOR3 hv=Ship->V_Location+GE->MyCamera.m_viewDir*1000000.0f;
			hv-=OrbitData.Center;
			OrbitData.CameraLookAt=hv.Normalize();
			OrbitData.CameraLookAt*=float(hv.Length);
			OrbitData.LookAtLength=float(hv.Length);
			D3DXMatrixLookAtLH(&la,&D3DXVECTOR3(0,0,0),&OrbitData.CameraLookAt,&OrbitData.Up);
			D3DXQuaternionRotationMatrix(&OrbitData.LookAtQuat,&la);
			D3DXQuaternionNormalize(&OrbitData.LookAtQuat,&OrbitData.LookAtQuat);
		}
	}
}

This is the entire function. It includes initializing the orbit and maintaining it. Even though this function says AUTOPILOT, I haven't included the auto functionality using waypoints-- that comes later.

If ANYONE has a better way, I would like to know what it is..... I've looked over and over again for example code, but I get nothing anywhere near what I am trying to do.

Honestly, your code is barely readable. You've got too much DirectX matrix stuff, and reliance on reading the values you calculated the previous frame, and not enough clean definition of what your own game state is.

Just looking at the stuff for looking in a particular direction - you want to keep the ship's orientation and position separate, right? Orbit without rotating the ship?

You shouldn't have to throw some vector 1000000 whatever out into space in order to try to keep the ship looking one way. It should come naturally from your math. Store how the ship is oriented. I'd just track it's forwards and up vectors.

Each frame, construct the quats, matrices, whatever you need to place the ship out of as little data as possible. What's an orbit's state? You need a radius, a center, a speed, an axis, and a phase. Radius and speed are read-only scalars (until the ship leaves orbit). The center and axis vectors are also constant during orbit. You only need to update the scalar phase every frame, to have the basis for placing the ship.

Use those 5 variables to build your quat/matrix/whatever you need to find the ship's position: one vector. Construct a translation matrix from that. Use your ship's orientation to build the rotation matrix, and multiply those two together in the correct order. If your camera is relative to the ship, use the transformed location coord, transformed (location + forwards) coord, and transformed up vector to build the look matrix.

If you want a game that's simple to maintain, you should not be getting your current matrix by multiplying the previous frame's matrix by something - you should rebuild it from scratch every frame. You may or may not need to store matrices eventually - in a space game, I can imagine the need for several relative coordinate systems - but each of those matrices can be built per frame from basic parameters like positions and axes.

Write a function that take in a few simple parameters and outputs a single Vector. That should be your orbit function. It shouldn't be aware of ships or cameras or whatnot.