To further elaborate:

max() returns whichever of the two parameters are larger.

min() returns whichever of the two parameters are smaller.

p.x and p.y are the location, in pixels, of the mouse on the screen.

The screen's width is 800, and the height is 600.

So:

p.x = max(min(p.x, 800), 0); //If 'p.x' is greater than the screen's width, it returns the screen's width (just to keep it in bounds).

p.x = max(min(p.x, 800), 0); //If 'p.x' is less than 0, it returns 0 (just to keep it in bounds).

And the same with 'p.y' and the screen's height.

float x = (p.x - 400) / 800.0f; //Subtracts half the screen's width from 'p.x', to put 'p.x' in a range from -400 to 400 with 0 being the center of the screen.

float x = (p.x - 400) / 800.0f; //Divides it by 800, to convert the range from "-400 to 400" to "-0.5 to 0.5".

And the same with 'p.y'...

D3DXVECTOR3 lookAt(x, 1.5f - y, -1.0f); //Creates a 3D math vector for Direct3D. Based off the name of the vector, it'll probably be used to reorient the camera to look in that direction.

(I would expect that GetCursorPos returns pixel indices in the range [0, nCols-1] and [0, nRows-1] for a nCols by nRows screen. If so, then the following 2 aspects are worth to be mentioned. Besides these details the posts above already stated what the snippet is good for.)

First, a given screen of 800 x 600 pixels has pixel indices ranging from 0 to 799 and from 0 to 599. Cursor positions are usually given by the pixel indices the hot spot is over, so the maximums should IMHO be limited to 799 and 599 instead of 800 and 600. (Also I'm not sure why the given code snippet needs to limit the cursor position at all.)

Second, a pixel is a tiny area, but nevertheless an area, while a D3DXVECTOR denotes an infinitesimal small, well, point in this case. To map from the one to the other, one has to decide what location on a pixel should be hit. If nothing "special" is done (as in the given solution), then the lower left corner will be hit. This causes the following asymmetry: The left lower pixel will result in (x,y) = (-0.5, -0.5), while the upper right pixel will result in (x,y) = (0.49875, 0.49833). If, on the other hand, the center should be hit (for symmetrical results), then half the extent of a pixel has to be considered.

Another consequence is that with the solution "as is" the (x,y) = (0,0) results for the pixel to the right and top of the real center of the screen. Actually the screen's center is not covered by a pixel, because it lies at the corners of 4 surrounding pixels, so you can't address it with the pointer device.