In a nutshell, I have a ray casting engine that uses rotation and some trigonometry, not matrices (vector math). I'd like to add something that draws a line from any two points, but when one of those points is beyond the viewing plane, it goes bat-attack crazy (skyrocketing in terrible directions).

Imagine shooting an arrow, but if one end of the arrow is outside the screen, it stretches the arrow obscenely, obviously not accurately representing the point anymore. This is a fake 3d world after all, such as wolfenstein. The calculation that's taking place for the verticle "z" axis is mearly scaling an arbitrary value.

I haven't found any information that addresses this, but after looking around I'm suspicious that using matrices might be the solution. I'd like to avoid having to learn that or adapt the engine I've already made, especially if that isn't the solution, or the only solution.

Here's some of the code I'm using:

pLoc is the x,y of the camera.
mLoc is the x,y of the point.
pRot is the rotation of the camera.
pHgh is the height (z) of the camera.
mHgh is the height (z) of the point.

On the grid, x increases to the right, and y down. The grid scheme of the world is 128 pixels, and that's also the max height for the camera. The screen is 320 by 200 pixels. And this is being coded for shockwave, although what I'm trying to understand would be the math of it.

xxx = pLoc.x-mLoc.x
yyy = pLoc.y-mLoc.y
 
moo = atan(xxx,-yyy)*(180/pi)
if moo < 0 then moo = moo+360
 
tmp = pRot+120-moo
if moo < 180 and pRot > 90 then
    tmp = tmp-360
else if moo > 180 and pRot > 270 then
    tmp = tmp-360
end if
 
m = sqrt(power(pLoc.x-mLoc.x,2)+power(pLoc.y-mLoc.y,2))
m = m*cos(pRot-moo)*(pi/180)      -- eliminates "fisheye" from distance
 
boo = (tmp*-320/60)-160           -- this is x spot on screen (x,y in world)
poo = (64-pHgh+mHgh)*(300/(m))    -- this is the y spot (vertical z in world)
goo = 128/m*2                     -- this is a converted distance for sizing.

Again, if the point goes off-screen, its position goes crazy: it's not being represented correctly. If the answer really is that this would work with vector math, that's fine, but if there's any chance to even fake it in this approach, I'd very much like to discover it. I would cherish any ideas or information that lead to the capture of a strategy (hopefully something extremely simple).

Thanks. Hugs.

I haven't messed with ray casting in years, so I'm a little rusty, but I do remember that you cast rays within your field of view. If you have a 90° FOV, you would cast rays from -45° to 45°. There should be no reason for the rendering rays to ever "go behind the camera". Still, on that point, you are a little unclear. You mention "behind the camera" in the title, but mention "offscreen" in the post. These can be mutually exclusive. Just because a point is in front of the camera, doesn't necessarily mean it is on screen. Offscreen can mean behind the camera or out of the FOV. You shouldn't be casting rays behind the player, unless you are doing ray casting for collision, and you should be clipping anything offscreen.