There are a handful of 3D engines out there for it already; Matt3D is a vector-based wireframe engine put to good use in a roller-coaster builder/simulation game and a racing game (that even supports two-player over the link port). However, it's 8-bit and so worlds created with it tend to be very small or distorted thanks to low-resolution (I tried writing a Quake game with it years ago).
The best "wall" engines (displaying a 3D world rather than a 3D object) have been the raycasters. Gemini impresses me the most; it's a Wolfenstein-level engine with objects, sliding "half-block" doors and moving wall blocks, all neatly texture-mapped.
I'm going to try, therefore, to get a vector-based "wall" engine (there must be a proper term for it) up and running. Keeping the map to 2D simplifies the maths a lot; hopefully with a few tricks things should be fast enough! Also, I'll use 16-bit arithmetic throughout to enhance the quality and scale of the maps.
Like DOOM, I'm isolating the vertices from the wall definitions. This way, I can cut down on the number of rotations/transformations required. The maths required to rotate a point around the origin is fairly simple;
Xr = Xo x sin(a) + Yo x cos(a)
Yr = Xo x cos(a) - Yo x sin(a)
For the moment, I'll use a simple lookup-table for wall heights. Ultimately, I'd like to have variable height walls, but these will do for the moment:
The translated X is just 48+(Xr/Yr), as the screen is 96 pixels wide.
Throwing in a few extra lines for walls...
As you might be able to see, there is no clipping if any part of a wall falls outside the viewing range. Also, the walls are not occluding eachother, something you'd really want.
