You're probably going to want to do fixed-point math in 16 or even 24 bits, depending on the accuracy you require.

In general this sort of work requires you to know where the significant bits of your calculation is currently at, and to exploit that to maintain accuracy at all stages. The fewer bits you're working with, the more attention you will have to pay to these issues.

I worked on a small ray-caster (like Wolfenstien 3D) using 16bit math, and lost interest before taking it very far, but the limitations of fixed point in 16bit was apparent even in relatively straight-forward math.

Benryves was workiing on a raycaster for the TI-83+ awhile back, using 16bit fixed-point, IIRC.

Here are a few links to his stuff:
Journal: 8-bit Raycasting Quake Skies and Animated Textures
Journal: A primitive 3D engine for the TI-83+
Web: http://benryves.com/journal

8-bit processor with 4KB of memory... that's going to be... interesting. First of all, is it 4KB for everything, or just for work RAM? (i.e. the code and model data is in separate ROM) Because in the case of the latter you could get a lot of speed up by using look-up tables. If everything has to fit in 4KB, tough luck getting many tables there.

Also: 8-bit integers may be enough to specify the shape of a given mesh. You won't be able to make very detailed meshes anyways if you pretend to have any sort of speed, not to mention memory usage =P If you can get away with 8-bit integers you're going to make your life a lot easier when it comes to performance.

Another thing is what kind of transformations you plan to do. Translation is easy. Scaling needs multiplications... Rotation is going to be madness the usual way. If you plan to do rotation you may want to use polar coordinates, because then to rotate you just need to add a value to the angle, and then you can convert it to X/Y/Z coordinates using a look-up table or something, reducing or even avoiding the multiplcations needed.

Clipping is going to be madness. I hope you have a plan for this one, especially for the near-Z plane (for the X/Y planes you could attempt to clip while rendering the triangles, for the far-Z plane just remove polygons after a certain distance, like pop-up for early 3D games used to work).

Also: cheat as much as you can. Seriously, if there's something that could be rendered with something that isn't standard polygons but rather a more restricted shape that could be done with a more specific algorithm, by all means try to do that. You can use this to reduce model complexity a lot and gain extra speed.

I'm pretty sure there's a lot more to be said, but that's what comes to my mind right now. It'd be useful to know how much memory do you actually have (if the code has to go in those 4KB or it's separate).

Ugh, dammit. Phones are terrible for accidentally clicking things you don't mean to. Sorry for the -1 Sik, maybe someone will come along and fix it.