So I just don't understand anything about how the rendering is done with your technique then. one ray per column ? how do you fill the column then, does not one ray = one pixel ? Do you have some kind of "ray to heightmap" intersection routine to render the terrain ? (like dichotomy or cone-based...?)
what is 2d, the scanning or the actual ray information ?
(I mean, a 3d ray bears 2 vectors of 3 scalars, a starting point and a direction, so your 2d rays stores what ? an origin pixel and a 2d direction ?)
about sheering, if a rotation were to be "simulated" by sheering my eyes would surely bleed. Can you just not do a space trick that would effectively make the rotation ?
Like when you create your rays, you trick the ray creation from the original data by rotating your origin space ? (if the pixels are your origin space when you scan the columns, then after extracting the coordinates (eg from loop indices), for your pixel, you just rotate them in place, and the following calculations will be space-tricked.)
edit: from this article https://en.wikipedia.org/wiki/Ray_casting, I start to vaguely see what was I missing in my mental conception of what you called ray casting. the folling of columns seems to be kind of retro-fed from the highest pixel, instead of casting one ray per pixel. But that seems very specific to games that will have only one or two objects to draw per column. And the terrain case actually seems very complex. If I believe my knowledge of how shadowing is done in parallax occlusion mapping, then its a quite involved algorithm and I fail to see how this could run real time. There must be something I am missing about heightmap rendering techniques. I feel this something must lay near the 2d-thingie concept that I can't quite grasp as well. Rotations would be obvious with 3d rays, and would not even deserve a question here, therefore my very simple first answer about the view matrix.