The way I understood it, for this rendering technique you need to first draw the scene at a smaller framebuffer(1/10th to 1/20th of the main resolution) where in the pixel shader you only render: mip level, U, V, texture ID, then read it back to CPU, render scene with the info from the previous frame's "small framebuffer" while on another CPU core analyzing this one and streaming + decompressing the required textures for the next frame.However what if you implement heavy hardware tessellation?This means you would be doing tessellation a minimum of 3 times each frame for each mesh(1 for the small framebuffer, 1 for the "main" pass and 1 for the RSM later).I thought about using stream out, but if I want to go crazy like 1 poly per pixel mesh density, the stream out would become a huge bottleneck?Is there some clever way to save an extra geometry pass?

If your level geometry is static and your camera movement is restricted, you can always pre-compute the pages that will be visible for a given camera location.

Actually you *just* need to determine what pages of textures you need, you don't even have to render anything before main scene, and just guess them by distance (this is advantage, when you need to render F.e. stuff behind player (reflective wall?!)) - of course you can use a lot more complicated heuristics to decide which pages you need to load (you can also shoot some rays, etc.). So basically you *don't* need geometry pass.

Also using some better heuristics can eliminate problems when you can quickly rotate around, or such - though I know this isn't much of a problem for fast HDD.

Of course there are disadvantages, you'll effectively hold data that you don't need for actual (respectively next) frame - so it's a trade-off, you save geometry pass, but you most likely will need more memory to store the data.