Here's my idea for how to handle RFB textures:
Let's say we have a texture for terrain that is 8kx8k, split into 512x512 textures with mipmaps, plus a 16x16 mipmapped texture for the mips that the 512 ones don't get (reasoning behind this is that the 512 mips will go 512-256-128-64-32-16-8-4-2-1, and at the level where it's at 1, you end up with 256 1x1 textures that still need a couple more mipmaps. The mipmapped 16x16 tex takes care of that). This is all stored in the system pool. Then, in the default pool, we have, say, 24 mipmapped 512x512 textures (i.e. 32MB in size), reserved for general terrain use.
First, the blocks of terrain nearest the camera are rendered first, using the requisite full-res 512x512 blocks that are stored in system mem. How they are used is a series of UpdateSurfaces are called, the source being the system mem textures, the destination being the first 4 default pool textures (I'll refer to the default pool textures as A, B, C, D, etc.). Basically, every mipmap of A-D are filled using the system mem blocks. Then, four larger chunks (x2) of terrain that are further out are rendered as well. They will use textures E-H. This time, the UpdateSurfaces that E-H use are not the highest level mipmaps of the areas they're over. I'm having trouble writing up how this part works, so I'll make a little ASCII diagram:
Here's part of the terrain laid out:
1 2 3 4
A * * * *
B * * * *
C * * * *
D * * * *
The camera is located between the cells (B|C)(2|3), which is also where the first 4 512x512 blocks were used (i.e. every cell covers 512x512 of the main texture). Anyways, for the next chunks that each cover a 1kx1k area, their bounds are A1 to B2, A3 to B4, C1 to D2 and C3 to D4. Since each chunk can only have 512x512 of the highest level however, the mipmap relationships between the source chunk in system mem and the destination chunk, E (or F or..) can't be 1 to 1. Instead, the 2nd level of a source chunk (256x256) is used to fill up 1/4 of the 1st level of the destination chunk, and when the bottom level of the destination chunk needs to be written to, part of the sub-mipmap texture (the 16x16 mentioned way back) is copied instead. This is repeated until all 24 destination chunks are full. Then, an algorithm that JC describes here is used, to prevent excessive texture thrashing in successive frames:
While ( memory allocation for new texture fails )
Find the least recently used texture.
If the LRU texture was not needed in the previous frame,
Free the most recently used texture that isn't bound to an
active texture unit
Basically, when chunks S-X are used, instead of cycling back and rewriting A-D again, S-X get rewritten to again and again until the terrain is done rendering. Then, in the next frame (assuming the camera hasn't moved) A-R don't have to be written to again. If the camera does move though, chunks will obviously have to replaced, bit by bit. It's possible that either a scrolling system not unlike Dangerous Dan/Commander Keen's could be used (if such a function exists in D3D9..), or in the worse case, chunks get rewritten in a slightly staggered fashion.
Obviously, this doesn't take into account frustum culling, which would complicate the management, but also let it be more aggressive. Plus, one nice thing is that streaming in the RFB texture from the HD could be done relatively easily as the camera moves around I imagine.
Anyways, this is something I shall want to experiment with in the near future. Any thoughts?