[quote name='e?dd' timestamp='1354050082' post='5004670']
Turn the input interface on its head. Rather than having an abstraction for buffer filling,

I have a function that compresses input as it reads from a stream; if I already have the buffer, then I'm basically memory mapping. If I'm reading into a buffer through an abstract interface,
[/quote]
I'm saying that you don't have to (manually) read in to a buffer at all. You could argue memory mapping a file is doing this, but it should be doing it (near) optimally and it's handled for you, for the most part.

how is this different than using a buffered stream?[/quote]
There's no need for buffers! You can implement a buffered stream derived from the second interface if you want, but it's not necessary for (memory-mapped) files or raw memory, which seem to be your primary data sources.

If I use an abstract interface that could be a memory buffer, or a file, and I use functions to read from it, how is this different than the streams I already have?
[/quote]

Let me attempt to clarify by implementing the byte_source interface for a region of memory:


class memory_byte_source : public byte_source
{
public:
memory_byte_source(const unsigned char *begin, const unsigned char *end) : begin(begin) end(end) { }

byte_range next_range()
{
byte_range ret(begin, end);
begin = end;
return ret;
}

private:
const unsigned char *begin;
const unsigned char *end;
};


The first call to next_range() gives you the first and only byte_range (no copies, the range is the memory). The second call returns an empty range, telling you that the underlying data source is exhausted. The implementation for a memory mapped file will look very similar, except for the part where you create the mapping (in the constructor, perhaps).

The compression algorithm may of course have to be adapted for the new interface, but I don't imagine that would be too hard as you're now reading bytes through a pointer, regardless of the interface's implementation.

I already don't read into a buffer. I pass the stream to a compression function, and it reads from it directly, and writes to an output stream directly.

My point is, all of my algorithms have been rewritten to take stream sources and destinations, and I could make a stream from a buffer in memory. However, this would be much slower than accessing the memory directly in the special case of memory to memory compression/decompression.
[/quote]

It solves this problem because the interface still allows direct access from memory, where it can be provided. For file access, it also allows either manual reads in to a buffer, or memory-mapping, which ever works best or is faster. In either case, it's the thinnest and fastest interface providing access to a range of bytes.

If that's not what you were asking for, I'm afraid I don't understand what you're after.

Or phrased another way: how about you present your actual interface and point to what you specifically don't like about it? Otherwise I fear we'll continue to talk past each other.

Through this way, the data is read into an array either beforehand, or during the read somehow, creating a bottleneck.[/quote]
There's no bottleneck with my method (beyond the unavoidable cost of disk access).

For OS's that do support memory mapping a file without actually having it reside in memory, I'm still limited by address space. If I can't map even 2gb of contiguous memory on a 32bit machine, then there is no way that I can handle the maximum file size if I were to compress an entire file at once.

For OS's that do support memory mapping a file without actually having it reside in memory, I'm still limited by address space. If I can't map even 2gb of contiguous memory on a 32bit machine, then there is no way that I can handle the maximum file size if I were to compress an entire file at once.

How do I do memory mapping for an encrypted file stream in an archive through the OS?