Jump to content

  • Log In with Google      Sign In   
  • Create Account

#Actuale‍dd

Posted 27 November 2012 - 03:02 PM

In my compression module, I had reworked the whole system to accept a stream as input, and a stream as output, however I want to take advantage of the raw speed and simplicity of compressing from memory to memory, as well. I can make a stream from an array of bytes, and easily pass one array for input, and one for output, but it would be slow, while accessing it in the same way that I access a file.


Turn the input interface on its head. Rather than having an abstraction for buffer filling, have an abstraction for returning pointers to successive ranges of bytes. In other words, not this (which I'm guessing is somewhat similar to what you have now):

class byte_source : uncopyable
{
    public:
        virtual ~byte_source() { }
        virtual std::size_t read_bytes(unsigned char *outbuf, std::size_t outbuf_size) = 0;
};

But this, instead:

struct byte_range
{
    byte_range() : begin(0), end(0) { }
    byte_range(const unsigned char *begin, const unsigned char *end) : begin(begin), end(end) { }

    bool empty() const { return begin == end; }

    const unsigned char * const begin;
    const unsigned char * const end;
};

class byte_source : uncopyable
{
    public:
        virtual ~byte_source() { }
        virtual byte_range next_range() = 0;
};

This second byte_source interface can be implemented trivially to return the range for a memory buffer. It can also be implemented in terms of a memory-mapped file, even when the OS imposes a fixed-size window (I think Windows restricts you to 4Gb at a time).

You can also implement a source that does buffered file reads as well, returning the range corresponding to its internal buffer each time, after refilling it.

#1e‍dd

Posted 27 November 2012 - 03:01 PM

In my compression module, I had reworked the whole system to accept a stream as input, and a stream as output, however I want to take advantage of the raw speed and simplicity of compressing from memory to memory, as well. I can make a stream from an array of bytes, and easily pass one array for input, and one for output, but it would be slow, while accessing it in the same way that I access a file.


Turn the input interface on its head. Rather than having an abstraction for buffer filling, have an abstraction for returning pointers to successive ranges of bytes. In other words, not this (which is I'm guessing is somewhat like you have now):

class byte_source : uncopyable
{
    public:
        virtual ~byte_source() { }
        virtual std::size_t read_bytes(unsigned char *outbuf, std::size_t outbuf_size) = 0;
};

But this, instead:

struct byte_range
{
    byte_range() : begin(0), end(0) { }
    byte_range(const unsigned char *begin, const unsigned char *end) : begin(begin), end(end) { }

    bool empty() const { return begin == end; }

    const unsigned char * const begin;
    const unsigned char * const end;
};

class byte_source : uncopyable
{
    public:
        virtual ~byte_source() { }
        virtual byte_range next_range() = 0;
};

This second byte_source interface can be implemented trivially to return the range for a memory buffer. It can also be implemented in terms of a memory-mapped file, even when the OS imposes a fixed-size window (I think Windows restricts you to 4Gb at a time).

You can also implement a source that does buffered file reads as well, returning the range corresponding to its internal buffer each time, after refilling it.

PARTNERS