This is just to note how things are, I'm about to do a major refactor in how it internally stores layers and pixels. I've always intended to be able to handle LAB and CMYK etc modes as well as RGB, and also 16 and 32 bits as well as 8 bit. Decided that this would be quite troublesome to implement later down the line (as the gimp developers found) so I'm going to create some new surface classes to handle this before I get carried away making more plugins.
The other thing I've wanted to address is memory use, and speed. At the moment I was naively creating a big surface for each layer the size of the image. I believe this is how the gimp does it. However, a look at photoshop suggests they may use a scheme to cut down on memory use. It is not actually necessary to store all those pixels, because in a lot of layers they will be unused. So I've been looking at various schemes to only store pixel data for the used portions of the layer.
My first thoughts were to store a bounding rect for the used pixel data. However, if you draw on opposite corners of the image, the data stored is far more than needed. This leads to using multiple bounding rects for multiple areas in each layer. This is slightly wasteful, but not bad. However, things may get tricky when drawing onto the layer, and having to combine these areas when they overlap.
Next thoughts .. what about simply using a very simple RLE scheme to store the number of 'blank' pixels, then the number of 'valid' pixels. This would very nicely cut down memory usage, and have the side effect of very significantly speeding up rendering of the whole image with all the layers.
Unfortunately, a snag : A lot of the image updates are for a region, or sub-rect, of the image. In order to update just a region, I'd have to decompress the whole layer (or at least process across it). Not good for speed.
There are lots of possible gettarounds for this problem like gridding up the layer, but it seems ugly. So I'm now rethinking whether the multiple rectangles might be easiest. They would be easier to process as regions (i.e. just reject all rectangles outside the region, and process those inside).