RLE is better if there are a lot of the same types of blocks in a certain area. It keeps track of them in the following manner:
say you have 5 A blocks, 3 B blocks, and 1 C blocks) as follows:AAAAABBBC
RLE would save that as:5A3B1C
Notice that the Z takes up a single byte ('Z') in the first line, while it takes up two bytes ('1Z') in the second. This really becomes a problem when you encounter a lot of different object types.
In my opinion, Octrees are much more space and memory efficient, but they will be harder to parse (from the programmer's standpoint). The reason is that RLE must be decompressed in memory where as Octrees don't.
whether or not a single item takes 1 or 2 bytes depends on the type of RLE.
in many forms of RLE, a single item will usually only take a single byte.
for example, a PCX-like RLE:
0x00-0xBF: encoded directly
0xC0-0xFF: RLE run (1-64 bytes), followed by byte.
most bytes, passed through (except the escape byte).
0xFE <count> <value> = RLE run
or, possibly, a compromise:
0x00-0xBF: passed through
0xC0-0xCF <value>: RLE run 1-16 (0xC0 <value> = Escaped Byte)
0xD0 <count> <value>: RLE Run (16-271)
0xD1 <count16> <value>: RLE Run (272-65536 / 65551)
0xE0-0xFF: passed through
in my engine, most of the voxel terrain is kept compressed in memory as well, but chunks may be decompressed on access, and revert to a compressed form later.
RLE is used here, mostly because it compresses/decompresses quickly.
block-level RLE can be faster than byte-level RLE, and avoids needing to flatten out or reconstitute the blocks (into a collection of "byte planes" or similar), but has the disadvantage that it doesn't compress as well (only blocks which are exactly the same may be compressed).
a more "fancy" idea I had also considered was basically doing something more PNG-like, basically using a Paeth predictor and Deflate, but this wouldn't be as usable for in-memory compression, due the higher encoding/decoding costs.
another trick is essentially skipping chunks which only contain a single type of voxel (such as air or stone), treating them like a single large block, but this works more in my engine because chunks are 16x16x16 (and stack vertically as well).
haven't personally messed with using octrees this way.