Recently, I put together a benchmark for general-purpose lossless data compression. While I was doing so it occurred to me that the standard corpora aren't very representative of the type of data people actually use data compression for these days, and part of what is missing is data from games.

I created a C++ lossless bit compression stream a couple of years back as a quick prototype.

It's something I might use later for a game I'm making?

It probably needs to be RLE before the actual packing; maybe something to add another day?

https://github.com/cole-anstey/bit-compression-stream/wiki

Sorry, to be clear I'm not looking for more implementations of compression algorithms—well, I am, but that's not what this thread is about. What I am trying to find is the type of data that game developers typically need to compress so I can include it in a set of data I'm putting together to test different algorithms.

In addition to being used by developers to judge which compression algorithm(s) might be appropriate for them, people developing codecs will almost certainly be using this data to help tune their compressors, so getting something like the type of data you want to compress into this corpus should help make codecs better at handling that type of data—meaning better compression for games. AFAIK none of the currently used corpora (such as the Canterbury Corpus, Silesa Compression Corpus, Calgary Corpus, etc.) include any game data.

Good point—in this case I'm definitely more interested in the published files. The development files could be an interesting data set, but the published files are important to a lot more people. I'm not trying to create a corpus with every different type of data available, I'm trying to create something which will be relevant for the vast majority of people.

As for memory mapped files, I think the only reasonable thing to is ignore the issue. Either such cases cannot benefit from compression or the algorithm is constrained by the hardware, so there isn't really anything we can do here but focus on the use cases which could be served by compression.

The fact that LZW is used for that is interesting—thanks for that. I've been generally uninterested in adding LZW support to Squash, I'll have to rethink that. Any chance you could share the names of some of the platforms which can decode LZW like that?

I was indeed talking about compressing the wav's to ogg/mp3 (after they're "ready" for the game).

wav->mp3/ogg/opus is outside the scope of what I'm working on (which is lossless, general purpose compression). Honestly I think lossy audio compression is more interesting, but it's not what the corpus is for. I've filed an issue about 3D models, and I'll start trying to find some information about the formats different engines use—hopefully add some of the output from one of them to the corpus. My current idea is to try to get a 3D model from one of the Blender foundation's open movie projects, then figure out how to get that exported as an asset from Unity or something.

Yes, I had to check whether its been revealed on a non-NDA basis first. Its the XBox One. This public powerpoint from Microsoft/AMD reveal that the asynchronous 'Move engines' (a fancier DMA unity, basically) can both compress and decompress LZ data -- I'm not sure how inclusive of all the different LZ compression formats that is, though. It also can decompress JPEG, can swizzle textures, and perform memset-type operations. I believe that the PS4 does at least jpeg, and might do LZ as well, but I'm not certain.

I suspected an NDA was why you didn't just come out and say it in you initial post; thanks for taking the time to research this, I have a feeling a lot of people in the compression community weren't aware, maybe we can get some better encoders.

According to http://www.redgamingtech.com/xbox-one-sdk-leak-part-3-move-engines-memory-bandwidth-performance-tech-tribunal/ the LZ is LZ77. It also seems to imply that it is either DEFLATE, zlib, or gzip (though I'm not sure how reliable that article is…) which would make sense. If that's true, I would definitely be looking at zopfli for things you only have to compress when building a release but decompress regularly.

I don't know if you want to ignore the memory-mapped files -- They are platform specific, but in practice the two platforms (Xbox One and PS4) are similar enough that there might be common ground. But I'm speculating' I don't know, and probably couldn't say if I did.

Right, but I don't really think much can be done about the issue. Using that feature will obviously restrict people to the supported codecs, and people will probably be more interested in in writing improved implementations of the relevant codecs, but AFAICT from the perspective of developing a corpus nothing changes; you are still using the same type of data.