Jump to content

  • Log In with Google      Sign In   
  • Create Account


Image compression


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
19 replies to this topic

#1 maya18222   Members   -  Reputation: 187

Like
0Likes
Like

Posted 24 June 2012 - 09:57 AM

Can anyone suggest any quick image compression techniques for rgb 888 data? So far I'm using RLE and compressing down each component into 565.

Short of going full scale with a jpeg library, is there Anything else worth investigating?

Sponsor:

#2 Rattenhirn   Crossbones+   -  Reputation: 1688

Like
0Likes
Like

Posted 24 June 2012 - 10:53 AM

Lossless: libpng or just zlib
Lossy: libjpeg

Edit: libpng, not libpn...

Edited by Rattenhirn, 24 June 2012 - 11:56 AM.


#3 ProfL   Members   -  Reputation: 564

Like
0Likes
Like

Posted 24 June 2012 - 11:18 AM

are you looking for an algorithm you implement yourself or a lib you just integrate?

#4 maya18222   Members   -  Reputation: 187

Like
0Likes
Like

Posted 24 June 2012 - 12:08 PM

This is a hobby project, so something I could implement myself really, purely for the learning experience

#5 Rattenhirn   Crossbones+   -  Reputation: 1688

Like
0Likes
Like

Posted 24 June 2012 - 01:35 PM

This is a hobby project, so something I could implement myself really, purely for the learning experience


In that case I recommend either going for an algorithm in the LZ* family (LZ77, LZ78, LZW, LZMA, in order of difficulty) since they are somewhat simple and achieve good compression on general data. This is what Zip & friends are using. And even the worst LZ algorithm should beat LRE in almost every case as far as compression goes.

If you want something images specific (and lossy), I guess you could go for jpeg or a subset of it. However, all lossy algorithms are, ironically, a lot more complex than lossless ones. So prepare for a big learning experience. ;)

#6 Álvaro   Crossbones+   -  Reputation: 12353

Like
0Likes
Like

Posted 24 June 2012 - 01:46 PM

I think a fun project would be to implement image compression using wavelets. If you aren't willing to learn some heavy Math, maybe you should stick to lossless algorithms.

#7 clb   Members   -  Reputation: 1778

Like
0Likes
Like

Posted 24 June 2012 - 01:53 PM

I can appreciate the value of DIY in learning, so this might not apply for OP, but here's my recommendation: the crunch library. It combines lossy and lossless compression techniques, and decodes directly into DXT (=BCx/S3TC) formats. Decoding is extremely fast, disk sizes are smaller than what .zip/.7z formats offer and because of DXT, is nicely ready-to-use for GPU. The library is contained in three .h files, and integrating it (both to my asset build toolchain and runtime loader) was only a few hours of work, when I already had an existing DXT format support. I can't recommend it enough for anyone looking for GPU-friendly 3D texture formats.
Me+PC=clb.demon.fi | C++ Math and Geometry library: MathGeoLib, test it live! | C++ Game Networking: kNet | 2D Bin Packing: RectangleBinPack | Use gcc/clang/emcc from VS: vs-tool | Resume+Portfolio | gfxapi, test it live!

#8 ProfL   Members   -  Reputation: 564

Like
0Likes
Like

Posted 24 June 2012 - 05:35 PM

I think the first steps would be to try to reduce the image data (lossy).
- convert 4x4 pixel blocks to YCbCr colors, save Y for every pixel, but Cb and Cr just once per block -> 18byte/16pixel and you might not see any difference
- make a delta compression, e.g. for Y and CB/CR lines calculate a difference to the previous line, in photographies it will usually reduce a lot of data as pre-step for further compressions
-although RLE is a nice start, go on with LZ77, then LZ78 and LZSS are not that different, but can increase the compression ratio

#9 L. Spiro   Crossbones+   -  Reputation: 12675

Like
0Likes
Like

Posted 25 June 2012 - 04:54 AM

My engine uses a proprietary format with the extension .LSI.
It is fully lossless and:
Always Smaller Than the Same Image in:
  • .BMP
  • .TGA
  • .GIF
  • .DDS
Sometimes Smaller Than the Same Image in:
  • .PNG
  • .JPG (and is not lossy)
I documented the routine here. Note that while I mentioned a lossy version of the routine in the blog, all notes above are in regards to the lossless version.
Index compression is what usually wins because the number of bits used for the table is not fixed. If you have only a few colors in your image, the table could easily end up being only 5 or 6 bits per entry or so. The table itself is then compressed using a modified LZW routine, and the look-up table is also compressed.


This should give you plenty of ideas for your own format.


L. Spiro

Edited by L. Spiro, 27 June 2012 - 11:52 PM.

It is amazing how often people try to be unique, and yet they are always trying to make others be like them. - L. Spiro 2011
I spent most of my life learning the courage it takes to go out and get what I want. Now that I have it, I am not sure exactly what it is that I want. - L. Spiro 2013
I went to my local Subway once to find some guy yelling at the staff. When someone finally came to take my order and asked, “May I help you?”, I replied, “Yeah, I’ll have one asshole to go.”
L. Spiro Engine: http://lspiroengine.com
L. Spiro Engine Forums: http://lspiroengine.com/forums

#10 Krypt0n   Crossbones+   -  Reputation: 2357

Like
0Likes
Like

Posted 25 June 2012 - 07:35 AM

...

L. Spiro

why don't you upload some sample pictures, so we could see with our own eyes what ratio you got and what usual png optimizers deliver? I'd be curious to validate the results.

#11 L. Spiro   Crossbones+   -  Reputation: 12675

Like
0Likes
Like

Posted 25 June 2012 - 08:27 AM

There is no public-domain decoder for .LSI images, so you would not be able to actually open the files, but when converting models it shows me the size of the .LSI file as a percentage of the original file.
Here are some results.
TGA/BMP files can offer significant size decreases (on the second screenshot there was a TGA amongst all the PNG’s compressed to 0.804009% of its original size (1,048,620 bytes to 8,431 bytes) but I did not include it in the shot).
LSTGARes.png

Here are .PNG files recompressed as .LSI. When the .LSI file is larger than the .PNG file (or whatever the source format is), the converter uses the original file, so the percent is 100%. That means .PNG won.
LSPNGRes.png

With .PNG it is often close, and .PNG sometimes wins, but .LSI wins often as well, and sometimes by quite a large margin (as low as 25% in this case).



L. Spiro

Edited by L. Spiro, 25 June 2012 - 09:12 AM.

It is amazing how often people try to be unique, and yet they are always trying to make others be like them. - L. Spiro 2011
I spent most of my life learning the courage it takes to go out and get what I want. Now that I have it, I am not sure exactly what it is that I want. - L. Spiro 2013
I went to my local Subway once to find some guy yelling at the staff. When someone finally came to take my order and asked, “May I help you?”, I replied, “Yeah, I’ll have one asshole to go.”
L. Spiro Engine: http://lspiroengine.com
L. Spiro Engine Forums: http://lspiroengine.com/forums

#12 ProfL   Members   -  Reputation: 564

Like
0Likes
Like

Posted 25 June 2012 - 01:11 PM

share at least the source png with us ;)

#13 L. Spiro   Crossbones+   -  Reputation: 12675

Like
0Likes
Like

Posted 26 June 2012 - 06:11 AM

Here are some images and their ratios.

.PNG = 40,725, .LSI = 45,094 = 110.72%
CORVETTEZ06 RIM.png


.PNG = 32,865, .LSI = 29,990 = 91.25%
CORVETTEZ06 BADGING MASK.png


.PNG = 14,460, .LSI = 5,209 = 36.02%
TIREBACK.png


.PNG = 55,564, .LSI = 44,546 = 80.16%
TIRE BUMP.png


.PNG = 99,420, .LSI = 130,098 = 130.86%
CORVETTEZ06 BRAKELIGHT.png


Photographic images tend to be larger than .PNG, but I haven’t seen any go above 175% yet, which is still quite acceptable given how small .PNG’s are.

On the other hand cartoony images and masks compress very well compared to .PNG’s.

Normal maps are generally similar in size.


L. Spiro

Edited by L. Spiro, 26 June 2012 - 08:17 PM.

It is amazing how often people try to be unique, and yet they are always trying to make others be like them. - L. Spiro 2011
I spent most of my life learning the courage it takes to go out and get what I want. Now that I have it, I am not sure exactly what it is that I want. - L. Spiro 2013
I went to my local Subway once to find some guy yelling at the staff. When someone finally came to take my order and asked, “May I help you?”, I replied, “Yeah, I’ll have one asshole to go.”
L. Spiro Engine: http://lspiroengine.com
L. Spiro Engine Forums: http://lspiroengine.com/forums

#14 L. Spiro   Crossbones+   -  Reputation: 12675

Like
0Likes
Like

Posted 26 June 2012 - 08:12 PM

I hit Quote instead of Edit. Please remove this post.


L. Spiro
It is amazing how often people try to be unique, and yet they are always trying to make others be like them. - L. Spiro 2011
I spent most of my life learning the courage it takes to go out and get what I want. Now that I have it, I am not sure exactly what it is that I want. - L. Spiro 2013
I went to my local Subway once to find some guy yelling at the staff. When someone finally came to take my order and asked, “May I help you?”, I replied, “Yeah, I’ll have one asshole to go.”
L. Spiro Engine: http://lspiroengine.com
L. Spiro Engine Forums: http://lspiroengine.com/forums

#15 Steveway   Members   -  Reputation: 302

Like
3Likes
Like

Posted 27 June 2012 - 04:24 AM

Since nobody bothered to do so till now, I ran a simple pngcrush -brute over some of your pictures.
(I only compressed the pictures that are "bigger" than your self-compressed ones.)
The results are:
25.800 Bytes
test


10.155 Bytes
test1


47.562 Bytes
test2


So, your own algorithm does not fare that well against a more correctly use .png.
Even though 2 of the examples are still bigger than yours (even though we have to belive that you don't lie to us since you neither posted one of those pictures or any other proof.) png still wins in most cases and has the benefits of being a free specification.
EDIT: Sounds a little bit provocative, don't take it too seriously, I just like the topic.

Edited by Steveway, 27 June 2012 - 04:46 AM.


#16 jrh2365   Members   -  Reputation: 595

Like
3Likes
Like

Posted 27 June 2012 - 10:28 AM

I haven't heard of pngcrush before, but apparently it has lots of room for improvement. I ran the images through pngout and optipng, and this is the result:

18,022 bytes
test1


4,303 bytes
test2


32,768 bytes
test3

Edited by jrh2365, 27 June 2012 - 10:29 AM.


#17 Krypt0n   Crossbones+   -  Reputation: 2357

Like
2Likes
Like

Posted 27 June 2012 - 11:32 AM

also aimed to try it with optipng, similar to what L.spiro does, it 'losslessly' reduces the images to as few colors as possible before it starts the binary compression.

in cases where this is not possible (photography), png was better anyway. which leads to: the lzw algorithm used in LSI is not reaching the compression ration of deflate in png, which shouldn't be of a surprise, the way it combines lz77 with huffman is very smart, IMHO.

#18 L. Spiro   Crossbones+   -  Reputation: 12675

Like
2Likes
Like

Posted 27 June 2012 - 11:50 PM

Even though 2 of the examples are still bigger than yours (even though we have to belive that you don't lie to us since you neither posted one of those pictures or any other proof.) png still wins in most cases and has the benefits of being a free specification.
EDIT: Sounds a little bit provocative, don't take it too seriously, I just like the topic.

I admit that I just used the .PNG images as they were in the model packages I got. I assume these represent the average use case.

I am not too disappointed at slightly losing to .PNG in lossless form (lossy compression still wins handily with pretty nice results). I will have to update my posts and blog, but the main point is that these are practical and easy-to-implement ideas for getting small image files while maintaining losslessness.

I have a few more ideas that were more complicated to implement so I have been putting them off, but maybe I will implement them soon and see how they fair against aggressively optimized .PNG files.


Also, my images will be “free specification” soon. I am not keeping secrets etc., which is why I documented the algorithms roughly in the first place. I will get around to making Photoshop plug-ins and an official specification eventually.


L. Spiro
It is amazing how often people try to be unique, and yet they are always trying to make others be like them. - L. Spiro 2011
I spent most of my life learning the courage it takes to go out and get what I want. Now that I have it, I am not sure exactly what it is that I want. - L. Spiro 2013
I went to my local Subway once to find some guy yelling at the staff. When someone finally came to take my order and asked, “May I help you?”, I replied, “Yeah, I’ll have one asshole to go.”
L. Spiro Engine: http://lspiroengine.com
L. Spiro Engine Forums: http://lspiroengine.com/forums

#19 Krypt0n   Crossbones+   -  Reputation: 2357

Like
2Likes
Like

Posted 28 June 2012 - 03:33 AM

I admit that I just used the .PNG images as they were in the model packages I got. I assume these represent the average use case.

yeah, you are right and it's sad that most tools don't have even minimal optimizations for png, just vanilla libpng mostly!

I am not too disappointed at slightly losing to .PNG in lossless form ... but the main point is that these are practical and easy-to-implement ideas for getting small image files while maintaining losslessness.

it's easy, but from the algorithm point of few it's also inferior as you use very basic compression ideas. you should read how to combine LZ.. algorithms with entropy encoded algorithms like huffman and artihmetic encoding. additionally some transformations like burrows wheeler transform, move to front, prediction coding etc. are very simple to implement and can be done as a pre-step to optimize the data for better compression ratios.
beside compression ratio, it's also important how fast it is to compress/uncompress and how much memory you need. png is very memory friendly, it works in tiny chunks of 64kb, which can be implemented on SPUs of the CELL (PS3), and also on embeded boards, even on GBA some games used it.


...(lossy compression still wins handily with pretty nice results). ...

feel free to post a png image of a previously lossy compressed LSI. I'm sure png will win once again.


I have a few more ideas that were more complicated to implement so I have been putting them off, but maybe I will implement them soon and see how they fair against aggressively optimized .PNG files.

you should first find out how exactly png works, to not reinvent the wheel, because the first few times, you will be inferior if you don't learn from previous work of others,my 2 cent.

Also, my images will be “free specification” soon. I am not keeping secrets etc., which is why I documented the algorithms roughly in the first place. I will get around to making Photoshop plug-ins and an official specification eventually.

may I suggest that you stop writing a proprietary format that probably nobody will ever use beside you and invest your time into a good png optimizer? your quantitization algorithms will probably work just as good if afterwards compressed with png, as they would work with LSI, but the ratio is probably better. Beside that, png has still a lot of potential to reach higher compression ratios and if you do it, your png-L.spiro plugin might be used by a lot of people. Isn't that the whole point of making something public?

I don't wan to kill your fun or motivation, just some suggestions ;)

#20 L. Spiro   Crossbones+   -  Reputation: 12675

Like
0Likes
Like

Posted 28 June 2012 - 06:32 AM

I think we have a misunderstanding.
My goal was never to beat .PNG. I know how it works (I wrote the decoder I am using for it) but that was too much trouble considering my goal: A format suited specifically for games, but still small in size for mobile devices and lossless for quality.

That means direct support for a lot of common formats used in games such as R5G6B5, A8, A8L8, R8G8, R16B16G16A16, etc.
That also means embedded mipmaps.


As for who may or may not end up using my plug-in or format, the goal of my engine is commercial and if it does well enough I will use it to start a middleware company. So it would be unhealthy for me to doubt whether or not people will end up using it.
Right or wrong, everything I make is made with the assumption that people will be using it someday.


Maybe someday I will get around to writing an encoder for .PNG and start applying it to my format, but I have a lot to do—I haven’t even had the time to finish my DXT compressor and make a release—so it is not a high priority right now.


L. Spiro
It is amazing how often people try to be unique, and yet they are always trying to make others be like them. - L. Spiro 2011
I spent most of my life learning the courage it takes to go out and get what I want. Now that I have it, I am not sure exactly what it is that I want. - L. Spiro 2013
I went to my local Subway once to find some guy yelling at the staff. When someone finally came to take my order and asked, “May I help you?”, I replied, “Yeah, I’ll have one asshole to go.”
L. Spiro Engine: http://lspiroengine.com
L. Spiro Engine Forums: http://lspiroengine.com/forums




Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS