Advertisement Jump to content
  • Advertisement
Sign in to follow this  
  • entries
  • comment
  • views

Compressing LDraw database

Sign in to follow this  


One of the main goal for QLMesh was to add some new formats I have been working with quite often, like Photoshop files of bdf fonts.

For 3D it is LDraw formats and DAZ Studio models.

LDraw is one of my favourite. I am currently working on extending Assimp to support .ldr and .mpd files. One of the major challenge is actually not drawing but embedding library definitions into the plugin. Original library it is about 250MB (compressed to ~40MB). That's quite large for Quicklook plugin. I started to work on some heavy compression/optimalization and current result is:

-rw-r--r--  1 piecuchp  staff    40M May 12 17:18 parts.db
-rw-r--r--  1 piecuchp  staff   2.2M May 12 17:18 parts.db.gz

That's much better. 2MB can be easily embedded into plugin, eg. using assembler module like this:

bits 64

section .rodata

global _ldrawlib
global _ldrawlib_end
global _ldrawlib_size

_ldrawlib:      incbin "parts.db.gz"
_ldrawlib_size: dd $-_ldrawlib

and later build with e.g. nasm: 

/opt/local/bin/nasm -fmacho64 ldraw_lib.asm -o ldraw_lib.o



Sometimes less is more. Working on reading gzip stream, I had to remove one of the compression optimisation. The uncompressed file is slightly bigger, but compressed one much smaller:

-rw-r--r--  1 piecuchp  staff    41M Jun 17 12:03 parts.db
-rw-r--r--  1 piecuchp  staff   1.5M Jun 17 12:03 parts.db.gz


Sadly, this is not the end of the story :) I had to increase the precision of the float numbers in the database (it is now 17 bits - sign:8bit:8bit) - it increased the size but also significantly affected the compression ratio:

-rw-r--r--  1 piecuchp  staff    67M Jul 11 08:55 parts.db
-rw-r--r--  1 piecuchp  staff    41M Jul 11 08:55 parts.db.gz

Seems like I am gonna have to live with such database for a while.

Screen Shot.jpg


Sign in to follow this  


Recommended Comments

There are no comments to display.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now
  • Advertisement
  • Advertisement
  • What is your GameDev Story?

    In 2019 we are celebrating 20 years of! Share your GameDev Story with us.

    (You must login to your account.)

  • Blog Entries

  • Similar Content

    • By achiga
      I am new to shader programming. I was learning how to detect edges using shaders these days. And I found the UnityChan toon shader project. But I found it difficulty to understand when I read its implementation of sobel filter.
      Especially for these s few lines:(From Line166)
              depthsDiag.x = Linear01Depth(SAMPLE_DEPTH_TEXTURE(_CameraDepthTexture,i.uv[1]+uvDist)); // TR
              depthsDiag.y = Linear01Depth(SAMPLE_DEPTH_TEXTURE(_CameraDepthTexture,i.uv[1]+uvDist*float2(-1,1))); // TL
              depthsDiag.z = Linear01Depth(SAMPLE_DEPTH_TEXTURE(_CameraDepthTexture,i.uv[1]-uvDist*float2(-1,1))); // BR
              depthsDiag.w = Linear01Depth(SAMPLE_DEPTH_TEXTURE(_CameraDepthTexture,i.uv[1]-uvDist)); // BL
              depthsAxis.x = Linear01Depth(SAMPLE_DEPTH_TEXTURE(_CameraDepthTexture,i.uv[1]+uvDist*float2(0,1))); // T
              depthsAxis.y = Linear01Depth(SAMPLE_DEPTH_TEXTURE(_CameraDepthTexture,i.uv[1]-uvDist*float2(1,0))); // L
              depthsAxis.z = Linear01Depth(SAMPLE_DEPTH_TEXTURE(_CameraDepthTexture,i.uv[1]+uvDist*float2(1,0))); // R
              depthsAxis.w = Linear01Depth(SAMPLE_DEPTH_TEXTURE(_CameraDepthTexture,i.uv[1]-uvDist*float2(0,1))); // B
              depthsDiag -= centerDepth;
              depthsAxis /= centerDepth;
      It seems to try to get the diagonal and axial depth values. But I don't know why we need to substract centerDepth from depthsDiag and divide depthsAxis by centerDepth?
      Another confusion comes from these a few lines when it tries to return the final color from fragment shader:(From line 191)
              float SobelX = dot(SobelH, float4(1,1,1,1));
        float SobelY = dot(SobelV, float4(1,1,1,1));   float Sobel = sqrt(SobelX * SobelX + SobelY * SobelY);       Sobel = 1.0-pow(saturate(Sobel), _Exponent);   //NK   float4 Col = tex2D(_MainTex, i.uv[0].xy);   Col = _EdgesColor * Col * (1.0 - Sobel) + Sobel;   return Col * lerp(tex2D(_MainTex, i.uv[0].xy), _TestColor, _BgFade);  
      What does Sobel = 1.0-pow(saturate(Sobel), _Exponent) do? And what dose Col = _EdgesColor * Col * (1.0 - Sobel) + Sobel mean?
      Sorry for my bad English and hope anyone could help me understand this.
      Links to the git
    • By NikiTo
      I am having an argument about this with a buddy now, and we don't believe each other.
    • By Afonso Matos
      Hello everyone, it's my first time posting on this forum. I'm very excited to meet you.
      Anyways, my first question is about procedurally generating a world. I'm seeing how to implement a procedural generated world, and placing objects after that seem to be a big challenge.
      Let's say I have a big object that doesn't fit inside a chunk or that it is spawned in the frontier of a chunk. How would you go about generating the whole thing using the nearby chunks? With terrain it's pretty easy because the terrain is not hardcoded, so the algorithm is entirely free. But when I want to place hardcoded objects like castles or trees, I can't just cut the castle in half - or can I?
      What's a way to approach this problem?
      I'm using LWJGL/OpenGL in Java for a 2D game.
    • By iamyoukou
      I'm very interested in the snow physics in Red Dead Redemption 2.
      Here is a demo video from YouTube.
      Does anyone know the techniques behind this snow physics?
      For example,
      What is the shading model of the snow? How do they create snow deformation?
    • By NewtonEinherjar
      I'm writing a little game for visual deficient people, but I'm having a hard time getting the mouse position. Let me explain :
      I need to know where in the table the mouse cursor is, without having a click, and then I want to play a sound. That sound would be different for every position. Any thoughts? Thanks, in advance!
      e.g., when the mouse is on the 1st box would be played the audio "a1", when it's on the 2nd box, "a2", and so on.
      I tried with:
      mouse_x, mouse_y = get_Position() if mouse_x and mouse_y == map[x][y] then if map[x][y] == 0.1 then Audio:play() But it makes a loop and the sound keeps playing forever!

Important Information

By using, you agree to our community Guidelines, Terms of Use, and Privacy Policy. is your game development community. Create an account for your GameDev Portfolio and participate in the largest developer community in the games industry.

Sign me up!