Sign in to follow this  
Endar

Codebloat and templates - big deal?

Recommended Posts

Endar    668
For my last engine, I was compiling it to a static library, the debug version of which actually came to a total of 18mb and the release version came to 16mb. For one part, I'm actually generating a total of 22 template classes using boost preprocessor magic. They're not very big themselves, but they do use other parts of the boost libraries. Is it possible that generating these 22 classes and using them is causing such ridiculous sizes for the libraries? I think I'm also using RTTI. Does this have a big impact on the size of a program or library? PS. For those who don't know, the dll of the Irrlicht open source graphics engine (version 1.0) is a total of 1.5mb, and it does approximately 22079460347 times better than mine (just an approximation of course :D ).

Share this post


Link to post
Share on other sites
RDragon1    1205
More importantly - how big is a release executable linked against that static library?

Did you strip out all debug information & symbols?

I doubt templates are taking up ~20mb. Code is pretty small.

And which boost libraries are you using?

Share this post


Link to post
Share on other sites
Endar    668
Quote:
Original post by iMalc
With a program that big it has to be due to a declaration of a large static array, not code bloat.


I have a single static class, which isn't very big at all. It's members are an integer and a dynamic array, which allocates the array space on the heap.

That's it. I have a lot of stuff with the word "static" in their names (not the keyword), but they have nothing to do with actual static variables.


Quote:
Original post by RDragon1
More importantly - how big is a release executable linked against that static library?

Did you strip out all debug information & symbols?

I doubt templates are taking up ~20mb. Code is pretty small.

And which boost libraries are you using?


I've compiled a basic tetris game (not completed) that uses my engine, and it is a total of 560kb with the library being 16mb.

I was also fairly sure that code is pretty small, but I was a little afraid that generating classes at compile time could rack up some space issues.

Okay, boost libraries. Do you mean actual libraries that I'm linking to? Because I'm linking only to the boost::regex library, but I'm #includ'ing a whole bunch of stuff to use:

  • boost::format
  • boost::variant
  • boost::regex
  • boost::preprocessor
  • boost::lexical_cast
  • boost::tuple
  • boost::type_traits
  • boost::function
  • boost::bind

So, with these ones, I'm simply including the headers, with the exception of regex, which I'm linking against it's library.

Yes, I'm as sure as I can be that I've set it to not include debug information. I've taken a look at the settings at anything that looks like it's associated with debugging, and it is all "no" for compiling a release version (and I did re-compile the library in release configuration).

Share this post


Link to post
Share on other sites
Xai    1838
the word "static" is extremely overloaded in the programming world. When they said it must be due to a large static library, they did not mean any code you have written yourself ... they meant code someone else had written, that you use, that is linked into your library or program statically (as a static library versus a dynamic link library).

For example the microsoft visual c++ runt time is about 0.5 mb, and the libc is about 2.5 mb ... many libraries are larger ...

besides boost, what do you link to in your program?

Share this post


Link to post
Share on other sites
Endar    668
Apart from boost? I link to the OpenGL libraries, the standard C++, and probably C stuff. I use tinyxml, but I'm compiling the source for that, and not linking to a library. Also the stuff for compiling windows programs using VC++ express.

That's it. Nothing special or unusual as far as I know.

Share this post


Link to post
Share on other sites
Nitage    1107
Quote:

I've compiled a basic tetris game (not completed) that uses my engine, and it is a total of 560kb with the library being 16mb.


Your executables are normal size - but your library is huge. What's probalby happening is that your library is full of stuff that's never being used - it's generally not a good idea to put explicit template instantiations in libraries, as they increase the size of the library even if they aren't used. Just stick with a header only library for template classes.

FYI, when your linker creates executables using your library, it only includes the code that's actually used, throwing away all the junk.

Share this post


Link to post
Share on other sites
Zahlman    1682
Quote:
Original post by Nitage
Quote:

I've compiled a basic tetris game (not completed) that uses my engine, and it is a total of 560kb with the library being 16mb.


Your executables are normal size - but your library is huge. What's probalby happening is that your library is full of stuff that's never being used - it's generally not a good idea to put explicit template instantiations in libraries, as they increase the size of the library even if they aren't used. Just stick with a header only library for template classes.

FYI, when your linker creates executables using your library, it only includes the code that's actually used, throwing away all the junk.


For statically linked libraries, yes. For dynamically linked libraries, it basically doesn't include anything - but you have to keep a copy of the DLL around in order for the exe to work :)

Share this post


Link to post
Share on other sites
Endar    668
Quote:
Original post by Nitage
Your executables are normal size - but your library is huge. What's probalby happening is that your library is full of stuff that's never being used - it's generally not a good idea to put explicit template instantiations in libraries, as they increase the size of the library even if they aren't used. Just stick with a header only library for template classes.

A header only library for template classes? I'm not quite understanding that. The template classes are being generated in a header file, but of course are being included in source files.

Quote:
Original post by Zahlman
For statically linked libraries, yes. For dynamically linked libraries, it basically doesn't include anything - but you have to keep a copy of the DLL around in order for the exe to work :)

You don't have to keep a copy of the library around when you're using a static library?

Share this post


Link to post
Share on other sites
Vorpy    869
In this case, static vs dynamic is referring to compile time vs run-time. A static library is linked at compile time, so the code is copied into the resulting executable (and there might even be some optimizations around the call boundaries). A dynamic library is linked at run-time, by the executable telling the operating system to load the dynamic library into its memory and locating the required functions. This allows the library to be replaced by a newer version without having to recompile; thus, the linking is more dynamic.

Only dynamic libraries are needed at run-time. The needed parts of static libraries were already inserted into the executable.

Share this post


Link to post
Share on other sites
Zahlman    1682
Quote:
Original post by Endar
Quote:
Original post by Nitage
Your executables are normal size - but your library is huge. What's probalby happening is that your library is full of stuff that's never being used - it's generally not a good idea to put explicit template instantiations in libraries, as they increase the size of the library even if they aren't used. Just stick with a header only library for template classes.

A header only library for template classes? I'm not quite understanding that. The template classes are being generated in a header file, but of course are being included in source files.


The idea is, don't try to instantiate the template classes or do anything in source files. Put the whole implementation of the template into the header (or a .inl file included from the header), and don't have any explicit instantiations. Then when the user includes your headers, the net effect is as if s/he had written the code as part of the project: nothing is put into the .lib, and only the instantiations that are *used* are put in the .exe.

Quote:
Quote:
Original post by Zahlman
For statically linked libraries, yes. For dynamically linked libraries, it basically doesn't include anything - but you have to keep a copy of the DLL around in order for the exe to work :)

You don't have to keep a copy of the library around when you're using a static library?


Nope. That's what makes it "static". :)

Share this post


Link to post
Share on other sites
RAZORUNREAL    567
Quote:
Original post by Zahlman
The idea is, don't try to instantiate the template classes or do anything in source files. Put the whole implementation of the template into the header (or a .inl file included from the header), and don't have any explicit instantiations. Then when the user includes your headers, the net effect is as if s/he had written the code as part of the project: nothing is put into the .lib, and only the instantiations that are *used* are put in the .exe.


Just to clarify, that's for when you want to expose a template class to the user of the static library. It doesn't apply to using templated classes in the library code itself.

Share this post


Link to post
Share on other sites
Endar    668
Quote:
Original post by RAZORUNREAL
Quote:
Original post by Zahlman
The idea is, don't try to instantiate the template classes or do anything in source files. Put the whole implementation of the template into the header (or a .inl file included from the header), and don't have any explicit instantiations. Then when the user includes your headers, the net effect is as if s/he had written the code as part of the project: nothing is put into the .lib, and only the instantiations that are *used* are put in the .exe.


Just to clarify, that's for when you want to expose a template class to the user of the static library. It doesn't apply to using templated classes in the library code itself.


Hmmmm. Okay, I'm a little lost, I think.

ToDo:

  • Put the whole implementation of the template into the header - Check
  • Don't have any explicit instantiations of these template classes in the source code of the library - Check


... Okay, I actually found some other stuff, functions that I'm generating to actually add pointers to the previously generated classes (which all inherit from the same base class) into an array for other stuff which has nothing important to do with any particular thing. So, I'm generating one function for each class I generate (all classes have different numbers of template parameters). It's much less code than the classes themselves and I still am not creating an instance of any of these classes.

So, what in the world is going on?

Maybe I should just set up my next engine (major re-write of the current one) to be compiled to a dll (which I was planning to do anyway) and just keep an eye on it's size after every major addition of a subsystem.

Share this post


Link to post
Share on other sites
Endar    668
Quote:
Original post by TrueTom
Let the linker create a map file and you see what's going on.


Done. It was created using my Tetris game project. I'm not sure how to create a map file, when compiling only the library. How do I know what's important and what's not?

Edit:: Keep in mind that all the hosting I have is free geocities, and that has a 4.2MB transfer limit every hour. The file is 1.8MB. If someone would suggest somewhere else to host the file, I'll post it somewhere with a larger transfer allowance.

Quote:
Original post by Shannon Barber
The actual .lib is 16MB or all the crap MSVC generates is 16MB?

Only thing I can think of is resources, and you added some huge bitmaps or something.


The actual lib file is 16MB. As far as I know, I haven't added any resources. The only thing that the lib has is compiled code.

Edit:: Also, on a sort of related tangent, do you guys have any good resources for learning how to compile dlls?

[Edited by - Endar on November 28, 2006 1:32:01 AM]

Share this post


Link to post
Share on other sites
RobTheBloke    2553
That's not all that unusual with static libs. The FBX SDK for example comes in at 35Mb - I have a static build of my animation pipeline lib that is 70+ Mb. All of that collapses to a 1.2Mb exe so who cares. If you want to distribute it in a smaller size, compile the lib as a dll.

Share this post


Link to post
Share on other sites
Endar    668
Quote:
Original post by RobTheBloke
That's not all that unusual with static libs. The FBX SDK for example comes in at 35Mb - I have a static build of my animation pipeline lib that is 70+ Mb. All of that collapses to a 1.2Mb exe so who cares. If you want to distribute it in a smaller size, compile the lib as a dll.


Really? What's the difference between dynamic and static libraries (besides the linking)? What makes them so different in sizes?

Share this post


Link to post
Share on other sites
ApochPiQ    23003
A static library has to retain annotations and metadata so the linker knows how to merge the static code with your own. This can all be safely stripped out when compiling as a DLL, which (more or less) is treated as if it were a chunk of our own EXE.

This is also why the total size of stuff like .obj files generated by the compiler will usually be much larger than the final binary.

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this