Jump to content

  • Log In with Google      Sign In   
  • Create Account

How Can I Reduce My Compile Times?


Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.

  • You cannot reply to this topic
21 replies to this topic

#1 hoffman_jwr   Members   -  Reputation: 102

Like
1Likes
Like

Posted 09 January 2011 - 05:18 PM

Hi there,
I am a graduate games programmer with around 6 months industry experience, and this dilemma is something that I often ponder. Up to this point, I have worked on a few projects which have grown to become substantial in size (approx. twenty - thirty thousand lines). Although I have strived for software orthogonality (i.e. modularity), and worked towards decoupling the various aspects of the systems I worked on, inevitably the projects bloat, interdependencies pop up, and compile times become long. I work in predominantly in C; I always ensure forward declarations over header includes, I use pointers wherever possible. Is it possible to modularise to the point where a module is completely free-standing? Would it be possible to place the various components of an engine into dll files (something I have never done)? On large scale projects, is this practice? - or does the solution reside in the one project file (somehow I doubt that)? dll's are windows specific; are there any platform unspecific ways to build linkable object files?

Sponsor:

#2 Steve132   Members   -  Reputation: 433

Like
0Likes
Like

Posted 09 January 2011 - 05:37 PM

Do you mean just creating a static library

#3 obhi   Members   -  Reputation: 180

Like
0Likes
Like

Posted 09 January 2011 - 05:41 PM

Grouping modules by what they do is a good practice. A lot of engines do that. The base dll defines only interfaces and plugins implement sub-systems. It will definitely reduce dependencies to only 1 dll (the base module). Having said that, a good way to reduce compile times is to use pre-compiled header files. The idea may seem trivial but having a good set of base headers that seldom change in a single header file and compiling that file once will reduce build times significantly. I am writing this from my personal experience.
What if everyone had a restart button behind their head ;P

#4 ApochPiQ   Moderators   -  Reputation: 15716

Like
0Likes
Like

Posted 09 January 2011 - 07:02 PM

It sounds like you already have a pretty good handle on the basics. Really it's a matter of keeping things as decoupled as possible, and minimizing the scope of dependencies. The real killers are things that crop up in, say, 75% of your code base and require vast recompiles every time something subtle changes.

One thing a lot of people overlook is dividing a single module into multiple header/code files. For instance, you may have a chunk of game logic that affects a lot of the engine; split it up into smaller groups based on whatever criteria makes the most sense, and go from there. This can radically reduce compile times when you only make localized, contained changes. Be careful, though, as on some compilers having too many files can actually end up costing you time due to the spin-up cost of starting compilation separately on every file.

Short of that... throw more hardware at it Posted Image


Honestly, though, 20-30 KLOC is nothing. Wait until you're in the millions of lines of code range and a full rebuild takes an hour or two. That's when it's worth doing major devilry to make your compile times faster.

#5 KulSeran   Members   -  Reputation: 2538

Like
0Likes
Like

Posted 09 January 2011 - 07:47 PM

For starters, it seems like you've been on the right track. But my question is going to be: what compile time is getting too large?

Look into a build solution like Cruise Control to take care of the full build+test cycle for you. That way, the longest build times are hopefully dealt with on a dedicated machine.

Make sure your build system is setup properly. For instance "make" by default requires the "-j" option to be added to get a multi-threaded build. Make sure your dependencies are properly being maintained if you don't have an IDE that generates them for you. Or better yet, spend the time to make sure it is automated.

Make sure your interfaces are being respected. Systems can get too tangled together when someone on the team decides to grab what was an internal header and use it somewhere because it had a feature they needed. In theory, you should be able to seperate collections of subsystems into static libraries and only rebuild them as needed (ie, compile Bullet Physics as a static lib, and add it to the project). Only go with the dll/so runtime loading if you gain something from it.

One thing a lot of people overlook is dividing a single module into multiple header/code files.


Seconded. If you have a "mathutil.h" floating around, consider breaking it up. Maybe it contains a lot of unrelated systems. You could end up with "interpolate.h", "springs.h", and "range_limits.h" as well, because people just tossed "math" stuff in there when they needed to wrap, clamp, mirror, spring, blend, accelerate_smooth... etc. Go further? have a spring1d.h spring2d.h and spring3d.h if there are enough functions to warrant it.

#6 Sneftel   Senior Moderators   -  Reputation: 1781

Like
0Likes
Like

Posted 09 January 2011 - 07:53 PM

Seconded (thirded?) about precompiled headers. A lot of programmers seem oddly sour on PCHes, possibly because their very first C++ experience was MSVC complaining that it couldn't find the #include <stdafx.h>. But I've had it reduce total project compile times by a factor of eighty.

#7 GothSeiDank   Members   -  Reputation: 156

Like
0Likes
Like

Posted 10 January 2011 - 06:20 AM

Like others said, use precompiled Headers. They exist for MSVC and for GCC as far as I know.
They already reduce your compile time by at least 60%.
Just make sure you only add "static" Headers to your include-file which the PCH is built from.
If you include headers there which are changed very often, you save nothing.
But adding there all the boost-, stl-, library-headers you never gonna change, reduces your compile time dramatically.
HTH
regards
If you say "pls", because it is shorter than "please", I will say "no", because it is shorter than "yes"
http://nightlight2d.de/

#8 hoffman_jwr   Members   -  Reputation: 102

Like
0Likes
Like

Posted 10 January 2011 - 10:26 PM

All very good advice, thanks to everyone! A common suggestion seems to be the use of precompiled headers, something that I will look into. When you say precompiled headers, do you mean header files that will not have to be rebuilt after a full clean and rebuild? For the record, we are using gcc, and the full rebuild time on our current project is around 6 minutes. Essentially, what I am working towards is a way to add (unchanging) modules to new projects without having to transfer all of the implementation files; I think this is a static library?

Thanks for not flaming me Posted Image

#9 GothSeiDank   Members   -  Reputation: 156

Like
0Likes
Like

Posted 11 January 2011 - 03:58 AM

All very good advice, thanks to everyone! A common suggestion seems to be the use of precompiled headers, something that I will look into. When you say precompiled headers, do you mean header files that will not have to be rebuilt after a full clean and rebuild? For the record, we are using gcc, and the full rebuild time on our current project is around 6 minutes. Essentially, what I am working towards is a way to add (unchanging) modules to new projects without having to transfer all of the implementation files; I think this is a static library?

Thanks for not flaming me Posted Image


Well, yes ofc, if you want to reuse self programmed modules, use static libraries.
Precompiled Headers is a newer technology which allows the Compiler to "precompile" Headerfiles you gonna use.
Basically it makes a static library out of all your headers. While this is technically not entirely true it explains best whats happening.
That reduces compile time too since the Headers do not need to be parsed every time you build if the headers are not changed.

See:
http://msdn.microsoft.com/en-us/library/z0atkd6c%28v=vs.80%29.aspx
http://gcc.gnu.org/onlinedocs/gcc/Precompiled-Headers.html
http://en.wikipedia.org/wiki/Precompiled_header

I would adive you to make a static or dynamic library out of your modules and to use PCH.
A combination is best.
If you say "pls", because it is shorter than "please", I will say "no", because it is shorter than "yes"
http://nightlight2d.de/

#10 hoffman_jwr   Members   -  Reputation: 102

Like
0Likes
Like

Posted 12 January 2011 - 04:30 PM

Awesome advice, have been looking into PCH and the creation of static libraries; exactly what I was looking for. Thanks to all. Posted Image

#11 Zoner   Members   -  Reputation: 232

Like
0Likes
Like

Posted 19 January 2011 - 01:56 PM

Precompiled Headers just let multiple C++ files share a common set of header files so the compiler doesn't have to waste time reading, parsing, and compiling that code over and over and over.

Other approaches to reduce build times are to:

1) thread the build process and have machines with lots of cores do the building (this is all good until linking however)

2) Merge more cpp files together (Unreal's build tool does this, via temp files and #include'ing the cpps)

3) Redesign various subsystems to use a PIMPL design so the dependency on the subsystem can be reduced to an opaque pointer of some sort. This is very similar conceptually to using virtual functions in a class, except instead of making the functions virtual, PIMPL makes the data virtual. As a bonus the functions can be non-virtual if you don't need inheritance so the penalty for doing a data abstraction is quite small. This approach really really helps avoid having to do full rebuilds when changing the implementation, as only interface changes will causes larger rebuilds. You can also think of this approach as 'make a C interface and use that, with a class wrapper helper to avoid polluting the global namespace'. C interfaces for FILE via fopen/fclose are a PIMPL design for instance. In addition making a C interface can help with other features in other systems (lua binding etc)

4) Keep template classes bare bones or simplistic since highly specialized templates (say a file system wrapper or something which could have a lot of platform specific code inside) require pulling in insane amounts of dependies into the header in order to be able to do everything, and this hurts build times a lot.

5) Not including windows.h in files that don't need it can really help but this can be difficult to accomplish at times especially if you need the thread sync primitives (CRITICAL_SECTION, Events etc). Its possible to do a PIMPL design with classes that wrap the sync primitives and allocate them all from a heap or a pool of sync primitives, but you will lose out on some features (difficulty in getting arrays of Events for use with WaitForMultipleObjects for instance).

6) Implement a scripting language, so you don't have to rebuild the C++ code for every little change.

7) Make as much of the codebase data driven as possible, again so you don't have to rebuild the C++ code for every little change.

8) If link times are long you could break the project up into DLLs for development, though inlined things like math libraries are still going to ripple out and rebuild everything. A game engine is far better off as a monolithic app though but that could just be the difference between a shipping build and a developmental one.

#12 GothSeiDank   Members   -  Reputation: 156

Like
0Likes
Like

Posted 19 January 2011 - 03:11 PM

6) Implement a scripting language, so you don't have to rebuild the C++ code for every little change.


I would recommend to use AngelScript for that, forums here on gamedev.net :D. Awesome ScriptingLanguage and easy to integrate.
If you say "pls", because it is shorter than "please", I will say "no", because it is shorter than "yes"
http://nightlight2d.de/

#13 Seaßourne   Members   -  Reputation: 104

Like
0Likes
Like

Posted 29 January 2011 - 10:51 PM

Use the PIMPL idiom. You code a public class and a private class. You keep all your private functions in the private class and just show the public/protected in the public. By doing this you memorise the headers in the public class and using also forward declarations will help minimise the header look up.

But also remember PIMPL is only useful for libraries, it's useless in main applications like exe's.

#14 adam4813   Members   -  Reputation: 659

Like
0Likes
Like

Posted 30 January 2011 - 12:51 AM

Another simple and obvious one I used to do to often, including iostream in nearly every file. I actaully rarely use it, mostly I just include string, vector, and map where needed, and I try not to use windows.h with WIN32_MEAN_AND_LEAN.

#15 0BZEN   Crossbones+   -  Reputation: 2016

Like
0Likes
Like

Posted 20 February 2011 - 04:44 PM

more megahurtz.

Not a fan of precompiled headers. Often it is a dump of various headers, and if you are not careful, can lead to poor practises.

But usually, reduce dependencies in code (kurbing on the number of headers, especially the ones involving templates or nasty buggers like windows.h, reducing their dependencies by using forward declarations, ..). Being disciplined about it and not going 'I'll fix that later' helps as well, as often it just creeps unnoticed (until you start slowing down again).

If you have big monolithic classes, that sometimes incurs a profusion of headers, and probably a rethink of what that class is suppose to do. Can't remember the name of the tool, but there are some that helps you track down dependencies and eliminate them. Sometimes it's not possible, and if something seems out of place, then your original design could be wrong in the first place.

Splitting a program into libraries is a way to force you to partition your code into logical units, and reduce dependencies between modules. however doing that after the fact is immensely painful. There is also a lot of time consumed by maintaining the discipline, and it is easy to over-do it.

Everything is better with Metal.


#16 Str3jda   Members   -  Reputation: 273

Like
0Likes
Like

Posted 20 February 2011 - 05:03 PM

You may minimalize parsing headers by creating one or more special cpp files that include all other cpps and defining special macro that prevents parsing them again. Ok, example:

-- project.cpp
#define COMPILE_SPEEDUP
#include "a.cpp"
#include "b.cpp"

--a.cpp
#ifdef COMPILE_SPEEDUP
...
#endif


--b.cpp
#ifdef COMPILE_SPEEDUP
...
#endif

As a result, project.cpp will compile everything and all headers are parsed just once. It certainly helped us.

System programmer at 2K Games


#17 RavNaz   Members   -  Reputation: 100

Like
0Likes
Like

Posted 04 March 2011 - 07:55 AM

fastest builds I've ever seen have been with unity builds.

http://buffered.io/2007/12/10/the-magic-of-unity-builds/

wickedly fast.

#18 Antheus   Members   -  Reputation: 2397

Like
0Likes
Like

Posted 04 March 2011 - 08:22 AM

fastest builds I've ever seen have been with unity builds.

http://buffered.io/2...f-unity-builds/

wickedly fast.


If this speeds up the builds, then your bottleneck are includes and include parsing. Precompiled headers would achieve similar effect.

If your bottleneck is linker, the above will have less of an impact.

#19 gekko   Members   -  Reputation: 478

Like
0Likes
Like

Posted 04 March 2011 - 02:47 PM

fastest builds I've ever seen have been with unity builds.

http://buffered.io/2...f-unity-builds/

wickedly fast.


Fast, yes. But it does come with side-effects.
-- gekko

#20 Pomnico   Members   -  Reputation: 110

Like
0Likes
Like

Posted 29 March 2011 - 03:04 AM

I've heard about one more method to increase performance of compile / link. In case you have a lot of source / header files and its disk operations that are slowing you down, you could create a RamDisk and copy your project to it. You could compile from the RamDisk in such a case, what would eliminate disk read operations cost. Never tried this myself, but I've heard it can increase compile time quite a lot in some cases.




Old topic!
Guest, the last post of this topic is over 60 days old and at this point you may not reply in this topic. If you wish to continue this conversation start a new topic.



PARTNERS