• Announcements

    • khawk

      Download the Game Design and Indie Game Marketing Freebook   07/19/17

      GameDev.net and CRC Press have teamed up to bring a free ebook of content curated from top titles published by CRC Press. The freebook, Practices of Game Design & Indie Game Marketing, includes chapters from The Art of Game Design: A Book of Lenses, A Practical Guide to Indie Game Marketing, and An Architectural Approach to Level Design. The GameDev.net FreeBook is relevant to game designers, developers, and those interested in learning more about the challenges in game development. We know game development can be a tough discipline and business, so we picked several chapters from CRC Press titles that we thought would be of interest to you, the GameDev.net audience, in your journey to design, develop, and market your next game. The free ebook is available through CRC Press by clicking here. The Curated Books The Art of Game Design: A Book of Lenses, Second Edition, by Jesse Schell Presents 100+ sets of questions, or different lenses, for viewing a game’s design, encompassing diverse fields such as psychology, architecture, music, film, software engineering, theme park design, mathematics, anthropology, and more. Written by one of the world's top game designers, this book describes the deepest and most fundamental principles of game design, demonstrating how tactics used in board, card, and athletic games also work in video games. It provides practical instruction on creating world-class games that will be played again and again. View it here. A Practical Guide to Indie Game Marketing, by Joel Dreskin Marketing is an essential but too frequently overlooked or minimized component of the release plan for indie games. A Practical Guide to Indie Game Marketing provides you with the tools needed to build visibility and sell your indie games. With special focus on those developers with small budgets and limited staff and resources, this book is packed with tangible recommendations and techniques that you can put to use immediately. As a seasoned professional of the indie game arena, author Joel Dreskin gives you insight into practical, real-world experiences of marketing numerous successful games and also provides stories of the failures. View it here. An Architectural Approach to Level Design This is one of the first books to integrate architectural and spatial design theory with the field of level design. The book presents architectural techniques and theories for level designers to use in their own work. It connects architecture and level design in different ways that address the practical elements of how designers construct space and the experiential elements of how and why humans interact with this space. Throughout the text, readers learn skills for spatial layout, evoking emotion through gamespaces, and creating better levels through architectural theory. View it here. Learn more and download the ebook by clicking here. Did you know? GameDev.net and CRC Press also recently teamed up to bring GDNet+ Members up to a 20% discount on all CRC Press books. Learn more about this and other benefits here.
Sign in to follow this  
Followers 0
hoffman_jwr

How Can I Reduce My Compile Times?

21 posts in this topic

Hi there,
I am a graduate games programmer with around 6 months industry experience, and this dilemma is something that I often ponder. Up to this point, I have worked on a few projects which have grown to become substantial in size (approx. twenty - thirty thousand lines). Although I have strived for software orthogonality (i.e. modularity), and worked towards decoupling the various aspects of the systems I worked on, inevitably the projects bloat, interdependencies pop up, and compile times become long. I work in predominantly in C; I always ensure forward declarations over header includes, I use pointers wherever possible. Is it possible to modularise to the point where a module is completely free-standing? Would it be possible to place the various components of an engine into dll files (something I have never done)? On large scale projects, is this practice? - or does the solution reside in the one project file (somehow I doubt that)? dll's are windows specific; are there any platform unspecific ways to build linkable object files?
1

Share this post


Link to post
Share on other sites
Grouping modules by what they do is a good practice. A lot of engines do that. The base dll defines only interfaces and plugins implement sub-systems. It will definitely reduce dependencies to only 1 dll (the base module). Having said that, a good way to reduce compile times is to use pre-compiled header files. The idea may seem trivial but having a good set of base headers that seldom change in a single header file and compiling that file once will reduce build times significantly. I am writing this from my personal experience.
0

Share this post


Link to post
Share on other sites
It sounds like you already have a pretty good handle on the basics. Really it's a matter of keeping things as decoupled as possible, and minimizing the scope of dependencies. The real killers are things that crop up in, say, 75% of your code base and require vast recompiles every time something subtle changes.

One thing a lot of people overlook is dividing a [i]single[/i] module into multiple header/code files. For instance, you may have a chunk of game logic that affects a lot of the engine; split it up into smaller groups based on whatever criteria makes the most sense, and go from there. This can radically reduce compile times when you only make localized, contained changes. Be careful, though, as on some compilers having too many files can actually end up costing you time due to the spin-up cost of starting compilation separately on every file.

Short of that... throw more hardware at it [img]http://public.gamedev.net/public/style_emoticons/default/wink.gif[/img]


Honestly, though, 20-30 KLOC is nothing. Wait until you're in the millions of lines of code range and a full rebuild takes an hour or two. [i]That's[/i] when it's worth doing major devilry to make your compile times faster.
0

Share this post


Link to post
Share on other sites
For starters, it seems like you've been on the right track. But [i]my [/i]question is going to be: [i]what[/i] compile time is getting too large?

Look into a build solution like [url="http://cruisecontrol.sourceforge.net/"]Cruise Control[/url] to take care of the full build+test cycle for you. That way, the longest build times are hopefully dealt with on a dedicated machine.

Make sure your build system is setup properly. For instance "make" by default requires the "-j" option to be added to get a multi-threaded build. Make sure your dependencies are properly being maintained if you don't have an IDE that generates them for you. Or better yet, spend the time to make sure it is automated.

Make sure your interfaces are being respected. Systems can get too tangled together when someone on the team decides to grab what was an internal header and use it somewhere because it had a feature they needed. In theory, you should be able to seperate collections of subsystems into static libraries and only rebuild them as needed (ie, compile Bullet Physics as a static lib, and add it to the project). Only go with the dll/so runtime loading if you gain something from it.

[quote name='ApochPiQ' timestamp='1294621351' post='4756375']
One thing a lot of people overlook is dividing a [i]single[/i] module into multiple header/code files.
[/quote]

Seconded. If you have a "mathutil.h" floating around, consider breaking it up. Maybe it contains a lot of unrelated systems. You could end up with "interpolate.h", "springs.h", and "range_limits.h" as well, because people just tossed "math" stuff in there when they needed to wrap, clamp, mirror, spring, blend, accelerate_smooth... etc. Go further? have a spring1d.h spring2d.h and spring3d.h if there are enough functions to warrant it.
0

Share this post


Link to post
Share on other sites
Seconded (thirded?) about precompiled headers. A lot of programmers seem oddly sour on PCHes, possibly because their very first C++ experience was MSVC complaining that it couldn't find the #include <stdafx.h>. But I've had it reduce total project compile times by a factor of [i]eighty[/i].
0

Share this post


Link to post
Share on other sites
Like others said, use precompiled Headers. They exist for MSVC and for GCC as far as I know.
They already reduce your compile time by at least 60%.
Just make sure you only add "static" Headers to your include-file which the PCH is built from.
If you include headers there which are changed very often, you save nothing.
But adding there all the boost-, stl-, library-headers you never gonna change, reduces your compile time dramatically.
HTH
regards
0

Share this post


Link to post
Share on other sites
All very good advice, thanks to everyone! A common suggestion seems to be the use of precompiled headers, something that I will look into. When you say precompiled headers, do you mean header files that will not have to be rebuilt after a full clean and rebuild? For the record, we are using gcc, and the full rebuild time on our current project is around 6 minutes. Essentially, what I am working towards is a way to add (unchanging) modules to new projects without having to transfer all of the implementation files; I think this is a static library?

Thanks for not flaming me [img]http://public.gamedev.net/public/style_emoticons/default/wink.gif[/img]
0

Share this post


Link to post
Share on other sites
[quote name='hoffman_jwr' timestamp='1294719969' post='4757057']
All very good advice, thanks to everyone! A common suggestion seems to be the use of precompiled headers, something that I will look into. When you say precompiled headers, do you mean header files that will not have to be rebuilt after a full clean and rebuild? For the record, we are using gcc, and the full rebuild time on our current project is around 6 minutes. Essentially, what I am working towards is a way to add (unchanging) modules to new projects without having to transfer all of the implementation files; I think this is a static library?

Thanks for not flaming me [img]http://public.gamedev.net/public/style_emoticons/default/wink.gif[/img]
[/quote]

Well, yes ofc, if you want to reuse self programmed modules, use static libraries.
Precompiled Headers is a newer technology which allows the Compiler to "precompile" Headerfiles you gonna use.
Basically it makes a static library out of all your headers. While this is technically not entirely true it explains best whats happening.
That reduces compile time too since the Headers do not need to be parsed every time you build if the headers are not changed.

See:
http://msdn.microsoft.com/en-us/library/z0atkd6c%28v=vs.80%29.aspx
http://gcc.gnu.org/onlinedocs/gcc/Precompiled-Headers.html
http://en.wikipedia.org/wiki/Precompiled_header

I would adive you to make a static or dynamic library out of your modules and to use PCH.
A combination is best.
0

Share this post


Link to post
Share on other sites
Awesome advice, have been looking into PCH and the creation of static libraries; exactly what I was looking for. Thanks to all. [img]http://public.gamedev.net/public/style_emoticons/default/laugh.gif[/img]
0

Share this post


Link to post
Share on other sites
Precompiled Headers just let multiple C++ files share a common set of header files so the compiler doesn't have to waste time reading, parsing, and compiling that code over and over and over.

Other approaches to reduce build times are to:

1) thread the build process and have machines with lots of cores do the building (this is all good until linking however)

2) Merge more cpp files together (Unreal's build tool does this, via temp files and #include'ing the cpps)

3) Redesign various subsystems to use a PIMPL design so the dependency on the subsystem can be reduced to an opaque pointer of some sort. This is very similar conceptually to using virtual functions in a class, except instead of making the functions virtual, PIMPL makes the data virtual. As a bonus the functions can be non-virtual if you don't need inheritance so the penalty for doing a data abstraction is quite small. This approach really really helps avoid having to do full rebuilds when changing the implementation, as only interface changes will causes larger rebuilds. You can also think of this approach as 'make a C interface and use that, with a class wrapper helper to avoid polluting the global namespace'. C interfaces for FILE via fopen/fclose are a PIMPL design for instance. In addition making a C interface can help with other features in other systems (lua binding etc)

4) Keep template classes bare bones or simplistic since highly specialized templates (say a file system wrapper or something which could have a lot of platform specific code inside) require pulling in insane amounts of dependies into the header in order to be able to do everything, and this hurts build times a lot.

5) Not including windows.h in files that don't need it can really help but this can be difficult to accomplish at times especially if you need the thread sync primitives (CRITICAL_SECTION, Events etc). Its possible to do a PIMPL design with classes that wrap the sync primitives and allocate them all from a heap or a pool of sync primitives, but you will lose out on some features (difficulty in getting arrays of Events for use with WaitForMultipleObjects for instance).

6) Implement a scripting language, so you don't have to rebuild the C++ code for every little change.

7) Make as much of the codebase data driven as possible, again so you don't have to rebuild the C++ code for every little change.

8) If link times are long you could break the project up into DLLs for development, though inlined things like math libraries are still going to ripple out and rebuild everything. A game engine is far better off as a monolithic app though but that could just be the difference between a shipping build and a developmental one.
0

Share this post


Link to post
Share on other sites
[quote]6) Implement a scripting language, so you don't have to rebuild the C++ code for every little change.[/quote]

I would recommend to use AngelScript for that, forums here on gamedev.net :D. Awesome ScriptingLanguage and easy to integrate.
0

Share this post


Link to post
Share on other sites
Use the PIMPL idiom. You code a public class and a private class. You keep all your private functions in the private class and just show the public/protected in the public. By doing this you memorise the headers in the public class and using also forward declarations will help minimise the header look up.

But also remember PIMPL is only useful for libraries, it's useless in main applications like exe's.
0

Share this post


Link to post
Share on other sites
Another simple and obvious one I used to do to often, including iostream in nearly every file. I actaully rarely use it, mostly I just include string, vector, and map where needed, and I try not to use windows.h with WIN32_MEAN_AND_LEAN.
0

Share this post


Link to post
Share on other sites
more megahurtz.

Not a fan of precompiled headers. Often it is a dump of various headers, and if you are not careful, can lead to poor practises.

But usually, reduce dependencies in code (kurbing on the number of headers, especially the ones involving templates or nasty buggers like windows.h, reducing their dependencies by using forward declarations, ..). Being disciplined about it and not going 'I'll fix that later' helps as well, as often it just creeps unnoticed (until you start slowing down again).

If you have big monolithic classes, that sometimes incurs a profusion of headers, and probably a rethink of what that class is suppose to do. Can't remember the name of the tool, but there are some that helps you track down dependencies and eliminate them. Sometimes it's not possible, and if something seems out of place, then your original design could be wrong in the first place.

Splitting a program into libraries is a way to force you to partition your code into logical units, and reduce dependencies between modules. however doing that after the fact is immensely painful. There is also a lot of time consumed by maintaining the discipline, and it is easy to over-do it.
0

Share this post


Link to post
Share on other sites
You may minimalize parsing headers by creating one or more special cpp files that include all other cpps and defining special macro that prevents parsing them again. Ok, example:

-- project.cpp
#define COMPILE_SPEEDUP
#include "a.cpp"
#include "b.cpp"

--a.cpp
#ifdef COMPILE_SPEEDUP
...
#endif


--b.cpp
#ifdef COMPILE_SPEEDUP
...
#endif

As a result, project.cpp will compile everything and all headers are parsed just once. It certainly helped us.
0

Share this post


Link to post
Share on other sites
fastest builds I've ever seen have been with unity builds.

[url="http://buffered.io/2007/12/10/the-magic-of-unity-builds/"]http://buffered.io/2007/12/10/the-magic-of-unity-builds/[/url]

wickedly fast.
0

Share this post


Link to post
Share on other sites
[quote name='RavNaz' timestamp='1299246923' post='4781734']
fastest builds I've ever seen have been with unity builds.

[url="http://buffered.io/2007/12/10/the-magic-of-unity-builds/"]http://buffered.io/2...f-unity-builds/[/url]

wickedly fast.
[/quote]

If this speeds up the builds, then your bottleneck are includes and include parsing. Precompiled headers would achieve similar effect.

If your bottleneck is linker, the above will have less of an impact.
0

Share this post


Link to post
Share on other sites
[quote name='RavNaz' timestamp='1299246923' post='4781734']
fastest builds I've ever seen have been with unity builds.

[url="http://buffered.io/2007/12/10/the-magic-of-unity-builds/"]http://buffered.io/2...f-unity-builds/[/url]

wickedly fast.
[/quote]

Fast, yes. But it does come with side-effects.
0

Share this post


Link to post
Share on other sites
I've heard about one more method to increase performance of compile / link. In case you have a lot of source / header files and its disk operations that are slowing you down, you could create a RamDisk and copy your project to it. You could compile from the RamDisk in such a case, what would eliminate disk read operations cost. Never tried this myself, but I've heard it can increase compile time quite a lot in some cases.
0

Share this post


Link to post
Share on other sites
I'm gonna throw this in: XCode 4 ;)

Seriously, it's blazing fast. You have to have a portable codebase or be targeting OS X though obviously.
0

Share this post


Link to post
Share on other sites
In Visual Studio 2008 there is an option in Project configuration.
Click (in main menu) [b]Project > Properties[/b] then (in left tree)[b] C/C++ > Code generation[/b] and finally[b] Enable minimal rebuild[/b] to [b]Yes[/b]
Try playing with it, I suppose it is project-dependent but once I got a ~3 minutes save on compile time for C++ project with ~1000 .h and .cpp files.
0

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!


Register a new account

Sign in

Already have an account? Sign in here.


Sign In Now
Sign in to follow this  
Followers 0