Sign in to follow this  

Speed up Compile Times

This topic is 3501 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

Hello, I'm not sure if this is the right forum to post this, but I have a question regarding the compile time of my project. I'm using Visual Studio 2005 and I have the common settings turned on to speed up compile time (Incremental Building, etc). But as my project grows, its taking much longer to compile it and its becoming a burden. The project itself is about 260 Files, ranging from my own math/collision libraries, DX and OpenGL interfaces, and utilities. I was wondering, if anyone knows some neat tips or tricks to speed up compile time (anything short of using pre-compiled headers). Thanks for any suggestions.

Share this post


Link to post
Share on other sites
reduce dependencies, use interfaces, forward-declare in headers instead of include, use "pImpl" technique, avoid "master" headers... separate code into libraries so you only have to compile the libs you change... if you have lots of computers sitting around there's always incredibuild, or you can go multi-core on your personal machine.

why not use pre-compiled headers?

Share this post


Link to post
Share on other sites
If you're doing incremental build then you shouldn't build all of your code each time, unless you're doing rebuild all. Unless all your code are highly dependable on each other that one change would cause all to rebuild, which I highly doubted. I would use precompiled header, minimize dependency, break code into libs if possible. Other than that you can look at JAM.

Share this post


Link to post
Share on other sites
If you're using a multicore machine and coding in C++, you could always upgrade to MSVC 2008 and add the /Mp switch to the compiler. It parallelizes the compilation process. The difference in speed is significant.

Share this post


Link to post
Share on other sites
Another little tip that I've found useful is that you can compile just a single .cpp file without changing anything it affects by finding the file in the file browser (usually to the left), right-clicking on it, and selecting "Compile". It's great for simply checking syntax and that sort, where you don't plan on actually testing the changes, just making sure it compiles.

Share this post


Link to post
Share on other sites
The /MP switch also works for MSVC 2005. It is an 'undocumented' feature though, so it has some rough edges and does not work in all cases.

Share this post


Link to post
Share on other sites
Quote:
Original post by emeyex
reduce dependencies, [...] use "pImpl" technique.


I'm in the process of adding this in, thanks for the suggestion.

Quote:
Original post by emeyex
separate code into libraries so you only have to compile the libs you change...


I guess I can pull some Math code out into a Math library, but in its defense, its mostly inlined functions which are in the header itself. Would this be efficient?

Quote:
Original post by emeyex
why not use pre-compiled headers?


Its a lot of work IMO, I was going to leave it for headers that I no longer need to work on. Which might be a long ways away.

Quote:
Original post by Hnefi
you could always upgrade to MSVC 2008 and add the /Mp switch to the compiler. It parallelizes the compilation process. The difference in speed is significant.


I'll see about getting MSVC 2008 form my school. Does it matter which version you get (Express, or Professional, or anything in between?)

Quote:
Original post by Icon
The /MP switch also works for MSVC 2005. It is an 'undocumented' feature though, so it has some rough edges and does not work in all cases.


I tried this but it said the switch is unknown. Maybe I need to update my compiler (service pack) or just switch to MSVC2008.

Share this post


Link to post
Share on other sites
Quote:
Original post by RealMarkP
Quote:
Original post by emeyex
separate code into libraries so you only have to compile the libs you change...


I guess I can pull some Math code out into a Math library, but in its defense, its mostly inlined functions which are in the header itself. Would this be efficient?


What is that 'defending'? Extra code in headers means slower compile times.

Personally I think you'll find a lot can be done with simply reducing the size of your commonly-used headers, and culling dependencies.

Share this post


Link to post
Share on other sites
Quote:
Original post by RealMarkP
Quote:
Original post by emeyex
why not use pre-compiled headers?


Its a lot of work IMO, I was going to leave it for headers that I no longer need to work on. Which might be a long ways away.

Bzzt. It's not a lot of work, and it will be much more effective than anything else you're trying.

Share this post


Link to post
Share on other sites
Quote:
I'll see about getting MSVC 2008 form my school. Does it matter which version you get (Express, or Professional, or anything in between?)

No, even the express version can use the /MP switch, but you always hav to add it yourself (it's off by default).


Quote:
Its a lot of work IMO, I was going to leave it for headers that I no longer need to work on. Which might be a long ways away..

Really consider using precompiled headers, the impact will be huge, trust us.
- make a general header, say you call it stdafx.h for example, and #include all API and libraries here
- make a translation unit dedicated for the compilation of the general header. Call it stdafx.cpp and add only one line in it : #include "stdafx.h"
- in your general project properties, select "Use precompiled header" in the "precompiled headers" tab
- right click on stdafx.cpp, properties, and select "Create precompiled header" in the "precompiled headers" tab
- in each .cpp file : remove all API and libraries includes, and #include "stdafx.h" at the top of the file


[Edited by - rolkA on May 15, 2008 10:57:50 AM]

Share this post


Link to post
Share on other sites
- reducing dependencies
- using forward declaration instead of inclusion of headers

btw I hate pImpl. It just makes me angry everytime I forget to add changes twice.
I only use it to hide details in classes which are open to the public.

Krazy (part of the KDE project) checks if the QT-headers you include are even needed. Would be nice to have something like that in general.

Quote:
Original post by emeyex
separate code into libraries so you only have to compile the libs you change


Don't do extra libs until you really want to make something plug-able or it is really something generic, which you will use in other projects.
It just makes things more complex.

Share this post


Link to post
Share on other sites
Quote:
Original post by RealMarkP
Quote:
Original post by emeyex
why not use pre-compiled headers?


Its a lot of work IMO, I was going to leave it for headers that I no longer need to work on. Which might be a long ways away.


Just precompile the standard c and c++ headers.

Those won't change a lot, obviously, but the compile time speed up can be huge.
Especially when you use a lot STL stuff.

Share this post


Link to post
Share on other sites
Only pre-compile headers that aren't likely to be changed by you. Anything more and the burden becomes more than the benefit.

That being said, I don't always find pre-compiled headers to be a huge win (They do help when used carefully). If you have a lot of small, modular projects, the benefits of pre-compiled headers reduces significantly, since there's no way (that I know of) to tell multiple projects to use a single precompiled header.

Share this post


Link to post
Share on other sites
Quote:
Original post by Rydinare
Only pre-compile headers that aren't likely to be changed by you. Anything more and the burden becomes more than the benefit.

That being said, I don't always find pre-compiled headers to be a huge win (They do help when used carefully). If you have a lot of small, modular projects, the benefits of pre-compiled headers reduces significantly, since there's no way (that I know of) to tell multiple projects to use a single precompiled header.

If you have multiple projects, then you include all the dependencies' headers in your StdAfx.h file. A lot of small, modular projects is the ideal case for precompiled headers.

Share this post


Link to post
Share on other sites
Not to derail but.. /MP cannot be used at the same time as /Gm (enable incremental build).

In which cases is one better than the other? As for release mode, Gm was already turned off for me, but I don't notice a difference in debug mode.. I post numbers when I try them both.

EDIT: Dropped my 3.5 min compile time to 2.5 min - so MP is an improvement but still not sure if it is faster than incremental builds).

[Edited by - aclysma on May 15, 2008 1:51:21 PM]

Share this post


Link to post
Share on other sites
Quote:
Original post by Sneftel
Quote:
Original post by Rydinare
Only pre-compile headers that aren't likely to be changed by you. Anything more and the burden becomes more than the benefit.

That being said, I don't always find pre-compiled headers to be a huge win (They do help when used carefully). If you have a lot of small, modular projects, the benefits of pre-compiled headers reduces significantly, since there's no way (that I know of) to tell multiple projects to use a single precompiled header.

If you have multiple projects, then you include all the dependencies' headers in your StdAfx.h file. A lot of small, modular projects is the ideal case for precompiled headers.


Hmmm. Do you want to explain further? Here's my problem with precompiled headers. Let's say I make a change to X's header and it's common to projects A, B and C. I don't want to have to rebuild the entirity of A, B and C if only a single file from each of those was using that dependency. That's where I've found precompiled headers to have issues, in cases where they force everything to rebuild.

Mind you, some of my comments are tailored towards a large project I work on (2 million LOC).

Share this post


Link to post
Share on other sites
Quote:
Original post by Driv3MeFar
Take a look at this article.

We use something similar to the single compilation unit idea at work, and it cut about 30 minutes off our build times.


Hmmm, but then that means every change = full rebuild. I like the premise and I could see it reducing our build time by 75% or more, but I don't think it's practical for large projects, due to the lack of modularity.

Share this post


Link to post
Share on other sites
Quote:
Original post by Rydinare
Hmmm. Do you want to explain further? Here's my problem with precompiled headers. Let's say I make a change to X's header and it's common to projects A, B and C. I don't want to have to rebuild the entirity of A, B and C if only a single file from each of those was using that dependency. That's where I've found precompiled headers to have issues, in cases where they force everything to rebuild.

If X is actually common to A, B, and C -- that is, it's needed in most of A's files, most of B's files, and most of C's files -- then there's no way around a large rebuild. That's the nature of changing headers that a lot of projects include. In a situation where things are widely separated, though, an individual non-system header will probably not show up in more than a few of your many, many small projects, so the effects will be limited. Compare to HugeApplicationB which depends on HugeLibraryA, where a change to HugeLibraryA's ToasterControl.h, included in HugeApplicationB's stdafx.h, forces an entire rebuild of HugeApplicationB (even the non-toaster-related parts).

Share this post


Link to post
Share on other sites
Quote:
Original post by Rydinare
Hmmm, but then that means every change = full rebuild. I like the premise and I could see it reducing our build time by 75% or more, but I don't think it's practical for large projects, due to the lack of modularity.

The "Everything in a big source file" approach works best when you can separate concerns a little bit. I'd hate putting a few million LoC into one compilation unit, but separating it into 10-20, each one representing a vaguely but pragmatically defined "area of functionality" works well. You'll find that a 100,000 line source file takes nowhere near 100 times as much time to compile as a 1,000 line file, especially if it doesn't include many other headers.

Share this post


Link to post
Share on other sites
You might want to use it for full builds anyways.

Does VS loose the ability to compile in parallel or not - my guess is: yes.
(Because parallelism means powering up multiple instances of the compiler one for each cpp-file, right?)

Share this post


Link to post
Share on other sites
I have a small project that used to take 70 seconds to compile. Not a lot, but a pain when it does. I used to have one monolithic header file for easier re-distribution.

I have since divided everything up into seperate header files and used forward declarations in all of my headers. That cut my compilation time to about 35 seconds.

I removed the /Gm (Incremental Build) and added the /MP (Multiprocessing) switch. My full builds went down to 16 seconds.

This is on a Pentium D (argg) Dual Core CPU. My Q6600 had even better performance.

I personally do not like pre-compiled headers, personal preference. But, most of my savings came from forward declaration and /MP switch.

P.S. I would remove any implementation from the header files. Compilers are pretty smart to inline simple functions without being asked to (Release Builds).

Share this post


Link to post
Share on other sites
What constitutes a long build time for you? 2 minutes? 260 files doesn't really seem like a whole lot to me. Where I'm currently working we have over a 142 visual studio projects with about 500,000+ lines of code that gets executed. On my work machine it takes 8 minutes to build. For those that are gonna ask machine specs. Quad Core Xeon 2.66, 4GB RAM, 73GB 15k SAS drives. Kinda beefy... Anyway, our nightly build process takes quite a bit longer due to building over the network but even then 30 min build times for 500k lines isn't bad. We aren't using any special build tools, products, or precompiled headers here.

I've also got a 50k+ line hobby project I'm working on at home on my MBP that compiles it in a couple of minutes or so. I'm not using any special flags, pre-compiled headers, etc...

Anyway...just curious if we have some pre-optimization going on here. Though the post was just asking about some helpful tips. Like I said...I'm just curious as to how long your builds are taking.

Share this post


Link to post
Share on other sites
Quote:
Original post by ArmitageIII87
What constitutes a long build time for you? 2 minutes? 260 files doesn't really seem like a whole lot to me. Where I'm currently working we have over a 142 visual studio projects with about 500,000+ lines of code that gets executed. On my work machine it takes 8 minutes to build. For those that are gonna ask machine specs. Quad Core Xeon 2.66, 4GB RAM, 73GB 15k SAS drives. Kinda beefy... Anyway, our nightly build process takes quite a bit longer due to building over the network but even then 30 min build times for 500k lines isn't bad. We aren't using any special build tools, products, or precompiled headers here.

I've also got a 50k+ line hobby project I'm working on at home on my MBP that compiles it in a couple of minutes or so. I'm not using any special flags, pre-compiled headers, etc...

Anyway...just curious if we have some pre-optimization going on here. Though the post was just asking about some helpful tips. Like I said...I'm just curious as to how long your builds are taking.


You have a sick machine. Our 2 million LOC builds take over 2 hours.

Share this post


Link to post
Share on other sites

This topic is 3501 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this