Sign in to follow this  

What should be stored in a repository?

This topic is 2390 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Recommended Posts

I'm currently trying to determine what should be kept in the repository for a typical game.

As I understand it, the code and art/music/general game assets should be hosted, but what about the:

[list][*]Project files for the code?[*]Third party libraries?[/list]

Are these things that should be hosted or should team members download/create them themselves locally?

Cheers.

Share this post


Link to post
Share on other sites
[quote name='Mybowlcut' timestamp='1306126891' post='4814458']
I'm currently trying to determine what should be kept in the repository for a typical game.

As I understand it, the code and art/music/general game assets should be hosted, but what about the:

[list][*]Project files for the code?[*]Third party libraries?[/list]

Are these things that should be hosted or should team members download/create them themselves locally?

Cheers.
[/quote]


The way I do it is ALL code files (even third party, unless they have a repository that I can update from then I use it.) and project files are kept. If the third party libraries aren't source code, I include the dlls etc out of the box. IMO a project should be compilable after downloading from a repository.

I then use dropbox.com and set it up as the media folder, so assets can just get stored there and iterations are put up and automatically downloaded to the local media folder.

It's simple it works and is free.

Share this post


Link to post
Share on other sites
[quote name='CRYP7IK' timestamp='1306128048' post='4814460']
The way I do it is ALL code files (even third party, unless they have a repository that I can update from then I use it.) and project files are kept. If the third party libraries aren't source code, I include the dlls etc out of the box. IMO a project should be compilable after downloading from a repository.

I then use dropbox.com and set it up as the media folder, so assets can just get stored there and iterations are put up and automatically downloaded to the local media folder.

It's simple it works and is free.
[/quote]
Sounds good. Not sure what you mean regarding the dropbox thing though. How does that work? I was thinking just plop the assets in the repository also..

Share this post


Link to post
Share on other sites
When someone joins a project, they should be able to download from the repo [u]everything they need to build the game from scratch[/u] ([i]besides software tool installations like Visual Studio[/i]).
[hr]
DropBox is kinda like another type of repository. It's as if he's got his code in one repo and his assets in another repo.

Share this post


Link to post
Share on other sites
[quote name='Hodgman' timestamp='1306130289' post='4814469']
When someone joins a project, they should be able to download from the repo [u]everything they need to build the game from scratch[/u] ([i]besides software tool installations like Visual Studio[/i]).
[hr]
DropBox is kinda like another type of repository. It's as if he's got his code in one repo and his assets in another repo.
[/quote]
Ah, I see.

What about the scenario where a library like SFML comes packaged for VS2005 & VS2008? Can we use different IDEs and just not put the project files under source control?

Share this post


Link to post
Share on other sites
For most professional environments you should have EVERYTHING necessary to recover the system.

If you are only storing the build system itself you can test it easily enough:
[list][*]Start a clean install of the OS.[*]Install the repository client.[*]Sync. (This can be in a separate branch within the system)[*]Disconnect from the network (just to make sure you aren't relying on something remotely)[*]Run the build script with a single button click or single batch file, and no other options.[*]Game should build completely without human intervention.[/list]
Note that this does NOT include things like "install the IDE", or "Install photoshop". All the executables and compilers and libraries and other neccessary items should be directly available within the tree.
There are also strong arguments for including copies of the tools necessary for creating and editing any binary file. For example, your art modeling tools. There is typically very little difference between keeping the install discs on a network location that is archived or in a versioning system that is also archived. This applies for all software, including Photoshop and Visual Studio, all the way to the versioning system tools themselves.

Since these files are not necessary for the build, you can give them a separate install script.

With that in place, you can rebuild any type of machine with:
[list][*]Start a clean install of the OS.[*]Install the repository client.[*]Sync. (The install tools can be in a separate branch so they aren't on all machines all the time.)[*]Disconnect from the network (just to make sure you aren't relying on something remotely)[*]Use a single batch file like InstallEngineer or InstallAritist or InstallBuildMachine.[*]After pressing no other buttons, your machine should churn and run install scripts and soon be ready for use[/list]
If you can't do single-click install or a single-click build, then you have too much room for error when you need to recover your project years from now.

Share this post


Link to post
Share on other sites
Thanks for the reply frob.

Does that mean the developers must be using the same IDE? I take it that you'd consider project files as necessary items in a repository.

Share this post


Link to post
Share on other sites
[quote name='Mybowlcut' timestamp='1306188695' post='4814775']Does that mean the developers must be using the same IDE? I take it that you'd consider project files as necessary items in a repository.[/quote]On every project I've worked on, there has been a standard set of tools. When it comes time to upgrade (e.g. VS2008 -> VS 2010), the whole team upgrades at the same time.


Project files should be in the repo -- unless you are automatically generating your project files with something like CMake, in which case you'd store your cmake project files instead of your generated project files.

Share this post


Link to post
Share on other sites
[quote name='Hodgman' timestamp='1306197498' post='4814826']
[quote name='Mybowlcut' timestamp='1306188695' post='4814775']Does that mean the developers must be using the same IDE? I take it that you'd consider project files as necessary items in a repository.[/quote]On every project I've worked on, there has been a standard set of tools. When it comes time to upgrade (e.g. VS2008 -> VS 2010), the whole team upgrades at the same time.


Project files should be in the repo -- unless you are automatically generating your project files with something like CMake, in which case you'd store your cmake project files instead of your generated project files.
[/quote]
What if the team consists of hobbyists? Some might not be able to afford the agreed upon IDE for example. I'm trying to determine how difficult it will be to have team members working on code in different IDEs.

Share this post


Link to post
Share on other sites
[quote name='Hodgman' timestamp='1306223050' post='4815007']
Personally, I always use visual studio, which is free.

If you can't agree on a common one, then you can use a tool like CMake to automatically generate project files for several different IDEs.
[/quote]
Ah. Good point, I had forgotten about Express.

Share this post


Link to post
Share on other sites
I don't know if I'd go quite as far as Hodgeman suggests -- though I can certainly see what he's getting at, and in a professional environment the additional risk-mitigation delivered by that solution is worth the cost of maintaining it. I've seen quite a number of AAA build systems ranging from "just VS project files" to very complicated messes implemented in no less than 4 distinct languages -- of the dozen or so I've seen across as many game studios, I'd say that its a nearly-even split between those that do include all the tools and those that assume a suitable suite of tools has been installed somewhere on the system.

IMO, what should be in the repository is all the *source* data (code, art assets, configurations, scripts, third-party headers/libraries, etc) that is necessary to build the software. This probably also includes the source data for any necessary in-house tools that are specific to the primary application.


A couple things to consider:
Be wary of unnecessary or user-specific files pouting your repository -- things like Windows' thumbnail files or machine/user-specific config files (such as Visual Studio user preference files) -- most Revision Control Systems let you configure that files ending in certain extensions are to be ignored, which is what I'd recommend.

Following on, its probably best not to check in any intermediate files -- its too easy for an unsuspecting dev to change them, and then wonder what's going on when the intermediate file has been regenerated. A good example of this is not to include makefiles or VS Project files in a repo if you are using CMake. In fact, disallow those file types from the repository as described above. This can be seen as an extension of the "DRY" principle (Don't Repeat Yourself) -- its good for databases, its good for code, and it turns out that its good for repositories too (which are basically databases of code).

An argument can be made for keeping the code/build and art repositories as separate entities. Artists and programmers tend to work quite differently, so it should be no surprise that they may not be served well by a single tool -- imagine teaching your art team how to use GIT, or shackling your code team to some fancy GUI-based VCS. Also many of the typical text-focused VCSs don't deal well with binary files like sounds or images, or don't provide/integrate with additional tooling (such as visual diff programs). Another argument, is that art seems to be holding, at least for now, to the old checkout-modify-checkin model, while many coders now prefer the more-modern, decentralized workflows made possible by DVCSs such as Git, Mercurial, Bazaar, Fossil and others. Using this separated setup, you would have a 3-part build -- one part builds the tools and executables, another builds the finalized art assets from the source assets ("cooks" them), and another sucks up all those files to create the disc-image or installer package.

Share this post


Link to post
Share on other sites
[quote name='simpler' timestamp='1306311262' post='4815486']Should the repository include the boost library if it's used?[/quote]I'd normally say yes, because I like to have all the project dependencies in there, so you can just check out the repo and build on any PC.

...but boost is so insanely huge that depending on your version control software, it can completely slow everything down...
Solutions to this that I've seen are:
1) Only check in the parts of boost that you're actually using.
2) Check in a ZIP file containing boost, and tell people to unzip it after they've checked out the repo.

Share this post


Link to post
Share on other sites
[quote name='Hodgman' timestamp='1306312559' post='4815494']
[quote name='simpler' timestamp='1306311262' post='4815486']Should the repository include the boost library if it's used?[/quote]I'd normally say yes, because I like to have all the project dependencies in there, so you can just check out the repo and build on any PC.

...but boost is so insanely huge that depending on your version control software, it can completely slow everything down...
Solutions to this that I've seen are:
1) Only check in the parts of boost that you're actually using.
2) Check in a ZIP file containing boost, and tell people to unzip it after they've checked out the repo.
[/quote]

Ok, thanks. Do you believe that you in the future can take for grant that people downloading your repo has boost on their computer? Ofcourse this depends on context but will boost become such a standard library?

Share this post


Link to post
Share on other sites
[font="arial, verdana, tahoma, sans-serif"][size="2"][quote name='Ravyne' timestamp='1306228931' post='4815039']
I don't know if I'd go quite as far as Hodgeman suggests -- though I can certainly see what he's getting at, and in a professional environment the additional risk-mitigation delivered by that solution is worth the cost of maintaining it. I've seen quite a number of AAA build systems ranging from "just VS project files" to very complicated messes implemented in no less than 4 distinct languages -- of the dozen or so I've seen across as many game studios, I'd say that its a nearly-even split between those that do include all the tools and those that assume a suitable suite of tools has been installed somewhere on the system.

IMO, what should be in the repository is all the *source* data (code, art assets, configurations, scripts, third-party headers/libraries, etc) that is necessary to build the software. This probably also includes the source data for any necessary in-house tools that are specific to the primary application.


A couple things to consider:
Be wary of unnecessary or user-specific files pouting your repository -- things like Windows' thumbnail files or machine/user-specific config files (such as Visual Studio user preference files) -- most Revision Control Systems let you configure that files ending in certain extensions are to be ignored, which is what I'd recommend.

Following on, its probably best not to check in any intermediate files -- its too easy for an unsuspecting dev to change them, and then wonder what's going on when the intermediate file has been regenerated. A good example of this is not to include makefiles or VS Project files in a repo if you are using CMake. In fact, disallow those file types from the repository as described above. This can be seen as an extension of the "DRY" principle (Don't Repeat Yourself) -- its good for databases, its good for code, and it turns out that its good for repositories too (which are basically databases of code).

An argument can be made for keeping the code/build and art repositories as separate entities. Artists and programmers tend to work quite differently, so it should be no surprise that they may not be served well by a single tool -- imagine teaching your art team how to use GIT, or shackling your code team to some fancy GUI-based VCS. Also many of the typical text-focused VCSs don't deal well with binary files like sounds or images, or don't provide/integrate with additional tooling (such as visual diff programs). Another argument, is that art seems to be holding, at least for now, to the old checkout-modify-checkin model, while many coders now prefer the more-modern, decentralized workflows made possible by DVCSs such as Git, Mercurial, Bazaar, Fossil and others. Using this separated setup, you would have a 3-part build -- one part builds the tools and executables, another builds the finalized art assets from the source assets ("cooks" them), and another sucks up all those files to create the disc-image or installer package.
[/quote]
A great post, thanks Ravyne. I'd like to ask a few questions:[/size][/font]
[font="arial, verdana, tahoma, sans-serif"] [/font]
[font="arial, verdana, tahoma, sans-serif"] [/font][font="arial, verdana, tahoma, sans-serif"][size="2"]I have a root folder "E:\Dev\Projects\SSZS". This folder will store the entire project. It has two folders inside it, "project" (the Visual Studio project) and "repository" (the project hosted on SVN). This seems like a nice structure because you don't mess up IDE-specific stuff with the repository stuff. The only problem is that if I run the code below, it can't find the font file because I'm running the application from within a separate folder to the repository folder:[/size][/font]
[font="arial, verdana, tahoma, sans-serif"] [/font]
[code]MyFont.LoadFromFile("gui\\fonts\\arial.ttf", 50);[/code]

What is the way around this? Do you keep your IDE project in with the repository files?
Also, is it possible to limit what file extensions the repository (e.g. Google Code) will take? Or will each user have to set it up in their SVN client?
[quote name='Hodgman' timestamp='1306312559' post='4815494']
[quote name='simpler' timestamp='1306311262' post='4815486']Should the repository include the boost library if it's used?[/quote]I'd normally say yes, because I like to have all the project dependencies in there, so you can just check out the repo and build on any PC.

...but boost is so insanely huge that depending on your version control software, it can completely slow everything down...
Solutions to this that I've seen are:
1) Only check in the parts of boost that you're actually using.
2) Check in a ZIP file containing boost, and tell people to unzip it after they've checked out the repo.
[/quote]
I've always wondered about only using parts of boost. Is it advisable in the context of wanting to save space? Is it easy to do?

Share this post


Link to post
Share on other sites
[quote name='Hodgman' timestamp='1306312559' post='4815494']
[quote name='simpler' timestamp='1306311262' post='4815486']Should the repository include the boost library if it's used?[/quote]I'd normally say yes, because I like to have all the project dependencies in there, so you can just check out the repo and build on any PC.

...but boost is so insanely huge that depending on your version control software, it can completely slow everything down...
Solutions to this that I've seen are:
1) Only check in the parts of boost that you're actually using.
2) Check in a ZIP file containing boost, and tell people to unzip it after they've checked out the repo.
[/quote]


This does not need to be so.

You can have multiple repositories, or multiple depots, or multiple sub-projects within a single repository.

I recommend that you have a tree that is separate from your main project. In the professional world I've seen it called "common", "external", "CM", and a few other names. This way you can still have the stuff and still recover, but it doesn't clutter your daily use machine.


You need to understand your own purpose for your version control system.

If your purpose is to provide a history and safety net for your own projects as you travel between home and school and other sites, then you probably don't need much more than just your source code. That will let you roll back minor bugs, but is otherwise of little value.

If your purpose is to collaborate with a large number of people then you need to share everything. You can't assume they'll have the same version of boost and other libraries, or the same version of compilers, or the same version of maya exporters, so all those items needs to be there.

If your purpose is so that you can re-create any build from any time in history by simply knowing a build number, then you absolutely need to store your tools, external programs, and external libraries. You will even want to store patches and service releases as you incorporate them to the game. You need to store all the compilers, since a compiler update can modify how builds were generated. You need to store the matching versions of photoshop and maya and other data-centric programs, since differences between versions may introduce small differences in the files, in turn causing the build to generate different results.


While it is true that these are big items, they change infrequently. For any substantial versioning system you are already going to need a rather nice st of machines; the added storage cost for a few extra gigabytes is trivial in comparison to the other costs.

Share this post


Link to post
Share on other sites
Yeah in the case where we stored a zipped copy of boost, it was in a separate "thirdparty" repo. The problem was that simply checking out this repo for the first time, or updating the repo when someone added a new 3rd-party library, became ridiculously slow with the un-zipped boost present.


Yeah, it's a pretty occasional hiccup due to it being in another infrequently changing repo, but these infrequent waits add up with large team sizes. The reason we went over to storing a zipped version of boost in that case, was that the time saved per "thirdparty" update multiplied by the team size was over a full day of lost man-hours. They didn't like the idea of spending hundreds of dollars just to update our (free) dependencies.

Share this post


Link to post
Share on other sites
You only want to *avoid* storing such things if they are being generated from a tool such as CMake, due to the reasons I mentioned above. Namely, if you store the project files and someone changes them, then someone else regenerates the project file from CMake, it will overwrite those changes. Instead, you want to encourage the first guy to make the necessary changes in the CMake configuration, and a good way to do that is to make sure that the VSProject files are never checked in. Every time the CMake config is updated, they have to get the latest version and re-run CMake to generate the new VS file. This also allows them to muck with their own project files without worry of breaking everyone else -- this lets them test out new configs locally before committing them to the CMake config for all to use.


If your project *only* targets visual studio as its development environment, then by all means go ahead and store the project and solution files -- the ones you have to watch out for are the machine or user-specific files -- things such as IDE customization and that sort of thing. Another caveat is that if you store the solution, and the solution names a dependency on lets say, Boost, which is referenced by the full path "C:/libs/boost/1_49" or something, then *everyone* that builds must also install boost at that exact path (they can't have it somewhere else, otherwise they'll need to change the path in the project file for theirs, and when they check in they break everyone else's build.

On way around this is to come up with a standard directory structure for these sorts of dependencies (3rd party libs, SDKs, tools, etc), and create on each machine environment variables for each relevant section -- On my machines, I define the environment variables DEVBIN for tool binaries, DEVSDK for SDKs, DEVLIB for libraries, and DEVSRC where I store all my projects. Of course, the structure of each has to be identical across machines, but at least this way I can move them anywhere on my hard disk -- for example, on my desktop I keep this stuff on my raided drive E, but on my laptop I don't have a secondary drive or partition, so its all on drive C.

This is a good middle ground between full user-centric customization and enforcing a strict machine format.

Share this post


Link to post
Share on other sites
[quote name='Ravyne' timestamp='1306538558' post='4816607']
You only want to *avoid* storing such things if they are being generated from a tool such as CMake, due to the reasons I mentioned above. Namely, if you store the project files and someone changes them, then someone else regenerates the project file from CMake, it will overwrite those changes. Instead, you want to encourage the first guy to make the necessary changes in the CMake configuration, and a good way to do that is to make sure that the VSProject files are never checked in. Every time the CMake config is updated, they have to get the latest version and re-run CMake to generate the new VS file. This also allows them to muck with their own project files without worry of breaking everyone else -- this lets them test out new configs locally before committing them to the CMake config for all to use.


If your project *only* targets visual studio as its development environment, then by all means go ahead and store the project and solution files -- the ones you have to watch out for are the machine or user-specific files -- things such as IDE customization and that sort of thing. Another caveat is that if you store the solution, and the solution names a dependency on lets say, Boost, which is referenced by the full path "C:/libs/boost/1_49" or something, then *everyone* that builds must also install boost at that exact path (they can't have it somewhere else, otherwise they'll need to change the path in the project file for theirs, and when they check in they break everyone else's build.

On way around this is to come up with a standard directory structure for these sorts of dependencies (3rd party libs, SDKs, tools, etc), and create on each machine environment variables for each relevant section -- On my machines, I define the environment variables DEVBIN for tool binaries, DEVSDK for SDKs, DEVLIB for libraries, and DEVSRC where I store all my projects. Of course, the structure of each has to be identical across machines, but at least this way I can move them anywhere on my hard disk -- for example, on my desktop I keep this stuff on my raided drive E, but on my laptop I don't have a secondary drive or partition, so its all on drive C.

This is a good middle ground between full user-centric customization and enforcing a strict machine format.
[/quote]
Thanks for the reply Ravyne. I think you've misunderstood my question though. My issue was where to create the project files on my PC - in with the repository folders or in a separate folder:


[color=#1C2837][size=2][font=arial, verdana, tahoma, sans-serif][size=2]I have a root folder "E:\Dev\Projects\SSZS". This folder will store the entire project. It has two folders inside it, "project" (the Visual Studio project) and "repository" (the project hosted on SVN). This seems like a nice structure because you don't mess up IDE-specific stuff with the repository stuff. The only problem is that if I run the code below, it can't find the font file because I'm running the application from within a separate folder to the repository folder:[/size][/font]
[font=arial, verdana, tahoma, sans-serif][/font]
[color=#660066]MyFont[/color][color=#666600].[/color][color=#660066]LoadFromFile[/color][color=#666600]([/color][color=#008800]"gui\\fonts\\arial.ttf"[/color][color=#666600],[/color][color=#000000] [/color][color=#006666]50[/color][color=#666600]);[/color]

What is the way around this? Do you keep your IDE project in with the repository files?[/size][/color]

Share this post


Link to post
Share on other sites
The "working directory" of your executable is configurable. When you double-click an exe in windows, it will be the exe's directory. When you double-click a shortcut, it will be the directory specified by the shortcut, and when you run an EXE from visual-studio, it will be the directory specified in the [font="Courier New"]project properties -> debugging[/font] dialog.

In most projects, you usually have to go into that directory and manually choose a different working directory, because visual studio's default (the project file's directory) is usually wrong.
However, it's also quite common for games to have a command-line option that specifies where the data directory is, e.g. you could run your game with [font="Courier New"]myGame.exe -data d:\mygame\bin\data\[/font], and then when your code executes [font="Courier New"]LoadFromFile("gui\\fonts\\arial.ttf")[/font], the game will turn it into "[font="Courier New"]d:\mygame\bin\data\ui\fonts\arial.ttf[/font]", and then file-loading will still work even if the working directory is wrong.

Why don't you want to store the project file inside the repository though? What makes it not a "repository file"? If someone else downloads the repo, won't they need the project files in order to build the project?

Share this post


Link to post
Share on other sites
[quote name='Hodgman' timestamp='1306573366' post='4816720']
The "working directory" of your executable is configurable. When you double-click an exe in windows, it will be the exe's directory. When you double-click a shortcut, it will be the directory specified by the shortcut, and when you run an EXE from visual-studio, it will be the directory specified in the [font="Courier New"]project properties -> debugging[/font] dialog.

In most projects, you usually have to go into that directory and manually choose a different working directory, because visual studio's default (the project file's directory) is usually wrong.
However, it's also quite common for games to have a command-line option that specifies where the data directory is, e.g. you could run your game with [font="Courier New"]myGame.exe -data d:\mygame\bin\data\[/font], and then when your code executes [font="Courier New"]LoadFromFile("gui\\fonts\\arial.ttf")[/font], the game will turn it into "[font="Courier New"]d:\mygame\bin\data\ui\fonts\arial.ttf[/font]", and then file-loading will still work even if the working directory is wrong.[/quote]
Crazy. I've never used that in all the years I've been programming... it sounds like it will solve that problem. :) Thanks!

[quote name='Hodgman' timestamp='1306573366' post='4816720']
Why don't you want to store the project file inside the repository though? What makes it not a "repository file"? If someone else downloads the repo, won't they need the project files in order to build the project?
[/quote]

I'm still very new to the whole repository thing, as well as CMake (I don't think it's fair to restrict what IDE is used, especially as we have at least on person on a Linux system), otherwise I'd be storing project files in the repository. I'll need to learn how to use CMake to create the CMake lists or whatever in order to generate project files for each IDE... most people have more trouble getting third party libraries set up than getting their IDE project files set up, so I'm not particularly worried about it at the moment.

Share this post


Link to post
Share on other sites

This topic is 2390 days old which is more than the 365 day threshold we allow for new replies. Please post a new topic.

If you intended to correct an error in the post then please contact us.

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

Sign in to follow this