Jump to content
  • Advertisement
linus_e

Open Source Gitlab is getting too slow - any ideas?

Recommended Posts

I am really annoyed at how slow GitLab became in my last project, looking for something more scalable for the next one. Using LFS is not an option unfortunately. Does anyone have any recommendations for working with large repos on other tools (Bitbucket, CodeCommit, ...)?

Share this post


Link to post
Share on other sites
Advertisement

If you added large binary files (e.g. game assets like models, textures, sounds etc.) to your repository, all VCSs (like git, Mercurial, CVS, Subversion) will become slow, because they were made for source code, i.e. text files.

These systems normally just store the difference to the previous state, which keeps the repositories small. But this only works with text files. When you change a binary file the whole file is stored, which bloats the repository and makes it slow.

I didn't solve this problem yet, I just don't add binary assets to the source repository at all. I have a Seafile instance running, which synchronizes my assets folder with a server.

Edited by trill41
Typo

Share this post


Link to post
Share on other sites
13 hours ago, trill41 said:

If you added large binary files (e.g. game assets like models, textures, sounds etc.) to your repository, all VCSs (like git, Mercurial, CVS, Subversion) will become slow, because they were made for source code, i.e. text files.

Most of them deal with large binary files pretty well... except for the DVCS systems like Git. Git doesn't cope with large files because every client stores the entire history, and because LFS (the hack to fix that issue) is build around an irreparably slow architecture on Windows.

My company has a single SVN repo containing every asset (mostly PNG and DAE files) and every built EXE file since 2012 -- about 20GB when the current version checked out on a client computer (plus 60GB for the .svn folder, WTF?!), but amazingly disk usage on the server is only about 26GB to store the full history because it apparently does manage to compress diffs of these files pretty well :o 

In my experience in the industry, Git, SVN and Perforce at the most common that I've seen.

@linus_e are you just looking for a Git alternative, or that plus a service provider that hosts repositories for you on the internet?

Share this post


Link to post
Share on other sites
On 9/8/2019 at 11:55 AM, trill41 said:

If you added large binary files (e.g. game assets like models, textures, sounds etc.) to your repository, all VCSs (like git, Mercurial, CVS, Subversion) will become slow, because they were made for source code, i.e. text files.

This is completely untrue for centralized VCSs like CVS, Subversion, or Perforce and only applies to DVCSs like git.

On 9/8/2019 at 11:55 AM, trill41 said:

These systems normally just store the difference to the previous state, which keeps the repositories small. But this only works with text files. When you change a binary file the whole file is stored, which bloats the repository and makes it slow.

This is also untrue of most VCSs. Most VCSs (including git) store snapshots, not differences. While these are stored efficiently using compression and referencing unchanged files (svn) or blocks of content (git), they are still snapshots. The only exceptions I know are darcs and pijul.

The main problem with binary file diffs for size reduction (semantic diffs are a separate issue) and compression isn't even that they are binary, but that they are often already compressed (jpgs and pngs for example), so a small change might change almost everything.

Edited by l0calh05t

Share this post


Link to post
Share on other sites

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now

  • Advertisement
×

Important Information

By using GameDev.net, you agree to our community Guidelines, Terms of Use, and Privacy Policy.

GameDev.net is your game development community. Create an account for your GameDev Portfolio and participate in the largest developer community in the games industry.

Sign me up!