Dependency Management in a Cross Platform, Multi Developer Environment

Started by
5 comments, last by adt7 7 years, 10 months ago

I've been working on a C++ project using CMake as my build-system generator and I'm about to get a second developer on board.

I work on OSX with dependencies installed using Homebrew.

They are going to work on Windows (they are going to work on some of the cross-platform issues, so this choice is deliberate).

What I'm struggling with, is how to manage the dependencies in the project so that it's as easy as possible to onboard this new developer and any future ones (agnostic of platform).

The approaches I've considered are as follows:

  • Make developers install all dependencies using brew/apt/downloading them and letting my FindX.cmake scripts sort it out (with special environment variables for Windows).
  • Include pre-compiled dependencies in a folder within the project and link to those (for all supported platforms).
  • Use git submodules to include the source of the dependencies in the project and use add_subfolder in CMake to compile them.

I think the second approach is the "right" one, as it's the least friction for new developers and means that the dependencies don't have to be compiled the first time.

The only concern I have about this is that at some point (before I'm ready to distribute the project) I want to switch to static linking (is this even a good idea?). This is fine for simple dependencies with no dependencies themselves.

However, what about the more complex ones like SDL? Do I then need to include static versions of all it's dependencies (OpenGL etc.), that doesn't seem great. I noticed SFML does this (in the /extlibs folder), but it doesn't include everything is depends on, only a few (for example OpenGL is not included), how do I know which ones I need to include and which I don't?

Advertisement

Including compiled libraries in a source repo has always lead to trouble somewhere down the road. Maybe you need different versions for similar platforms that have slightly different dependencies? (thinking of different Linux distros here).

I think the most practical way is to let developers install "well known" libraries from their OS's package manager and put less well known things as a submodule and have them compile along your own source.

These days I use the git submodule approach, with CMake wired up to build all the submodules too. It's a bit of a pain to setup, but it honestly works extremely well, provided you don't have source code licensing issues, or dependencies that need to be distributed as binaries...

On previous projects I have maintained pre-compiled libraries for each platform on a central server, and have each developer rsync them to their devbox on a nightly basis. Its not pretty, but it is very flexible and easy.

Ultimately, I feel like the answer may just be to use a better build system with dependency management built in. Java has Gradle, Rust has Cargo... for some reason C++ still limps along without anything.

Tristam MacDonald. Ex-BigTech Software Engineer. Future farmer. [https://trist.am]

Every actual job that I've had has used option #2. If using Git, sometimes these dependencies are in a separate repo (e.g. Project and ProjectDeps which are checked out / cloned to side-by-side locations by each developer) to avoid bloating your main repo with large binary files, seeing that git sucks at that...

The open source / nix world tends to use #1, and some kind of package manager.

Including compiled libraries in a source repo has always lead to trouble somewhere down the road. Maybe you need different versions for similar platforms that have slightly different dependencies? (thinking of different Linux distros here).

I think the most practical way is to let developers install "well known" libraries from their OS's package manager and put less well known things as a submodule and have them compile along your own source.

That maybe true, but as I'm only aiming to support a very limited set of platforms to begin with (not myriad Linux distros) I'm less concerned about this. My aim is to have the process of on-boarding new people as easy as possible, but also to have everyone working from a "known good" state, to rule our library version issues, etc.

These days I use the git submodule approach, with CMake wired up to build all the submodules too. It's a bit of a pain to setup, but it honestly works extremely well, provided you don't have source code licensing issues, or dependencies that need to be distributed as binaries...

On previous projects I have maintained pre-compiled libraries for each platform on a central server, and have each developer rsync them to their devbox on a nightly basis. Its not pretty, but it is very flexible and easy.

Ultimately, I feel like the answer may just be to use a better build system with dependency management built in. Java has Gradle, Rust has Cargo... for some reason C++ still limps along without anything.

I like the sound of the ultimate solution, and using these types of systems is what I've done in the past... if only one existed for C++.

As for the submodule approach, how do you handle dependencies that don't have their own CMakeLists.txt (to pick a well-known example, Lua), do you find a fork with a CMakeLists.txt (e.g. LuaDist/lua) or is there a way I can get CMake to figure out how to use their makefile (even as I'm typing this it sounds insane)?

EDIT: Further to that, how do you handle all of the set calls that need to be made to turn off features in your dependencies builds (e.g. building tests), doing it in the CMakeLists.txt is fine, but messy, is that just something you have to live with doing it this way?

Every actual job that I've had has used option #2. If using Git, sometimes these dependencies are in a separate repo (e.g. Project and ProjectDeps which are checked out / cloned to side-by-side locations by each developer) to avoid bloating your main repo with large binary files, seeing that git sucks at that...

The open source / nix world tends to use #1, and some kind of package manager.

From what I've seen of your posts around here you've got a pretty impressive job history and been involved in some big projects, so if that's how you've worked in the past it's making me think that whilst it may not be the ideal world approach, it's probably the best one.

As for the submodule approach, how do you handle dependencies that don't have their own CMakeLists.txt (to pick a well-known example, Lua), do you find a fork with a CMakeLists.txt (e.g. LuaDist/lua) or is there a way I can get CMake to figure out how to use their makefile (even as I'm typing this it sounds insane)?

For the most part, I just write the CMake file for them. It's rarely more than a handful of lines for something tiny like Lua, and most larger libraries seem to use CMake already.

EDIT: Further to that, how do you handle all of the set calls that need to be made to turn off features in your dependencies builds (e.g. building tests), doing it in the CMakeLists.txt is fine, but messy, is that just something you have to live with doing it this way?

The option() command is your friend:
[source]option(BUILD_EXAMPLES off)
option(BUILD_CPU_DEMOS off)
option(BUILD_BULLET3 off)
option(BUILD_BULLET2_DEMOS off)
option(BUILD_EXTRAS off)
option(BUILD_UNIT_TESTS off)

add_subdirectory(bullet)

option(GLFW_BUILD_EXAMPLES off)
option(GLFW_BUILD_TESTS off)
option(GLFW_BUILD_DOCS off)
option(GLFW_INSTALL off)

add_subdirectory(glfw)

option(YAML_CPP_BUILD_TOOLS off)

add_subdirectory(yaml-cpp)
[/source]

Tristam MacDonald. Ex-BigTech Software Engineer. Future farmer. [https://trist.am]

For the most part, I just write the CMake file for them. It's rarely more than a handful of lines for something tiny like Lua, and most larger libraries seem to use CMake already.


Yeah, that's true. Even most of the small libraries I use are using CMake, and you're right, for something like Lua it's trivial to write one.

The option() command is your friend


I didn't realise option could be used that way, that's so much easier than what I've been doing. I've been using set like this, which is a real pain as the description has to match exactly.

[source]set(SDL_SHARED false CACHE BOOL "Build a shared version of the library" FORCE)[/source]

I think I'll go for using git submodules for anything open source and then for the few non-open dependencies I have I'll just bundle the binaries in the repo. Thanks for the help.

This topic is closed to new replies.

Advertisement