Continuous Integration On Platforms

Started by
9 comments, last by SeanMiddleditch 7 years, 4 months ago

Hello forum!

Currently, I'm trying to build one project from its time beginning to exist on all platforms concurrently and regularly.

Hence I tried to get a logical grasp around the procedure first:

I'm mainly developing on my Windows 7 machine via Visual Studio, changes will be pushed via the TeamExplorer to my Git on BitBucket.

Now, I thought that my CI (TeamCity, open to other suggestions) should trigger on changes in the repository. Within the repository would reside a CMake-file, that dictates how to build the project. It would also check what platform is currently build on/for.

Therefore, my CI needs to cmake the project first, right? But how would it figure for which platform? And where to cmake the files or rather save the made files?

My testing platform runs on a VM, would this VM need to copy the repository and then run it? I tried running the CMakeLists.txt on my IDE of choice already, that worked. So, do I need another CI installation on my Linux guest with a trigger on the repository?

Maybe I got that wrong already or there are just multiple ways of doing it.

My aim is supporting Linux and then providing another build process for Android (using C++).

I'm not sure where what step would have to happen. If I didn't explain my question/situation clearly enough, just tell me!

And as always: Thanks for taking your time to read my thread!

Advertisement

Currently, I'm trying to build one project from its time beginning to exist on all platforms concurrently and regularly. Hence I tried to get a logical grasp around the procedure first:
CI is automating

1. Download source code

2. Unpack the source code (if not cloned from a repository)

3. Build

4. Install

5. Test (if you have them)

for most platforms or all platforms you support. In other words, exactly what a user would do with the sources. It needs download or unpack programs, build programs like cmake and make, it needs compilers and libraries.

You have to program the CI to do all the above, although many CI systems have stuff out of the box for something as common as "download/install".

Therefore, my CI needs to cmake the project first, right? But how would it figure for which platform? And where to cmake the files or rather save the made files?
Your confusion may be in not understanding that you also need the platform you want to test on. If you want to CI on Windows, Linux, and Mac, you need a Windows system, a Linux system, and a Mac system. Each of these systems runs a CI client controlled by the central CI. Often however, "a system" is merely a VM image that is started (perhaps also by the central CI, no experience how that works).

The CI client at the target system starts with downloading the sources, and then running cmake. cmake is being executed inside the target system, and it finds out what system it is running on, just like cmake finds out it's on a windows system, if you run it manually at your own system.

As for where to put the generated files, cmake does out-of-source builds, so I would make build instructions to do that, and then code the CI to follow those build instructions, so you are testing advice you give to the user.

The central CI mostly fires CI clients into action, it does administration of build results, and pending builds. It also provides nice web pages for you to view.

I have used buildbot for a number of years for CI on target systems, but those were physical systems also being used as desktop system, rather than VMs. Don't know how it works with the latter.

So, let me try to get an idea for it:

Usually, the central CI is on a server, which is my computer in this case. Is that right?

What is a build agent then? Is that another CI client on a different computer?

My computer pushes a change to the git, the central CI registers that something has been committed via a trigger that I have to set up on it.

Would it now notify the other CI clients? I see no use in this, as the other CIs could solely have triggers set up for that and do their own thing.

Ugh. This is a terribly complex area. I can barely find any easy-to-follow-tutorials either, while this being a complete new area for me.

I could manage to simply set a CI up for my windows machine. It triggers upon a change on my local project, builds that project via some MCVS build setting and done.

However this is faaar from what I want.

My CI shall react on changes on git, okay, this seems doable. Does this mean, all I need is a CI on each platform? Like, install TeamCity or whatever on my Linux VM?

Usually, the central CI is on a server, which is my computer in this case. Is that right? What is a build agent then? Is that another CI client on a different computer?

Build agents in Teamcity are the machines/installations that actually build your stuff. TC separates the management software (the Web UI you see and interact with plus it's systems) from the actual build runners. With this setup its easily possible to have multiple build agents, running on different machines in different software environments, fulfilling different build requirements. Example: your Windows build probably has to be build on a Windows machine, while your Linux build on some Linux distro, so in this case you can install an agent on a Linux box or vm, and another on a Win one. You create your build configurations for each one, and when you start a build on one of them, it will be built using the machine that fulfills all the requirements (having the right OS, toolset etc. installed). When the build finished, you have a build artifact, that is the result of the process. It can be the built application itself, or anything else, it's something you have to set up. Also a build in this context doesn't just mean we're building the application and doing nothing else, it usually involves extra steps, like running unit tests on the result, publishing the result based on the unit tests, code quality checks etc., all the CI servers I've worked with had the option to use shell scripted build steps, and with that you can do anything you want basicly.

Now the question is: Do you need this? If this is just your hobby project that you do alone, and the only thing you need is to build it sometimes to different targets, then I wouldn't bother with setting up an entire CI server, it's probably easier and more productive just to whip up some shell scripts that you run manually on an appropriate system when you need to generate a build.

So I need only one TC client and multiple build agents? When I wanted to add a build agent, it only offered me a windows one to download. I found that weird.

I thought about going over to Jenkins, as I would have only 3 build agents that. That might become a bit problematic in the future, if every build agent is only one platform.

Now the question is: Do you need this? If this is just your hobby project that you do alone, and the only thing you need is to build it sometimes to different targets, then I wouldn't bother with setting up an entire CI server, it's probably easier and more productive just to whip up some shell scripts that you run manually on an appropriate system when you need to generate a build.

I do this alone. However in terms of hobby, well, I try to learn these things as I might need them in the future. Especially, since I want to be able to offer my software on multiple platforms and not just one.

Not sure if just some shell scripts would be sufficient. I just want to commit a new change via my windows computer and then directly see, if it works on the other platforms.

So I need only one TC client and multiple build agents? When I wanted to add a build agent, it only offered me a windows one to download. I found that weird. I thought about going over to Jenkins, as I would have only 3 build agents that. That might become a bit problematic in the future, if every build agent is only one platform.

There should be a zip distribution option where it offers the install msi of the windows build agent. (Setting up the whole thing in linux requires a bit more manual work, but the documentation helps: https://confluence.jetbrains.com/display/TCD10/Setting+up+and+Running+Additional+Build+Agents#SettingupandRunningAdditionalBuildAgents-InstallingAdditionalBuildAgents) The whole stuff is written in java, and it's compatible with OpenJDK, so theoretically the build agents can run on anything that OpenJDK supports. In smaller projects the 3 agent cap shouldn't be a problem I think: in theory if you install an agent in a Win env, one in a Linux env and one on some kind of OSX running box, then you covered most of the things needed. With this setup you can build desktop apps for these OS-s, you can build an iOS app with the OSX agent, an Android app in any of them, a WP app in the Windows one etc. you probably get the idea. (Of course you need to install the needed build tools along with the build agents.) What you lose really is the ability to do multiple builds of the same type in parallel. Now probably you could do all of this with Jenkins too (it has a slave-master system, although I never used it), but I wouldn't really bother with it, we use Jenkins at work, and keeping one instance up and running properly is sometimes a challenge already. (We do web dev with Java+Spring+Angular and use Jenkins as the main CI server. We're evaluating TC at the moment, because Jenkins can be a real pain in the ass sometimes for the stuff we use it for.)

Usually, the central CI is on a server, which is my computer in this case. Is that right? What is a build agent then? Is that another CI client on a different computer? My computer pushes a change to the git, the central CI registers that something has been committed via a trigger that I have to set up on it. Would it now notify the other CI clients? I see no use in this, as the other CIs could solely have triggers set up for that and do their own thing.
At work, the setup is:

* You push changes in git.

* Central CI picks up the changes, begins compiling them for Windows, PS4 and Xbone.

* When each is compiled, it will instruct a slave (different CI computer) to test that build

** Windows build is tested on a Windows PC, PS4 build tested on a PS4, Xbone build on an XBone :)

* If there were no compilation errors, and if all the slaves report back that all tests were successful, then the central CI server approves your commit and pushes it into the master branch.

Basically, the Central CI server isn't capable of testing all the different builds, which is why the slaves are required. You also don't want the slaves acting independently, because if any of them fails, the central server rejects the commit as being unstable, otherwise it accepts it as a stable commit.

* Central CI picks up the changes, begins compiling them for Windows, PS4 and Xbone. * When each is compiled, it will instruct a slave (different CI computer) to test that build

I have a question: How is your central CI able to compile the code for all platforms? What compiler would it use?

My idea was to let the central CI clone the code on git, then activate its slaves/agents and let them compile and test the code.
Is this bad?

Since they would also report back to the central CI, if they could build or compile the code.

You also don't want the slaves acting independently, because if any of them fails, the central server rejects the commit as being unstable, otherwise it accepts it as a stable commit.

This is really interesting. My git experience is limited, but I often see those results about "Build Successful".

I have a question: How is your central CI able to compile the code for all platforms? What compiler would it use? My idea was to let the central CI clone the code on git, then activate its slaves/agents and let them compile and test the code. Is this bad?
No that's not bad. It can be required in some cases (platforms that don't supply compilers for your main OS -- e.g. Apple love to try and lock you into using MacOS), and it could speed up your builds as you'd have multiple machines sharing the work of compilation :)

Building code for one platform from a different platform is called "cross compiling". In the (console) games industry, we almost always compile builds for all of our platforms from Windows, using a cross-compiler for each of those platforms.

It can be required in some cases (platforms that don't supply compilers for your main OS -- e.g. Apple love to try and lock you into using MacOS)

That is actually my case!

n the (console) games industry, we almost always compile builds for all of our platforms from Windows, using a cross-compiler for each of those platforms.

Thanks! Another concept I can pick up to my "tool-box".

Are there any CI you would call beginner-friendly?

This topic is closed to new replies.

Advertisement