Both solutions incur hardware or subscription costs. In the case of Jenkins, we need servers (and we just learnt that RackSpace terminated its OSS program at the end of the year) and, in the case of CircleCI & AppVeyor, we will likely need more resource than they provide for free to open source projects. However, Google X and Tweag I/O have indicated that they would be happy to help cover such costs.
Both solutions incur some developer costs. However, by all accounts and our own experience, this is substantially lower for the CircleCI & AppVeyor solution. In particular, it completely eliminates server maintenance costs. Integration with GitHub is straight forward and Phabricator also seems to have some support.
Overall, CircleCI & AppVeyor are more cost-effective.
Jenkins is fully customisable and provides complete flexibility wrt to the architectures and operations systems used in build machines.
With CircleCI & AppVeyor, we are dependent on the features provided by the hosting services and limited in the architectures and OSes that are directly supported. In particular, we will have to fall back to cross-compiling and using QEMU for less commonly used platforms. Our base line, the Tier 1 platforms, as well as Android and iOS on ARM are supported by CircleCI & AppVeyor.
Overall, Jenkins is more flexible.
Security is crucial as we are planning to run untrusted and unreviewed code in PRs/differentials in the CI environment.
To provide adequate security, we need to add sandboxing to Jenkins (which is possible, but more work). However, even with that, the security track record of Jenkins is not very good.
In the case of CircleCI & AppVeyor, security is taken care of by the provider.
Overall, CircleCI & AppVeyor are more secure.
With forkability, we refer to a user's ability to fork GHC, modify it, and run their own CI instance (not using GHC HQ's infrastructure) without any significant extra work. This significantly improves scaling and takes load of the GHC HQ's infrastructure. CircleCI & AppVeyor is forkable, but Jenkins is not (as a user needs to set up their own infrastructure).
While Jenkins can be combined with other Docker/Kubernetes to improve scaling, this is difficult (e.g., Tweag I/O had some rather bad experiences with this). In contrast, this is handled by the provider in the CircleCI & AppVeyor case.
Overall, CircleCI & AppVeyor are more scalable.
With our own infrastructure, reliability is in our own hands, but we will not be able (as it would be too expensive) to provide any real reliability guarantees or support with guaranteed turn around times.
In the case of CircleCI & AppVeyor we depend on the providers for a reliable service. Ben talked to the Rust team who were very positive about CircleCI in this respect, but at the end of the day, it will be hard to know without trying (given we want to rely mostly on free open-source plans).
Overall, let's call this a tie.
A quick back-of-the-envelope calculation suggests that to simply keep up
with our current average commit rate (around 200 commits/month) for the
four environments that we currently build we need a bare minimum of:
200 commit/month * 4 build/commit (Linux/i386, Linux/amd64, OS X, Windows/amd64) * 2.5 CPU-hour/build (approx. average across platforms for a validate) / (2 CPU-hour/machine-hour) (CircleCI appears to use 2 vCPU instances) / (30*24 machine-hour/month) ~ 2 machines
Note that this doesn't guarantee reasonable wait times but rather merely
ensure that we can keep up on the mean. On top of this, we see around
300 differential revisions per month. This requires another 3 machines
to keep up.
So, we need at least five machines but, again, this is a minimum;
modelling response times is hard but I expect we would likely need to
add at least two more machines to keep response times in the
contributor-friendly range, especially considering that under Circle CI
we will lose the ability to prioritize jobs (by contrast, with Jenkins
we can prioritize pull requests as this is the response time that we
really care about). Now consider that we would like to add at least
three more platforms (FreeBSD, OpenBSD, Linux/aarch64, all of which may
be relatively slow to build due to virtualisation overhead) as well as a
few more build configurations on amd64 (LLVM, unregisterised, at least
one cross-compilation target) and a periodic slow validation and we may
be at over a dozen machines.
Linux/x86_64 (CircleCI): build & store artefacts works
macOS/x86_64 (CircleCI): build & store artefacts works
Windows/x86_64 (AppVeyor): asked for increased limits
Update (Jul. 24th, 2018): We wrote a small web application meant to act as a bridge between Phabricator and Circle CI. We have successfully used it to run Circle CI builds against Phabricator differentials. The code lives here. We will soon be deploying it and trying to use it for several job types. See here for an example of using this setup.
Below, we track all work that needs to be done until we have achieved the following specific goals:
automatic per-commit builds of master for Linux/x86_64 on CircleCI (per each block of commits from one PR/Differential would be sufficient),
automatic builds of Linux/i386, macOS/x86_64 & Windows/x86_64 on CircleCI and AppVeyor at least once per day,
automatic Linux/x86_64 build for each PR/Differential on CircleCl,
automatic generation of all release artefacts on Linux/i386, Linux/x86_64, macOS/x86_64 & Windows/x86_64 on CircleCI and AppVeyor at least once per day,
end-to-end testing by creating a source distribution, from that a binary distribution, and use that for regression testing. (This has the advantage of also testing the distribution creation process.)
Relevant Trac tickets
The following are issues about the CI system itself,
Talk to CircleCI about increased limits for the free plan for GHC. Determine how much on top of that we need.
AppVeyor has a 1h limit for OSS projects by default. Determine how much on top of the OSS plan we need and whether they are willing to relax the limits for a visible OSS project. Status: wrote email, but no response yet.
Per-commit build on Linux/x86_64
Probably easiest to just trigger those builds from GitHub (as all commits are mirrored there anyway).
Need to get builds on every individual commit (e.g., to do easy bisection) — see #14505 (closed). Alternative: Use GitHub PRs instead of pushing to master directly.
Daily builds on Linux/i386, macOS/x86_64 & Windows/x86_64
Implement AppVeyor build config. Blocked: waiting for AppVeyor to increase time limit. (Apparently, Rust are on a payed plan, so maybe we have to do that, too.)
Linux/i386 ought to be a small change on Linux/x86_64, or is there more to it?