<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div style="font-family: monospace; font-size: 15px;">[RESENT MESSAGE — see <a href="https://mail.haskell.org/pipermail/ghc-devops-group/2017-October/000004.html" class="">https://mail.haskell.org/pipermail/ghc-devops-group/2017-October/000004.html</a>]</div><div style="font-family: monospace; font-size: 15px;">[Includes messages from Ben and me to which it responds.]</div><div class=""><br class=""></div><div><blockquote type="cite" class=""><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;" class=""><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif; color:rgba(0, 0, 0, 1.0);" class=""><b class="">From: </b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;" class="">"Boespflug, Mathieu" <<a href="mailto:m@tweag.io" class="">m@tweag.io</a>><br class=""></span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;" class=""><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif; color:rgba(0, 0, 0, 1.0);" class=""><b class="">Subject: </b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;" class=""><b class="">Aw: [GHC DevOps Group] DevOps: Next steps</b><br class=""></span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;" class=""><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif; color:rgba(0, 0, 0, 1.0);" class=""><b class="">Date: </b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;" class="">5. Oktober 2017 um 09:29:10 GMT+11<br class=""></span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;" class=""><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif; color:rgba(0, 0, 0, 1.0);" class=""><b class="">To: </b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;" class="">Ben Gamari <<a href="mailto:ben@well-typed.com" class="">ben@well-typed.com</a>><br class=""></span></div><div style="margin-top: 0px; margin-right: 0px; margin-bottom: 0px; margin-left: 0px;" class=""><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif; color:rgba(0, 0, 0, 1.0);" class=""><b class="">Cc: </b></span><span style="font-family: -webkit-system-font, Helvetica Neue, Helvetica, sans-serif;" class="">Jonas Pfenniger Chevalier <<a href="mailto:jonas.chevalier@tweag.io" class="">jonas.chevalier@tweag.io</a>>, <a href="mailto:ghc-devops-group@haskell.org" class="">ghc-devops-group@haskell.org</a><br class=""></span></div><br class=""><div class=""><div class="">Hi Ben,<br class=""><br class="">many thanks for your detailed and thoughtful reply. I won't myself<br class="">address your points one by one (I expect Manuel will jump in), but I<br class="">do want to ground the discussion with the following remarks:<br class=""><br class="">* What are the requirements that the current Jenkins effort building<br class="">towards? I seem to remember some page on the GHC wiki stating these<br class="">and then comparing various alternatives, but I can't find it now, so<br class="">maybe I dreamed it. The blog post [1] mentions alternatives but<br class="">doesn't evaluate them, nor does it state the requirements.<br class="">* A key requirement I think is not just that this kind of<br class="">infrastructure should not take time to setup given scarce development<br class="">resources, but more importantly that none of the maintenance be<br class="">bottlenecked on a single person managing a custom fleet of machines<br class="">whose state cannot be reproduced.<br class="">* Better yet if anyone that forks GHC (with a single click on GitHub)<br class="">gets a local copy of the CI by the same token, which can then be<br class="">modified at will.<br class="">* If we can get very quick wins today for at least 3 of the 4 "Tier 1"<br class="">platforms, that's already a step forward and we can work on the rest<br class="">later, just like Rust has (see below).<br class=""><br class="">I'll copy here an experience report [2] from the Rust infra authors<br class="">from before they switched to a Travis CI backed solution:<br class=""><br class=""><blockquote type="cite" class="">* Our buildbot-based CI / release infrastructure cannot be maintained<br class="">by community members, is generally bottlenecked on Alex and myself.<br class=""></blockquote><br class="">Sounds like this applies equally to the current Harbourmaster setup.<br class="">Perhaps to the Jenkins based one also?<br class=""><br class=""><blockquote type="cite" class="">* Our buildbot configuration has reliability issues, particularly around<br class="">managing dynamic EC2 instances.<br class=""></blockquote><br class="">Sounds familiar. Is any OS X automated testing happening at this<br class="">point? I heard some time befor ICFP that one or both of the OS X build<br class="">bots had fallen off the edge of the Internet.<br class=""><br class=""><blockquote type="cite" class="">* Our nightly builds sometimes fail for reasons not caught during CI and<br class="">are down for multiple days.<br class=""></blockquote><br class="">This matches my experience when adding CircleCI support: the tip of<br class="">the master branch at the time had failing tests.<br class=""><br class=""><blockquote type="cite" class="">* Packaging Rust for distribution is overly complex, involving<br class="">many systems and source repositories.<br class=""></blockquote><br class="">Yup. But admittedly this is an orthogonal issue.<br class=""><br class=""><blockquote type="cite" class="">* The beta and stable branches do not run the test suite today.<br class="">With the volume of beta backports each release receives this is<br class="">a freightening situation.<br class=""></blockquote><br class="">I assume this is not the case for us. But it's unclear where I'd look<br class="">to find a declarative description of what's going on for each branch?<br class="">Can each branch define their own way to perform CI?<br class=""><br class=""><blockquote type="cite" class="">* As certain core Rust tools mature we want to deliver them as part of<br class="">the Rust distribution, and this is difficult to do within the<br class="">current infrastructure / build system design. Distributing<br class="">additional tools with Rust is particularly crucial for those<br class="">intimately tied to compiler internals, like the RLS and clippy.<br class=""></blockquote><br class="">Also a familiar situation, though again an orthogonal issue.<br class=""><br class="">So it sounds like at this cross road we've been seeing a lot of the<br class="">same things the Rust team has experienced. The jurisprudence they've<br class="">established here is pretty strong. If we want to address the very same<br class="">problems then we need:<br class=""><br class="">1. Reproducible cloud instances that are created/destroyed on-demand,<br class="">and whose state doesn't drift over time. That way, no problems with<br class="">build bots that eventually disappear.<br class="">2. A declarative description of the *entire infrastructure and test<br class="">environment*, for each target platform, so that it can be replicated<br class="">by anyone who wants to so, in a single command. That way we're not<br class="">blocked on any single person to make changes to it.<br class=""><br class="">I believe reusing existing managed CI solutions. But let's discuss.<br class="">Just know that we'd be happy to contribute towards any paid<br class="">subscription necessary. So that shouldn't be a barrier.<br class=""><br class="">Best,<br class=""><br class="">Mathieu<br class=""><br class="">[1] <a href="https://ghc.haskell.org/trac/ghc/blog/jenkins-ci" class="">https://ghc.haskell.org/trac/ghc/blog/jenkins-ci</a><br class="">[2] <a href="https://internals.rust-lang.org/t/rust-ci-release-infrastructure-changes/4489" class="">https://internals.rust-lang.org/t/rust-ci-release-infrastructure-changes/4489</a><br class="">--<br class="">Mathieu Boespflug<br class="">Founder at <a href="http://tweag.io" class="">http://tweag.io</a>.<br class=""><br class=""><br class="">On 4 October 2017 at 19:30, Ben Gamari <<a href="mailto:ben@well-typed.com" class="">ben@well-typed.com</a>> wrote:<br class=""><blockquote type="cite" class="">Manuel M T Chakravarty <<a href="mailto:manuel.chakravarty@tweag.io" class="">manuel.chakravarty@tweag.io</a>> writes:<br class=""><br class=""><blockquote type="cite" class="">Hi Ben,<br class=""><br class=""></blockquote>Hi Manuel,<br class=""><br class="">Thanks again for your help here!<br class=""><br class=""><blockquote type="cite" class="">Since we talked last week, I have talked with Mathieu and Jonas (our<br class="">resident DevOps guru) about the whole CI situation and our discussion<br class="">about automating the production of build artefacts for GHC to make the<br class="">release process less labour-intensive. I am adding both to CC, so that<br class="">they can correct me if I am getting anything wrong.<br class=""><br class="">When we talked on the phone, you mentioned that we need to be able to<br class="">support all the Tier 1 platforms, and we both concluded that this<br class="">implies the need for using Jenkins and we can’t, e.g., use CircleCI as<br class="">they only support macOS and Linux. Mathieu and Jonas explained to me<br class="">that this is actually not the case. Apparently, Rust solves this issue<br class="">by building Linux and macOS artefacts on CircleCI, Windows on<br class="">Appveyor, and everything else using QEMU on CircleCI (e.g., FreeBSD<br class="">could be done that way and eventually ARM builds).<br class=""><br class=""></blockquote>Indeed when starting this I looked a bit at what rustc does. By my<br class="">recollection, they don't actually perform builds on anything but<br class="">Linux/amd64. Instead they build cross-compilers on x86-64, use these to<br class="">build their testsuite artifacts, and then run these under qemu (and in<br class="">some cases, e.g. FreeBSD, they don't even do this).<br class=""><br class="">While in general I would love to be able to do everything with<br class="">cross-compiled binaries from Linux/amd64, our cross-compilation story<br class="">may be a bit lacking to pull this off at the moment. Moritz Angerman has<br class="">been making great strides in this area recently but it's going to be a<br class="">while until we can really make this work. In particular, our Template<br class="">Haskell story will need quite some work before we can reliably do a full<br class="">cross-compiled testsuite run.<br class=""><br class="">In general I'm a bit skeptical of moving to a solution that relegates<br class="">non-Linux/amd64 builds to a VM. Non-Linux/amd64 platforms have<br class="">commercial users and do deserve first-class CI support. Furthermore,<br class="">without KVM or hypervisor support (which, as far as I can tell, CircleCI<br class="">does not provide [1]) I'm not sure that virtualisation will allow us to<br class="">get where we want to be in terms of test coverage and build response<br class="">time due to the cost of virtualisation. Without hardware support qemu<br class="">can be rather expensive.<br class=""><br class=""><blockquote type="cite" class="">They convinced me that this is a worthwhile direction to consider for<br class="">the following reasons:<br class=""><br class="">* Jenkins is a fickle beast: apparently scaling Jenkins to work<br class="">reliably when running tests against multiple PRs on distributed<br class="">infrastructure is hard — we ran into significant problems in a client<br class="">project recently.<br class=""><br class=""></blockquote><br class="">I agree that Jenkins is a rather fickle beast; indeed it can be<br class="">positively infuriating to work with. However, I've not yet noticed the<br class="">scaling issues you describe. What in particular did you observe?<br class=""><br class=""><blockquote type="cite" class="">* All the custom set up and maintaining of build nodes etc required by<br class="">Jenkins disappears. (Mathieu built the CircleCI setup that he<br class="">contributed recently quite quickly, so there really is little overhead<br class="">in setting this up.)<br class=""><br class=""></blockquote>I'm not sure that the difference here is actually so great. Yes, in the<br class="">case of Jenkins you do have physical machines to administer. However,<br class="">this typically isn't the hard part. If you look at Rust's configuration,<br class="">they have roughly a dozen Docker environments which they had to setup<br class="">and maintain; this effort will likely far outweigh the setup cost of the<br class="">machines themselves. This has certainly been the case for Jenkins and I<br class="">suspect it would be true of CircleCI as well; this is simply the cost to<br class="">entry for cross-platform testing.<br class=""><br class="">Moreover, we can't write off the cost of integrating with CircleCI. Of<br class="">course, if we do decide to move to GitHub then perhaps this cost shrinks<br class="">dramatically. However, until this decision is made it seems like we need<br class="">to assume that Phabricator integration will be necessary.<br class=""><br class=""><blockquote type="cite" class="">* The problems we discussed with possibly not having enough Rackspace<br class="">capacity for the transition disappears.<br class=""><br class=""></blockquote>In some sense this is true; however, it seems like we are trading one<br class="">commodity of finite supply for another. We currently have Rackspace<br class="">credit and consequently these instances can be considered to be<br class="">essentially free.<br class=""><br class="">While CircleCI is does offer four free containers for open source<br class="">projects (and perhaps a bit more in our case if we ask), I'm skeptical<br class="">that this will be enough; currently our four build bots give us<br class="">multi-day wait times which makes development remarkably painful. The<br class="">appeal of Jenkins is that we can shorten this timescale as well as grow<br class="">our test coverage with the resources that we already have.<br class=""><br class="">Let's have a brief look at what resources we may need.<br class=""><br class="">A quick back-of-the-envelope calculation suggests that to simply keep up<br class="">with our current average commit rate (around 200 commits/month) for the<br class="">four environments that we currently build we need a bare minimum of:<br class=""><br class=""> 200 commit/month<br class=""> * 4 build/commit (Linux/i386, Linux/amd64,<br class=""> OS X, Windows/amd64)<br class=""> * 2.5 CPU-hour/build (approx. average across platforms<br class=""> for a validate)<br class=""> / (2 CPU-hour/machine-hour) (CircleCI appears to use 2 vCPU instances)<br class=""> / (30*24 machine-hour/month)<br class=""> ~ 2 machines<br class=""><br class="">note that this doesn't guarantee reasonable wait times but rather merely<br class="">ensure that we can keep up on the mean. On top of this, we see around<br class="">300 differential revisions per month. This requires another 3 machines<br class="">to keep up.<br class=""><br class="">So, we need at least five machines but, again, this is a minimum;<br class="">modelling response times is hard but I expect we would likely need to<br class="">add at least two more machines to keep response times in the<br class="">contributor-friendly range, especially considering that under Circle CI<br class="">we will lose the ability to prioritize jobs (by contrast, with Jenkins<br class="">we can prioritize pull requests as this is the response time that we<br class="">really care about). Now consider that we would like to add at least<br class="">three more platforms (FreeBSD, OpenBSD, Linux/aarch64, all of which may<br class="">be relatively slow to build due to virtualisation overhead) as well as a<br class="">few more build configurations on amd64 (LLVM, unregisterised, at least<br class="">one cross-compilation target) and a periodic slow validation and we may<br class="">be at over a dozen machines.<br class=""><br class="">All of this appears to put us well outside CircleCI's offering to<br class="">open-source projects. Of course, it may be worth asking whether they are<br class="">willing to extend GHC a more generous offer. However, I don't think we<br class="">can count on this and I'm not certain that <a href="http://Haskell.org" class="">Haskell.org</a> is currently in a<br class="">position to be able to shoulder such a financial burden.<br class=""><br class=""><blockquote type="cite" class="">* We also don’t need to worry about a macOS box either.<br class=""><br class=""></blockquote>Quite true.<br class=""><br class=""><blockquote type="cite" class="">Also, Jonas could help us getting things running and, I think, his<br class="">wealth of experience would be very useful. (At least, I would be very<br class="">grateful for his advise.)<br class=""><br class="">I think, this route has the potential to get us to where we want to be<br class="">quite quickly and in a manner that is very little effort to maintain<br class="">once set up. What do you think?<br class=""><br class=""></blockquote>Indeed I can see that there are many advantages to the CircleCI option.<br class="">The ease of bringing up a Linux/amd64 build environment which easily<br class="">scales and requires no administration is quite enticing. However, I am a<br class="">skeptical that it will be as easy to get the full suite of builds that<br class="">we are aiming to produce. I would be quite curious to see what Jonas has<br class="">to say on the matter of non-Linux platforms. Seeing a simple<br class="">configuration which compiles and tests even a FreeBSD/amd64 build in a<br class="">reasonable amout of time may well be enough to convince me.<br class=""><br class="">Thanks again for your help on this!<br class=""><br class="">Cheers,<br class=""><br class="">- Ben<br class=""><br class=""><br class="">[1] <a href="https://circleci.com/docs/1.0/android/" class="">https://circleci.com/docs/1.0/android/</a><br class=""></blockquote>_______________________________________________<br class="">Ghc-devops-group mailing list<br class=""><a href="mailto:Ghc-devops-group@haskell.org" class="">Ghc-devops-group@haskell.org</a><br class="">https://haskell.org/cgi-bin/mailman/listinfo/ghc-devops-group<br class=""></div></div></blockquote></div><br class=""></body></html>