Migrating from Make to Hadrian (for packagers)

Sam Derbyshire - 2022-08-05

As the Hadrian build system for GHC has reached maturity and the old Make-based build system is becoming increasingly costly to maintain, the GHC maintenance team has decided that it is finally time to remove GHC’s Make-based build system. GHC 9.4 will be the last release series compatible with Make, which will be limited to booting with GHC 9.0. From 9.6 onwards, the only supported way to build GHC will be to use Hadrian.

This blog post will give an overview of using Hadrian, which should help packagers migrate from the old Make-based build system.

The Hadrian build system

Hadrian is a modular, statically-typed, extensible build system for GHC, introduced in the paper Non-recursive Make Considered Harmful. It consists of a Haskell executable implemented using the shake library, and is used to configure and build GHC.

Building Hadrian

Contributors to GHC will be accustomed to running the ./hadrian/build script, which builds and runs Hadrian. This script calls out to cabal, which fetches the dependencies of the Hadrian package from Hackage before building the resulting Haskell executable. While this is convenient for developers, it isn’t appropriate for build environments in which one doesn’t have access to the network (e.g. in order to enforce a hermetic build environment). For that reason, Hadrian provides a set of scripts for bootstrapping the build system from source tarballs. These can be found in the hadrian/bootstrap directory.

Bootstrapping Hadrian

The Hadrian bootstrap scripts are driven by a set of precomputed build plans; these depend on the version of the bootstrap GHC being used. A typical workflow might look like the following:

• Locally:
  • Choose a build plan appropriate for the bootstrap GHC version, such as hadrian/bootstrap/plan-bootstrap-8.10.7.json. (These build plans can also be generated manually from a cabal-install build plan; see generate_bootstrap_plans)
  • Fetch the sources needed by the build plan: bootstrap.py fetch -w <path_to_ghc> --deps plan-bootstrap-8.10.7.json -o 8_10_7_bootstrap_sources.tar.gz
• In the build environment:
  • Provision the bootstrap-sources tarball generated above.
• In your GHC build script:
  • Build Hadrian using the bootstrap script: bootstrap.py -w <path_to_ghc> --bootstrap-sources 8_10_7_bootstrap_sources.tar.gz
  • Build GHC using the resulting Hadrian executable, located by default in bin/hadrian, e.g. bin/hadrian -j --flavour=perf+debug_info -w <path_to_ghc>

An example of how to use these bootstrap scripts can be seen in the ghcs-nix repository. This repository contains Nix expressions specifying how to build many GHC versions, with both Make and Hadrian.

From now on, we will assume that you have built Hadrian (either via ./hadrian/build or via bootstrapping), referring to the hadrian executable agnostically.

Using Hadrian

How does Hadrian replace make?

To build GHC, we begin as before by running ./boot (if necessary, i.e. a configure file doesn’t already exist) and then ./configure <args>. As with Make, the build environment is determined by the configure script, which will read provided arguments as well as environment variables. For example, the selection of the bootstrap compiler is done via the GHC environment variable, and the selection of the C compiler uses the CC environment variable. This is unchanged, and details can be found on the GHC wiki.

Once the configure script is run, we replace make commands with hadrian commands. The fundamental command to build GHC with Hadrian is

hadrian <args>

Hadrian stages

GHC is a self-hosted compiler. This means that we need to provide a GHC executable in order to compile GHC. In Hadrian, this is done in stages:

• The stage0 compiler is the bootstrap compiler: a user-provided executable which will be used to compile GHC. The bootstrap compiler is chosen via the GHC variable passed to the configure script.
• The stage1 compiler is the compiler built using the stage0 compiler; it runs on the build platform and produces code for the target platform. The stage1 compiler is limited in that it does not support dynamic code loading via the internal bytecode interpreter.
• The stage2 compiler is the compiler built using the stage1 compiler; it runs on the target platform. The stage2 compiler is necessary for the implementation of Template Haskell and GHC plugins.

In Hadrian, build artifacts are put in a subdirectory of the build folder (by default, _build) corresponding to the stage of the compiler used to perform the build. This means that the stage2 compiler will be found (if using the default build directory, _build) at _build/stage1/bin/ghc.

Hadrian provides meta-targets which can be used to build particular subsets of the compiler. A typical Hadrian command, which builds a library or executable for a given stage, looks like

hadrian <stage>:{lib,exe}:<package name>

For example, hadrian stage2:lib:base will build the stage2 base library, and put it into the _build/stage1 subdirectory.

Flavours and flavour transformers

A Hadrian build flavour is a pre-defined collection of build settings that fully define a GHC build. These are described here. The flavour being used determines the ways in which GHC and its libraries will be built, as described in the Hadrian documentation. This replaces the variables of the make build system such as GhcLibWays, DYNAMIC_GHC_PROGRAMS, DYNAMIC_BY_DEFAULT.

A flavour is set using the --flavour command-line argument, e.g. hadrian/build --flavour=perf. As a packager you probably want to use either the release or perf flavour:

Flavours can be modified using flavour transformers. For example, the profiled_ghc flavour transformer compiles the GHC library and executable with cost-centre profiling enabled. One can, e.g., apply the profiled_ghc transformer to the perf flavour with hadrian --flavour=perf+profiled_ghc.

The full list of supported flavour transformers is available here.

Building GHC for distribution

Packagers will be interested in the binary-dist and binary-dist-dir Hadrian targets. For example, the command

hadrian/build --flavour=release binary-dist

will produce a complete release binary distribution tarball, while the binary-dist-dir target produces the directory only (not the tarball). The resulting bindist will be placed in _build/bindist.

When building the binary-dist target, documentation (namely, Haddock documentation and GHC’s Sphinx-built user’s guide) will be built by default. Building of documentation can be disabled using Hadrian’s --docs=... command-line flag. If you don’t want to build documentation, there are options to disable building various parts of the documentation. For example, if you don’t have Sphinx available, you can disable the parts of the documentation which require it:

# Build only the documentation for the base package, without using sphinx
hadrian {..} docs:base --docs=no-sphinx

Further information about configuring the documentation built by Hadrian can be found in the Hadrian readme.

Large-integer implementation

GHC supports several implementations of the Integer/Natural types and operations on them. The selection of the implementation is done using the --bignum Hadrian argument, e.g. --bignum=gmp to use the GMP library, or --bignum=native to use a pure-Haskell implementation.

Key-value settings

While we expect that the mechanisms described above will suffice for most builds, Hadrian also provides a fine-grained key-value configuration mechanism for modifying the command-lines passed to each of the tools run by Hadrian. For instance, one can pass an additional argument to all GHC invocations via:

hadrian {..} "*.*.ghc.*.opts += -my-ghc-option"

Passing an additional option when compiling the ghc library only:

hadrian {..} "*.ghc.ghc.*.opts += -my-ghc-option"

These settings can also be placed in a hadrian.settings file in the build root (by default _build), instead of passing them in the command line.

Hadrian currently supports the following key-value settings

• (<stage> or *).(<package name> or *).ghc.{hs, c, cpp, link, deps, *}.opts
  Arguments passed to GHC invocations.
  • hs for arguments passed to GHC when compiling Haskell modules
  • c for arguments passed to the C compiler
  • cpp for arguments passed to the C++ compiler
  • link for arguments passed during linking
  • deps for arguments to a ghc -M command, which outputs dependency information between Haskell modules
• (<stage> or *).(<package name> or *).cc.{c, deps, *}.opts
  Arguments passed directly to the C compiler.
• (<stage> or *).(<package name> or *).cabal.configure.opts
  Arguments passed to the cabal configure step.
• (<stage> or *).(<package name> or *).hsc2hs.run.opts
  Arguments passed when running Hsc2Hs.

These Hadrian key-value settings are useful to replace the assignment of Make variables, even though Hadrian is not intended to be a one-to-one replacement of Make; recovering the behaviour with Hadrian might require a few tweaks.

Consider for example the way that Make passes flags to GHC when compiling source Haskell files. Make has several different variables, such as SRC_HC_OPTS, WAY_<way>_<pkg>_OPTS, EXTRA_HC_OPTS. These are passed in order, with later flags overriding previous ones. With Hadrian, things are much simpler, and one can usually achieve the same goal by simply setting the *.*.ghc.hs.opts Hadrian key-value setting.

The following table serves as a general guideline in migrating the use of Make variables (bearing the above caveats in mind):

To pass module-specific or way-specific options, e.g. passing a C pre-processor option when compiling specific modules in a certain way (as when using a Way_<way>_<pkg>_OPTS Make variable), please use the programmatic interface described below.

Programmatic configuration

If the above configuration mechanisms aren’t sufficient, it is also possible to directly add new configurations to Hadrian. This allows finer-grained changes, such as changing the options when compiling a specific module or set of modules. If you really need to do this, you can read about it in the Hadrian manual. A good starting place to look for inspiration is Settings.Packages, which contains the arguments used to build GHC and the libraries it depends upon. The documentation for Shake is also a helpful resource, as Hadrian uses the Shake EDSL to implement its build rules.

Further support

If you are having any issues with packaging GHC after these changes, or find yourself needing to use the programmatic interface, please open an issue on the issue tracker, so that we can work together to modify Hadrian for your needs.