The purpose of the
BinaryBuilder.jl Julia package is to provide a system for compiling 3rd-party binary dependencies that should work anywhere the official Julia distribution does. In particular, using this package you will be able to compile your large pre-existing codebases of C, C++, Fortran, Rust, Go, etc... software into binaries that can be downloaded and loaded/run on a very wide range of machines. As it is difficult (and often expensive) to natively compile software packages across the growing number of platforms that this package will need to support, we focus on providing a set of Linux-hosted cross-compilers. This package will therefore set up an environment to perform cross-compilation for all of the major platforms, and will do its best to make the compilation process as painless as possible.
Note that at this time, BinaryBuilder itself runs on Linux
x86_64 and macOS
x86_64 systems only, with Windows support under active development. On macOS and Windows, you must have
docker installed as the backing virtualization engine. Note that Docker Desktop is the recommended version; if you have Docker Machine installed it may not work correctly or may need additional configuration.
Suppose that you have a Julia package
Foo.jl which wants to use a compiled
libfoo shared library. As your first step in writing
Foo.jl, you may compile
libfoo locally on your own machine with your system compiler, then using
Libdl.dlopen() to open the library, and
ccall() to call into the exported functions. Once you have written your C bindings in Julia, you will naturally desire to share the fruits of your labor with the rest of the world, and this is where
BinaryBuilder can help you. Not only will
BinaryBuilder aid you in constructing compiled versions of all your dependencies, but it will also build a wrapper Julia package (referred to as a JLL package) to aid in installation, versioning, and build product localization.
The first step in the
BinaryBuilder journey is to create a build recipe, usually named
build_tarballs.jl. The Julia community curates a tree of build recipes, Yggdrasil, that already contains many examples of how to write a
build_tarballs.jl file. These files contain information such as the name, version and source locations for a particular build, as well as the actual steps (in the form of a
bash script) and the products that should be generated by the build.
The result of a successful build is an autogenerated JLL package, typically uploaded to the JuliaBinaryWrappers github organization. Binaries for each version of every build are uploaded to the GitHub releases page of the relevant JLL package. Finally, a registration request is opened against the
General Julia registry, so that packages such as the aforementioned
Foo.jl can simply
pkg> add libfoo_jll to download the binary artifacts as well as the autogenerated Julia wrapper code. See also the FAQ, build tips, build troubleshooting and tricksy gotchas for help with common problems.
BinaryBuilder.jl supports an interactive method for building the binary dependencies and capturing the commands used to build it into a
build_tarballs.jl file: the Wizard interface. To launch it, run
state = BinaryBuilder.run_wizard()
and follow the instructions on-screen. You can watch an asciinema demo of the use of the wizard.
The wizard is a great tool, especially to get started with BinaryBuilder and create your first simple recipes for new packages. However, it lacks the full control of all options you can use in a
build_tarballs.jl script. To generate this file (explained in greater detail in Building Packages), one can clone
Yggdrasil, copy an existing build recipe, modify it, and submit a new pull request. Manually editing the
build_tarballs.jl script is also the recommended way when you want to update an existing recipe, rather then starting from scratch with the wizard.
build_tarballs.jl script can be used as a command line utility, it takes a few options and as argument the list of triplets of the targets. You can find more information about the syntax of the script in the Command Line section or by running
julia build_tarballs.jl --help
You can build the tarballs with
julia build_tarballs.jl --debug --verbose
--debug option will drop you into the BinaryBuilder interactive shell if an error occurs. If the build fails, after finding out the steps needed to fix the build you have to manually update the script in
build_tarballs.jl. You should run again the above command to make sure that everything is actually working.
build_tarballs.jl takes as argument the comma-separated list of triplets for which to build the tarballs, you can select only a few of them. For example, with
julia build_tarballs.jl --debug --verbose aarch64-linux-musl,arm-linux-musleabihf
you'll run the build script only for the
arm-linux-musleabihf target platforms.
If you decide to use this workflow, however, you will need to manually open pull requests for Yggdrasil.
If you already have access to the GitHub Codespaces service, you can use BinaryBuilder and all the workflows described above in your browser or with Visual Studio Code, on any operating system, including those not natively supported by the package! Head to Yggdrasil and create a new Codespace.
BinaryBuilder.jl wraps a root filesystem that has been carefully constructed so as to provide the set of cross-compilers needed to support the wide array of platforms that Julia runs on. This RootFS is then used as the chroot jail for a sandboxed process which runs within the RootFS as if that were the whole world. The workspace containing input source code and (eventually) output binaries is mounted within the RootFS and environment variables are setup such that the appropriate compilers for a particular target platform are used by build tools.
Reproducible builds are a set of software development practices that create an independently-verifiable path from source to binary code.
BinaryBuilder.jl puts into place many of the practices needed to achieve reproducible builds. For example, the building environment is sandboxed and uses a fixed tree structure, thus having a reproducible build path. The toolchain used by
BinaryBuilder.jl also sets some environment variables and enforces certain compiler flags which help reproducibility.
BinaryBuilder.jl does not guarantee to always have reproducible builds, it achieves this goal in most cases. Reproducibility in
BinaryBuilder.jl includes also the generated tarballs: they are created with
Tar.jl, which takes a few measures to ensure reproducibility of tarballs with the same git tree hash. If you rebuild multiple times the same package, with the same version of BinaryBuilder, the generated tarball which contains the main products (i.e. not the log files which are known not to be reproducible) should always have the same git tree hash and SHA256 sum, information which are printed to screen at the end of the build process and stored in the
Artifacts.toml file of the JLL package.
There are however some caveats:
- reproducibility can only be expected when using the toolchain offered by
- there are very specific cases where the macOS C/C++ toolchain does not produce reproducible binaries. This happens when doing debug builds (
-gflag) and not building object files with deterministic names separately (e.g. if directly building and linking a program or a shared library from the source file, letting the compiler create the intermediate object files automatically with random names). We have decided not to take action for this case because in practice most packages use build systems which compile intermediate object files with deterministic names (which is also the only way to take advantage of
BinaryBuilder.jluses extensively) and typically do not do debug builds, thus sidestepping the issue entirely.
BinaryBuilder has been covered in some videos, you may want to check them out if you want to know more about the framework (the date is specified in parentheses, to make it clear how old/new the videos are):
- 10 tips on how to build better binaries: JuliaCon 2018 talk by Elliot Saba introducing an early version of BinaryBuilder (2018-08-09)
- Introduction to BinaryBuilder.jl: live building session by Mosè Giordano (2020-04-10)
- BinaryBuilder.jl - The Subtle Art of Binaries That Just Work: JuliaCon 2020 workshop by Elliot Saba and Mosè Giordano to guide users through the use of BinaryBuilder (2020-07-25)
- Your first BinaryBuilder.jl recipe with Julia: live building by Miguel Raz Guzmán Macedo (2021-04-07)
- BinaryBuilder.jl — The Subtle Art of Binaries That "Just Work": AlpineConf 2021 talk by Elliot Saba and Mosè Giordano, starts at 4:19:00 (2021-05-15)
- BinaryBuilder.jl — Using Julia's Pkg to deliver binary libraries: PackagingCon 2021 talk by Mosè Giordano & Elliot Saba (2021-11-10)