API reference

Dependency

A Dependency represents a set of Products that must be satisfied before a package can be run. These Products can be libraries, basic files, executables, etc...

To build a Dependency, construct it and use build(). To check to see if it is already satisfied, use satisfied().

QemuRunner

A QemuRunner represents an "execution context", an object that bundles all necessary information to run commands within the container that contains our crossbuild environment. Use run() to actually run commands within the QemuRunner, and runshell() as a quick way to get an interactive shell within the crossbuild environment.

UserNSRunner

A UserNSRunner represents an "execution context", an object that bundles all necessary information to run commands within the container that contains our crossbuild environment. Use run() to actually run commands within the UserNSRunner, and runshell() as a quick way to get an interactive shell within the crossbuild environment.

WizardState

Building large dependencies can take a lot of time. This state object captures all relevant state of this function. It can be passed back to the function to resume where we left off. This can aid debugging when code changes are necessary. It also holds all necessary metadata such as input/output streams.

audit(prefix::Prefix; platform::Platform = platform_key();
                      verbose::Bool = false,
                      silent::Bool = false,
                      autofix::Bool = false)

Audits a prefix to attempt to find deployability issues with the binary objects that have been installed within. This auditing will check for relocatability issues such as dependencies on libraries outside of the current prefix, usage of advanced instruction sets such as AVX2 that may not be usable on many platforms, linkage against newer glibc symbols, etc...

This method is still a work in progress, only some of the above list is actually implemented, be sure to actually inspect Auditor.jl to see what is and is not currently in the realm of fantasy.

autobuild(dir::AbstractString, src_name::AbstractString, platforms::Vector,
          sources::Vector, script::AbstractString, products::Function,
          dependencies::Vector; verbose::Bool = true, debug::Bool = false)

Runs the boiler plate code to download, build, and package a source package for a list of platforms. src_name represents the name of the source package being built (and will set the name of the built tarballs), platforms is a list of platforms to build for, sources is a list of tuples giving (url, hash) of all sources to download and unpack before building begins, script is a string representing a bash script to run to build the desired products, which are listed as Product objects within the vector returned by the products function. dependencies gives a list of dependencies that provide build.jl files that should be installed before building begins to allow this build process to depend on the results of another build process. Setting debug to true will cause a failed build to drop into an interactive shell so that the build can be inspected easily.

build(runner, dep::Dependency; verbose::Bool = false, force::Bool = false,
      autofix::Bool = false, ignore_audit_errors::Bool = true,
      debug::Bool = false)

Build the dependency for given platform (defaulting to the host platform) unless it is already satisfied. If force is set to true, then the dependency is always built. Runs an audit of the built files, printing out warnings if hints of unrelocatability are found. These warnings are, by default, ignored, unless ignore_audit_errors is set to false. Some warnings can be automatically fixed, and this will be attempted if autofix is set to true.

build_tarballs(ARGS, src_name, sources, script, platforms, products,
               dependencies)

This should be the top-level function called from a build_tarballs.jl file. It takes in the information baked into a build_tarballs.jl file such as the sources to download, the products to build, etc... and will automatically download, build and package the tarballs, generating a build.jl file when appropriate. Note that ARGS should be the top-level Julia ARGS command- line arguments object. This function does some rudimentary parsing of the ARGS, call it with --help in the ARGS to see what it can do.

collapse_symlinks(files::Vector{String})

Given a list of files, prune those that are symlinks pointing to other files within the list.

collect_files(path::AbstractString, predicate::Function = f -> true)

Find all files that satisfy predicate() when the full path to that file is passed in, returning the list of file paths.

If you have a sharded build on Github, it would be nice if we could get an auto-generated build.jl just like if we build serially. This function eases the pain by reconstructing it from a releases page.

supported_platforms()

Return the list of supported platforms as an array of Platforms. These are the platforms we officially support building for, if you see a mapping in get_shard_hash() that isn't represented here, it's probably because that platform is still considered "in beta".

satisfied(dep::Dependency; verbose::Bool = false, isolate::Bool = true)

Return true if all results are satisfied for this dependency.

analyze_instruction_set(oh::ObjectHandle; verbose::Bool = false)

Analyze the instructions within the binary located at the given path for which minimum instruction set it requires, taking note of groups of instruction sets used such as avx, sse4.2, i486, etc....

Some binary files (such as libopenblas) contain multiple versions of functions, internally determining which version to call by using the cpuid instruction to determine processor support. In an effort to detect this, we make note of any usage of the cpuid instruction, disabling our minimum instruction set calculations if such an instruction is found, and notifying the user of this if verbose is set to true.

Note that this function only really makes sense for x86/x64 binaries. Don't run this on armv7l, aarch64, ppc64le etc... binaries and expect it to work.

Canonicalize URL to a file within a GitHub repo

Canonicalize a GitHub repository URL

Change the script. This will invalidate all platforms to make sure we later
verify that they still build with the new script.

download_osx_sdk(;automatic::Bool = automatic_apple, verbose::Bool = false,
                  version::AbstractString = "10.10")

Apple restricts distribution and usage of the macOS SDK, a necessary component to build software for macOS targets. Please read the Apple and Xcode SDK agreement for more information on the restrictions and legal terms you agree to when using the SDK to build software for Apple operating systems: https://images.apple.com/legal/sla/docs/xcode.pdf.

If automatic is set, this method will automatically agree to the Apple usage terms and download the macOS SDK, enabling building for macOS.

To set this on an environment level, set the BINARYBUILDER_AUTOMATIC_APPLE environment variable to "true".

download_source(state::WizardState)

Ask the user where the source code is coming from, then download and record the relevant parameters, returning the source url, the local path it is stored at after download, and a hash identifying the version of the code. In the case of a git source URL, the hash will be a git treeish identifying the exact commit used to build the code, in the case of a tarball, it is the sha256 hash of the tarball itself.

downloads_dir(postfix::String = "")

Builds a path relative to the downloads_cache.

edit_script(state::WizardState, script::AbstractString)

For consistency (and security), use the sandbox for editing a script, launching vi within an interactive session to edit a buildscript.

filter_object_files(files)

Given a list of files, filter out any that cannot be opened by readmeta() from ObjectFile.

get_shard_url(target::String = "base"; squashfs::Bool = use_squashfs)

Returns the sha256 hash for a rootfs image (tarball/squashfs).

get_shard_url(target::String = "base"; squashfs::Bool = use_squashfs)

Returns the URL from which a rootfs image (tarball/squashfs) can be downloaded

getuid()

Wrapper around libc's getuid() function

init_git_config(repo, state)

Ask the user for their username and password for a repository-local .git/config file. This is used during an interactive wizard session.

instruction_mnemonics(path::AbstractString)

Dump a binary object with objdump from our super-binutils, returning a list of instruction mnemonics for further analysis with analyze_instruction_set().

Note that this function only really makes sense for x86/x64 binaries. Don't run this on armv7l, aarch64, ppc64le etc... binaries and expect it to work.

This function returns the list of mnemonics as well as the counts of each, binned by the mapping defined within instruction_categories.

interactive_build(state::WizardState, prefix::Prefix,
                  ur::Runner, build_path::AbstractString)

Runs the interactive shell for building, then captures bash history to save
reproducible steps for building this source. Shared between steps 3 and 5

is_ecryptfs(path::AbstractString; verbose::Bool=false)

Checks to see if the given path (or any parent directory) is placed upon an ecryptfs mount. This is known not to work on current kernels, see this bug for more details: https://bugzilla.kernel.org/show_bug.cgi?id=197603

This method returns whether it is encrypted or not, and what mountpoint it used to make that decision.

is_for_platform(h::ObjectHandle, platform::Platform)

Returns true if the given ObjectHandle refers to an object of the given platform; E.g. if the given platform is for AArch64 Linux, then h must be an ELFHandle with h.header.e_machine set to ELF.EM_AARCH64.

In particular, this method and platform_for_object() both exist because the latter is not smart enough to deal with :glibc and :musl yet.

match_files(state::WizardState, prefix::Prefix,
            platform::Platform, files::Vector; silent::Bool = false)

Inspects all binary files within a prefix, matching them with a given list of files, complaining if there are any files that are not properly matched and returning the set of normalized names that were not matched, or an empty set if all names were properly matched.

minimum_instruction_set(counts::Dict, is_64bit::Bool)

This function returns the minimum instruction set required, depending on whether the object file being pointed to is a 32-bit or 64-bit one:

• For 32-bit object files, this returns one of [:pentium4, :prescott]

• For 64-bit object files, this returns one of [:core2, :sandybridge, :haswell]

normalize_name(file::AbstractString)

Given a filename, normalize it, stripping out extensions. E.g. the file path "foo/libfoo.tar.gz" would get mapped to "libfoo".

Pick the first platform for use to run on. We prefer Linux x86_64 because
that's generally the host platform, so it's usually easiest. After that we
go by the following preferences:
    - OS (in order): Linux, Windows, OSX
    - Architecture: x86_64, i686, aarch64, powerpc64le, armv7l
    - The first remaining after this selection

platform_for_object(oh::ObjectHandle)

Returns the platform the given ObjectHandle should run on. E.g. if the given ObjectHandle is an x86_64 Linux ELF object, this function will return Linux(:x86_64). This function does not yet distinguish between different libc's such as :glibc and :musl.

print_autoconf_hint(state::WizardState)

Print a hint for projets that use autoconf to have a good ./configure line.

provide_hints(state::WizardState, path::AbstractString)

Given an unpacked source directory, provide hints on how a user might go about building the binary bounty they so richly desire.

rewrite_squashfs_uids(path, new_uid)

In order for the sandbox to work well, we need to have the uids of the squashfs images match the uid of the current unpriviledged user. Unfortunately there is no mount-time option to do this for us. However, fortunately, squashfs is simple enough that if the id table is uncompressed, we can just manually patch the uids to be what we need. This functions performs this operation, by rewriting all uids/gids to new_uid.

rootfs_dir(postfix::String = "")

Builds a path relative to the rootfs_cache.

runshell(platform::Platform = platform_key())

Launch an interactive shell session within the user namespace, with environment setup to target the given platform.

set_global_git_config(username, email)

Sets up a ~/.gitconfig with the given username and email.

Sets up travis for an existing repository

setup_workspace(build_path::String, src_paths::Vector,
                src_hashes::Vector, platform::Platform,
                extra_env::Dict{String, String};
                verbose::Bool = false, tee_stream::IO = stdout,
                downloads_dir = nothing)

Sets up a workspace within build_path, creating the directory structure needed by further steps, unpacking the source within build_path, and defining the environment variables that will be defined within the sandbox environment.

This method returns the Prefix to install things into, and the runner that can be used to launch commands within this workspace.

shards_dir(postfix::String = "")

Builds a path relative to the shards_cache.

step1(state::WizardState)

It all starts with a single step, the unabashed ambition to leave your current stability and engage with the universe on a quest to create something new, and beautiful and unforseen. It all ends with compiler errors.

This step selets the relevant platform(s) for the built binaries.

step2(state::WizardState)

This step obtains the source code to be built and required binary dependencies.

step34(state::WizardState)

Starts initial build for Linux x86_64, which is our initial test target platform. Sources that build properly for this platform continue on to attempt builds for more complex platforms.

step3_audit(state::WizardState, platform::Platform, prefix::Prefix)

Audit the prefix.

step3_interactive(state::WizardState, prefix::Prefix, platform::Platform,
                  ur::Runner, build_path::AbstractString)

The interactive portion of step3, moving on to either rebuild with an edited script or proceed to step 4.

step3_retry(state::WizardState)

Rebuilds the initial Linux x86_64 build after things like editing the script file manually, etc...

step4(state::WizardState, ur::Runner, platform::Platform,
      build_path::AbstractString, prefix::Prefix)

The fourth step selects build products after the first build is done

target_envs(target::String)

Given a target (this term is used interchangeably with triplet), generate a Dict mapping representing all the environment variables to be set within the build environment to force compiles toward the defined target architecture. Examples of things set are PATH, CC, RANLIB, as well as nonstandard things like target.

translate_symlinks(root::AbstractString; verbose::Bool=false)

Walks through the root directory given within root, finding all symlinks that point to an absolute path within root, and rewriting them to be a relative symlink instead, increasing relocatability.

update_linkage(prefix::Prefix, platform::Platform, path::AbstractString,
               old_libpath, new_libpath; verbose::Bool = false)

Given a binary object located at path within prefix, update its dynamic linkage to point to new_libpath instead of old_libpath. This is done using a tool within the cross-compilation environment such as install_name_tool on MacOS or patchelf on Linux. Windows platforms are completely skipped, as they do not encode paths or RPaths within their executables.

update_qemu(;verbose::Bool = false)

Update our QEMU and Linux kernel installations, downloading and installing them into the qemu_cache directory that defaults to deps/qemu.

update_rootfs(triplets::Vector{AbstractString};
              automatic::Bool = automatic_apple, verbose::Bool = true,
              squashfs::Bool = use_squashfs)

Updates the stored rootfs containing all cross-compilers and other compilation machinery for the given triplets. If automatic is set, when downloading Apple SDKs, you will automatically accept the Apple license agreement and download the macOS SDK for usage in targeting macOS. See the help for download_osx_sdk() for more details on this.

versioninfo()

Helper function to print out some debugging information

yn_prompt(state::WizardState, question::AbstractString, default = :y)

Perform a [Y/n] or [y/N] question loop, using default to choose between the prompt styles, and looping until a proper response (e.g. "y", "yes", "n" or "no") is received.