README.md

# `download_utils`

> A Bazel extension to download archives, files and packages for use within Bazel targets

## Getting Started

Add the following to `MODULE.bazel`:

```py
bazel_dep(name="download_utils", version="0.0.0")
```

Use the repository rules in `MODULE.bazel` to download artifacts:

### `download_archive`

Download an archive and unpack it:

```py
download_archive = use_repo_rule("@download_utils//download/archive:defs.bzl", "download_archive")
download_archive(
    name = "archive",
    urls = ["https://some.thing/archive.tar"],
)
```

### `download_file`

Download a single file and optionally make it executable:

```py
download_file = use_repo_rule("@download_utils//download/file:defs.bzl", "download_file")
download_file(
    name = "file",
    output = "executable",
    executable = True,
    urls = ["https://some.thing/executable-amd64-linux"],
)
```

### `download_deb`

Download a Debian package, unpack it then unpack the `data.tar.{xz,zst}`:

```py
download_deb = use_repo_rule("@download_utils//download/deb:defs.bzl", "download_deb")
download_deb(
    name = "deb",
    integrity = "sha256-vMiq8kFBwoSrVEE+Tcs08RvaiNp6MsboWlXS7p1clO0=",
    urls = ["https://some.thing/test_1.0-1_all.deb"],
    commands = {
        "chmod": [
            "$(location @coreutils)",
            "chmod",
            "u+x",
            "some-script.sh",
        ],
    },
    links = {
        "etc/test/fixture.txt": "fixture.txt",
    },
    tools = [
        "@coreutils",
    ],
)
```

# `download_template`

Use an extension to template the attributes based on provided `substitutions`:

```py
download = use_extension("@download_utils//download/template:defs.bzl", "download_template")
download.archive(
    name = "coreutils-{version}-{triplet}",
    srcs = ["entrypoint"],
    links = {
        "coreutils{executable.extension}": "entrypoint",
    },
    # Run `bazel run @download_utils//download/template/lock coreutils/lock.json` to lock integrities
    lock = "//coreutils:lock.json",
    strip_prefix = "coreutils-{version}-{rust.triplet}",
    substitutions = {
        "version": [
            "0.0.28",
            "0.1.0",
        ],
        "triplets": [
            "arm64-linux-gnu",
            "amd64-linux-gnu",
            "arm64-linux-musl",
            "amd64-linux-musl",
            "amd64-windows-msvc",
            "arm64-macos-darwin",
            "amd64-macos-darwin",
        ],
    },
    # Run `bazel run @download_utils//download/template/lock:upload coreutils/lock.json` to mirror binaries and lock integrities
    uploads = [
        "https://gitlab.arm.com/api/v4/projects/bazel%2Fdownload_utils/packages/generic/coreutils/{version}/{rust.archive.basename}",
    ],
    urls = [
        "https://gitlab.arm.com/api/v4/projects/bazel%2Fdownload_utils/packages/generic/coreutils/{version}/{rust.archive.basename}",
        "https://github.com/uutils/coreutils/releases/download/{version}/coreutils-{version}-{rust.archive.basename}",
    ],
)

# Run `bazel mod tidy` to update these
use_repo(
    download,
    "coreutils-0.0.28-amd64-linux-gnu",
    "coreutils-0.0.28-amd64-linux-musl",
    "coreutils-0.0.28-amd64-macos-darwin",
    "coreutils-0.0.28-amd64-windows-msvc",
    "coreutils-0.0.28-arm64-linux-gnu",
    "coreutils-0.0.28-arm64-linux-musl",
    "coreutils-0.0.28-arm64-macos-darwin",
    "coreutils-0.1.0-amd64-linux-gnu",
    "coreutils-0.1.0-amd64-linux-musl",
    "coreutils-0.1.0-amd64-macos-darwin",
    "coreutils-0.1.0-amd64-windows-msvc",
    "coreutils-0.1.0-arm64-linux-gnu",
    "coreutils-0.1.0-arm64-linux-musl",
    "coreutils-0.1.0-arm64-macos-darwin",
)
```

Extra substitutions, such as `{rust.archive.basename}` are created using the `download_template.substitution` and `download_template.substitutions` APIs:

```py
download = use_extension("@download_utils//download/template:defs.bzl", "download_template")

# Create derived `{cpu}` substitution based on an incomming `{triplet}` substitution.
download.substitution(
    name = "cpu",
    match = "{triplet}",
    select = {
        "//conditions:default": "{triplet.split('-')[0]}",
    },
)

# Read substitutions from a JSON file
download.substitutions(
    srcs = [
        "substitutions.json",
    ],
)
```

The layout of the JSON is as so:

```JSON
{
  "<name>": "<value>",
  "<name>": {
    "<match>": {
        "<key>": "<value>",
    },
  },
}
```

Concretly, as an example, two substitutions:

```json
{
  "os": "{triplet.split('-')[1]}",
  "rust.vendor": {
    "{os}": {
      "macos": "apple",
      "windows": "pc",
      "//conditions:default": "unknown"
    }
  }
}
```

`@download_utils` has a set of subtitutions already registered. See `download/template/*.json`

### Integrity

The [sub-resource integrity (SRI)][sri] is not required for secure URLs. For non-secure (`http`, `ftp`) it is. It is
recommended to _always_ add the `integrity` to allow reproducible builds and sharing of downloads. When the integrity is
omitted and the rule is resolved, the correct SRI is output to the terminal. The easiest way to download the artifact is
to query the targets within the repository: `bazelisk query @archive//...`.

### Patches

The rules accept `patches` to modify the content after download. Examples are provided in the [end to end][e2e] tests.

Patches are applied _after_ the `BUILD.bazel` file is written so can be used to customise the targets exposed to the
Bazel build. _Usually_, however, a custom `BUILD.bazel` file is provided to the `build` argument.

### Commands

Hermetic commands can be ran against the unpacked repository. Hermetic binaries can be provided via the `tools` argument and be used with `$(location <label>)` in the
`commands`. The [end to end][e2e] tests provided examples of this.

### Sources

By default, the `BUILD.bazel` file exports all files downloaded. This can be controlled with the `srcs` attribute to
customise the files that are exposed to the Bazel build.

### Links

Symlinks/hardlinks can be created in the download repositories by providing the `links` map of target files to link names.

[sri]: https://developer.mozilla.org/en-US/docs/Web/Security/Subresource_Integrity
[commands]: lib/commands.bzl
[e2e]: e2e/MODULE.bazel

## Hermeticity

This ruleset is entirely hermetic and does not require anything from the system.

The `commands` supports `$(location <label>)` for running hermetic commands on the downloaded data.

## Release Registry

The project publishes the relevant files to GitLab releases for use when a version has not been added to the upstream [BCR][bcr].

This is often the case for pre-release versions.

Add the following to `.bazelrc`:

```
# `bzlmod` pre-release registries
common --registry https://bcr.bazel.build
common --registry=https://gitlab.arm.com/bazel/download_utils/-/releases/v1.0.0-alpha.1/downloads
```

Then a GitLab release version can be used in `bazel_dep`.

[bcr]: https://registry.bazel.build/