Easily track Linux binary provenance with Chalk

This article explains how to track the changes that happen to a Linux binary using Chalk

Chalk is an open-source project that lets you inject metadata into Linux binaries (and JAR files and Docker images and so on; we’re focusing on Linux binaries here).

Grab the Chalk binary.

curl -fsSL "https://dl.crashoverride.run/chalk/chalk-0.6.6-$(uname -s)-$(uname -m)" > chalk && chmod +x chalk
export CHALK=$(pwd)/chalk

Now let’s build Caddy, a static web server written in Golang, from source. First, install Golang 1.25+. Then fetch the Caddy repo and build it.

git clone https://github.com/caddyserver/caddy
cd caddy
go build ./cmd/caddy

It builds the caddy binary in the current directory.

$ file ./caddy
./caddy: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=6d7a0f74b5d43c76f54f4232036cea496ae8824b, with debug_info, not stripped

Now we’ll mark up the caddy binary with chalk. You’ll see a bunch of info in the output about the insertion process itself.

$ $CHALK insert ./caddy
warn:  Could not find or install cosign; cannot sign or verify.
info:  /home/admin/caddy/caddy: chalk mark successfully added
info:  /home/admin/.local/chalk/chalk.log: Open (sink conf='default_out')
info:  Full chalk report appended to: ~/.local/chalk/chalk.log
[
  {
    "_OPERATION": "insert",
    "_DATETIME": "2026-02-10T17:38:47.125+00:00",
    "_CHALKS": [
      {
        "CHALK_ID": "C4T64D-B36G-R64R-K3C5HP",
        "PRE_CHALK_HASH": "5cbc7aa20a2ebbb7ef25a7a03054159b98341772464d3d73d1b9141372e4ad4e",
        "PATH_WHEN_CHALKED": "/home/admin/caddy/caddy",
        "ARTIFACT_TYPE": "ELF",
        "ORIGIN_URI": "https://github.com/caddyserver/caddy",
        "COMMIT_ID": "5ff50779ccf1d5698bba48d140b713cff1a09390",
        "COMMIT_SIGNED": true,
        "BRANCH": "master",
        "AUTHOR": "Matt Holt <[email protected]>",
        "COMMITTER": "GitHub <[email protected]>",
        "COMMIT_MESSAGE": "Update LLM disclosure requirements in SECURITY.md\n\nClarified disclosure requirements for LLMs in security reports.",
        "DATE_AUTHORED": "2026-02-09T21:40:41.000+00:00",
        "TIMESTAMP_AUTHORED": 1770673241000,
        "DATE_COMMITTED": "2026-02-09T21:40:41.000+00:00",
        "TIMESTAMP_COMMITTED": 1770673241000,
        "CHALK_VERSION": "0.6.6",
        "METADATA_ID": "5XAWJ1-F79Q-B7J9-NZX4YN",
        "_VIRTUAL": false,
        "_CURRENT_HASH": "5cbc7aa20a2ebbb7ef25a7a03054159b98341772464d3d73d1b9141372e4ad4e"
      }
    ],
    "_AUTHOR": "Matt Holt <[email protected]>",
    "_BRANCH": "master",
    "_COMMITTER": "GitHub <[email protected]>",
    "_COMMIT_ID": "5ff50779ccf1d5698bba48d140b713cff1a09390",
    "_COMMIT_MESSAGE": "Update LLM disclosure requirements in SECURITY.md\n\nClarified disclosure requirements for LLMs in security reports.",
    "_COMMIT_SIGNED": true,
    "_DATE_AUTHORED": "2026-02-09T21:40:41.000+00:00",
    "_DATE_COMMITTED": "2026-02-09T21:40:41.000+00:00",
    "_ENV": {
      "PWD": "/home/admin/caddy",
      "XDG_SESSION_TYPE": "tty",
      "USER": "admin",
      "PATH": "/home/admin/.cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin",
      "SSH_TTY": "/dev/pts/0"
    },
    "_OP_ARGV": [
      "/home/admin/tmp/chalk",
      "insert",
      "./caddy"
    ],
    "_OP_CHALKER_VERSION": "0.6.6",
    "_ORIGIN_URI": "https://github.com/caddyserver/caddy",
    "_TIMESTAMP_AUTHORED": 1770673241000,
    "_TIMESTAMP_COMMITTED": 1770673241000,
    "_OP_CHALK_COUNT": 0,
    "_OP_UNMARKED_COUNT": 0
  }
]

Most of what is durably injected into the binary are within the _CHALKS value in that output. We can find exactly what’s been injected into the binary by running strings against it and filtering on MAGIC.

$ strings ./caddy | grep MAGIC | jq
{
  "MAGIC": "dadfedabbadabbed",
  "CHALK_ID": "C4T64D-B36G-R64R-K3C5HP",
  "CHALK_VERSION": "0.6.6",
  "TIMESTAMP_WHEN_CHALKED": 1770745127125,
  "DATETIME_WHEN_CHALKED": "2026-02-10T17:38:47.125+00:00",
  "ARTIFACT_TYPE": "ELF",
  "AUTHOR": "Matt Holt <[email protected]>",
  "BRANCH": "master",
  "CHALK_RAND": "a65164ebe32d13fb",
  "CODE_OWNERS": "# This is the official list of Caddy Authors for copyright purposes.\n# Authors may be either individual people or legal entities.\n#\n# Not all individual contributors are authors. For the full list of\n# contributors, refer to the proj
ect's page on GitHub or the repo's\n# commit history.\n\nMatthew Holt <[email protected]>\nLight Code Labs <[email protected]>\nArdan Labs <[email protected]>\n",
  "COMMITTER": "GitHub <[email protected]>",
  "COMMIT_ID": "5ff50779ccf1d5698bba48d140b713cff1a09390",
  "COMMIT_MESSAGE": "Update LLM disclosure requirements in SECURITY.md\n\nClarified disclosure requirements for LLMs in security reports.",
  "COMMIT_SIGNED": true,
  "DATE_AUTHORED": "2026-02-09T21:40:41.000+00:00",
  "DATE_COMMITTED": "2026-02-09T21:40:41.000+00:00",
  "HASH": "a4b5c40bbcacc8574c72141c9762252c878dfc089164303c5e4d0e4dce65b447",
  "INJECTOR_COMMIT_ID": "f74c891bce100229365f7a3939dc89421f1beb58",
  "ORIGIN_URI": "https://github.com/caddyserver/caddy",
  "PLATFORM_WHEN_CHALKED": "Linux aarch64",
  "METADATA_ID": "5XAWJ1-F79Q-B7J9-NZX4YN"
}

Chalk’s defaults capture quite a bit of useful information. (You can learn more about each key by running `chalk help metadata <key>`.) But we can also customize what it collects to be even more relevant to each particular project. Specifically, let's capture the go version used to build caddy.

Chalk lets us define custom metadata keys with a simple DSL we write in config files we’ll tell Chalk to load. In the config DSL we must first tell Chalk the type and name of the key and how to collect it. Then we must tell Chalk to actually use it by default.

echo '
# Define the new metadata by name and how and when to collect it.
# (Custom keys must be prefixed with X_ or _X_.
keyspec X_GO_VERSION {
  kind: ChalkTimeHost
  type: string
  value: strip(run("go version"))
}

# Tell Chalk to actually use this new key by default.
mark_template mark_default {
  key.X_GO_VERSION.use = true
}' > go_config.c4m

Now we register the config file with Chalk.

$ $CHALK load ./go_config.c4m
warn:  Could not find or install cosign; cannot sign or verify.
info:  Attempting to load module from: ./go_config.c4m


 Configuring Component: /home/admin/caddy/go_config
 Finished configuration for /home/admin/caddy/go_config
info:  [testing config]: Validating configuration.
info:  [testing config]: Configuration successfully validated.
info:  Configuration replaced in binary: /home/admin/tmp/chalk
info:  /home/admin/.local/chalk/chalk.log: Open (sink conf='default_out')
info:  Full chalk report appended to: ~/.local/chalk/chalk.log

Now let’s re-run chalk insert and check the new MAGIC that Chalk embeds.

$ $CHALK insert ./caddy
... chalk output ...
$ strings ./caddy | grep MAGIC | jq
{
  "MAGIC": "dadfedabbadabbed",
  "CHALK_ID": "C4T64D-B36G-R64R-K3C5HP",
  "CHALK_VERSION": "0.6.6",
  "TIMESTAMP_WHEN_CHALKED": 1770745252430,
  "DATETIME_WHEN_CHALKED": "2026-02-10T17:40:52.430+00:00",
  "ARTIFACT_TYPE": "ELF",
  "AUTHOR": "Matt Holt <[email protected]>",
  "BRANCH": "master",
  "CHALK_RAND": "3568a583d76b2bdf",
  "CODE_OWNERS": "# This is the official list of Caddy Authors for copyright purposes.\n# Authors may be either individual people or legal entities.\n#\n# Not all individual contributors are authors. For the full list of\n# contributors, refer to the proj
ect's page on GitHub or the repo's\n# commit history.\n\nMatthew Holt <[email protected]>\nLight Code Labs <[email protected]>\nArdan Labs <[email protected]>\n",
  "COMMITTER": "GitHub <[email protected]>",
  "COMMIT_ID": "5ff50779ccf1d5698bba48d140b713cff1a09390",
  "COMMIT_MESSAGE": "Update LLM disclosure requirements in SECURITY.md\n\nClarified disclosure requirements for LLMs in security reports.",
  "COMMIT_SIGNED": true,
  "DATE_AUTHORED": "2026-02-09T21:40:41.000+00:00",
  "DATE_COMMITTED": "2026-02-09T21:40:41.000+00:00",
  "HASH": "a4b5c40bbcacc8574c72141c9762252c878dfc089164303c5e4d0e4dce65b447",
  "INJECTOR_COMMIT_ID": "f74c891bce100229365f7a3939dc89421f1beb58",
  "ORIGIN_URI": "https://github.com/caddyserver/caddy",
  "PLATFORM_WHEN_CHALKED": "Linux aarch64",
  "X_GO_VERSION": "go version go1.25.7 linux/arm64",
  "METADATA_ID": "QH1ZT6-ZGSK-NPDX-M7PDAT"
}

And now we’re storing the version of Go this binary was built with too! Now copy that caddy binary somewhere else. Copy it to your Mac laptop.

$ uname -a
Darwin MacBook-Pro-2.local 25.2.0 Darwin Kernel Version 25.2.0: Tue Nov 18 21:07:05 PST 2025; root:xnu-12377.61.12~1/RELEASE_ARM64_T6020 arm64
$ file ./caddy
./caddy: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=6d7a0f74b5d43c76f54f4232036cea496ae8824b, with debug_info, not stripped
$ strings caddy | grep MAGIC | jq
{
  "MAGIC": "dadfedabbadabbed",
  "CHALK_ID": "C4T64D-B36G-R64R-K3C5HP",
  "CHALK_VERSION": "0.6.6",
  "TIMESTAMP_WHEN_CHALKED": 1770745252430,
  "DATETIME_WHEN_CHALKED": "2026-02-10T17:40:52.430+00:00",
  "ARTIFACT_TYPE": "ELF",
  "AUTHOR": "Matt Holt <[email protected]>",
  "BRANCH": "master",
  "CHALK_RAND": "3568a583d76b2bdf",
  "CODE_OWNERS": "# This is the official list of Caddy Authors for copyright purposes.\n# Authors may be either individual people or legal entities.\n#\n# Not all individual contributors are authors. For the full list of\n# contributors, refer to the project's page on GitHub or the repo's\n# commit history.\n\nMatthew Holt <[email protected]>\nLight Code Labs <[email protected]>\nArdan Labs <[email protected]>\n",
  "COMMITTER": "GitHub <[email protected]>",
  "COMMIT_ID": "5ff50779ccf1d5698bba48d140b713cff1a09390",
  "COMMIT_MESSAGE": "Update LLM disclosure requirements in SECURITY.md\n\nClarified disclosure requirements for LLMs in security reports.",
  "COMMIT_SIGNED": true,
  "DATE_AUTHORED": "2026-02-09T21:40:41.000+00:00",
  "DATE_COMMITTED": "2026-02-09T21:40:41.000+00:00",
  "HASH": "a4b5c40bbcacc8574c72141c9762252c878dfc089164303c5e4d0e4dce65b447",
  "INJECTOR_COMMIT_ID": "f74c891bce100229365f7a3939dc89421f1beb58",
  "ORIGIN_URI": "https://github.com/caddyserver/caddy",
  "PLATFORM_WHEN_CHALKED": "Linux aarch64",
  "X_GO_VERSION": "go version go1.25.7 linux/arm64",
  "METADATA_ID": "QH1ZT6-ZGSK-NPDX-M7PDAT"
}

Chalk embeds this metadata into the binary itself so no matter where it gets deployed you’ll retain this info.

Savvy Go users will know that you can get similar information without Chalk using go version -m, but among other reasons, Chalk 1) doesn’t require you to have Chalk installed to query this information and 2) lets you decide what Chalk injects and 3) isn’t limited to Go binaries.

One last point about integrity. While we can use strings to observe the metadata Chalk injects into artifacts, and we did this here to show how simple Chalk is, we recommend using chalk extract to pull this metadata out of binaries instead of directly inspecting the binaries. This is for a few reasons including that in production we’d be using Chalk to sign artifacts after we inject metadata. chalk extract will validate signatures which lets us rest assured that the binary hasn’t been tampered with.

And we’ve just talked about Crash itself. Crash Override offers a platform on top of Chalk that helps you easily discover and manage artifacts throughout your environments, but it’s all built on this simple open-source tool you can already start using today to track provenance on your own.