Detecting Drift Between Dev Container and Prod Image

A coding agent runs inside a devcontainer for two weeks. It edits source files, installs three new system packages with apt-get to make a build script work, upgrades pip to a pre-release because the current version raises a deprecation warning, and bakes everything into a working development image. The PR it eventually opens touches the application source — but never the production Dockerfile.

The CI build for production passes. The image deploys. In production, the same code raises a runtime error that did not appear in dev.

The drift was not in the code. It was in the environment: a libssl3 version, a pip version, a system locale, a CA bundle. The dev environment had those changes; the prod image did not. And because the agent worked entirely within the dev container, no PR ever surfaced the divergence.

This article shows the cosign-attestation-diff pattern that detects dev-vs-prod environmental drift at PR time, before the production image ships.

Why Devcontainers and Agents Conspire to Hide Drift

Devcontainers (the VS Code spec, GitHub Codespaces, Coder workspaces) exist precisely because dev environments need to be richer than prod: shells, debuggers, language servers, build chains, test fixtures. The whole point is divergence from prod.

Agents accelerate this divergence in three ways:

1. They install ad-hoc system packages. "I need graphviz to render this diagram for you." Fine in dev. The package never makes it into the prod Dockerfile.
2. They upgrade language toolchains. "Pinning to pip==25.0 to silence this warning." The dev container picks up the upgrade; the prod build still uses whatever was in requirements.txt.
3. They write code that depends on dev-only behaviour. A library that is present in the devcontainer's broader Python distribution but absent from the prod's distroless image. Code works locally; crashes in prod.

None of this shows up in a code diff. A reviewer looking at the PR sees clean Python and clean tests. The drift is invisible until production exposes it.

The Cosign Attestation Diff Approach

Both the dev container image and the prod image should carry signed attestations of their environment: SBOM, base image digest, system package list, language runtime version. At PR time, diff the two attestations and surface any environmental delta the PR would cause.

Step 1: Attest Both Images

When the devcontainer image is built (typically out-of-band, on a schedule, by a maintainer of the .devcontainer/Dockerfile):

DEV_IMAGE="ghcr.io/myorg/devcontainer@${DEV_DIGEST}"

syft "$DEV_IMAGE" -o cyclonedx-json > dev.sbom.json

# Cosign --key accepts a file path or one of the documented KMS schemes
# (awskms://, gcpkms://, azurekms://, hashivault://, openbao://, k8s://).
# It does NOT accept env://. To sign with a key held in a CI secret,
# write it to a tmpfs file first and clean up on exit.
echo "$COSIGN_PRIVATE_KEY" > /dev/shm/cosign.key
trap 'rm -f /dev/shm/cosign.key' EXIT

cosign attest \
  --predicate dev.sbom.json \
  --type cyclonedx \
  --key /dev/shm/cosign.key \
  "$DEV_IMAGE"

When the prod image is built (on every PR and every main-branch push):

PROD_IMAGE="ghcr.io/myorg/api@${PROD_DIGEST}"

syft "$PROD_IMAGE" -o cyclonedx-json > prod.sbom.json

echo "$COSIGN_PRIVATE_KEY" > /dev/shm/cosign.key
trap 'rm -f /dev/shm/cosign.key' EXIT

cosign attest \
  --predicate prod.sbom.json \
  --type cyclonedx \
  --key /dev/shm/cosign.key \
  "$PROD_IMAGE"

In CI flows that already provision an OIDC token (GitHub Actions, GitLab, etc.), a simpler alternative is keyless signing — drop --key and cosign attest --yes will use the workflow's OIDC identity instead of a long-lived key.

Both attestations are stored in the registry and discoverable via cosign download attestation.

Step 2: PR-Time Diff

A workflow that runs on every PR:

# .github/workflows/dev-prod-drift.yml
name: Dev/prod environment drift
on:
  pull_request:
    paths:
      - 'src/**'
      - '.devcontainer/**'
      - 'Dockerfile'
      - '**/requirements*.txt'
      - '**/package*.json'
      - '**/go.mod'

jobs:
  drift:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      pull-requests: write
    steps:
      - uses: actions/checkout@v4

      - name: Resolve dev container image
        id: dev
        run: |
          DEV_REF=$(jq -r '.image' .devcontainer/devcontainer.json)
          echo "image=$DEV_REF" >> "$GITHUB_OUTPUT"

      - name: Build prod image
        id: prod
        run: |
          # `docker build` alone does NOT populate .RepoDigests — that field
          # is set only by `docker pull` or after `docker push` to a registry
          # whose response carries a digest. For a digest captured at build
          # time without a push, use `docker buildx build --metadata-file`,
          # which writes containerimage.digest into a JSON file:
          docker buildx build --metadata-file build-meta.json -t prod-candidate:pr --load .
          DIGEST=$(jq -r '."containerimage.digest"' build-meta.json)
          echo "image=prod-candidate:pr" >> "$GITHUB_OUTPUT"
          echo "digest=$DIGEST" >> "$GITHUB_OUTPUT"

      - name: Generate prod SBOM
        run: syft prod-candidate:pr -o cyclonedx-json > prod.sbom.json

      - name: Fetch dev SBOM attestation
        run: |
          # `cosign verify-attestation` emits one DSSE envelope per matching
          # attestation, line-delimited. If the image carries multiple
          # cyclonedx attestations (e.g. one per build), `jq -r '.payload'`
          # over multi-document input is non-deterministic about which one
          # wins. `jq -s` slurps the stream into an array; `[-1]` selects
          # the most recently signed envelope, which is the one we want.
          cosign verify-attestation \
            --certificate-identity-regexp='^https://github\.com/myorg/' \
            --certificate-oidc-issuer='https://token.actions.githubusercontent.com' \
            --type cyclonedx \
            "${{ steps.dev.outputs.image }}" \
            | jq -s '.[-1].payload' -r | base64 -d \
            | jq '.predicate' > dev.sbom.json

      - name: Diff
        run: python3 scripts/env-drift.py dev.sbom.json prod.sbom.json > drift.md

      - uses: marocchino/sticky-pull-request-comment@v2
        with:
          path: drift.md

Step 3: A Drift Diff That Highlights What Matters

Not every diff between dev and prod is meaningful. Dev should have gdb, vim, bash. Prod should not. The diff script needs to know what to flag:

#!/usr/bin/env python3
# scripts/env-drift.py
import json, sys
from collections import defaultdict

# Packages we expect ONLY in dev — these are not drift, they are by design
DEV_ONLY_ALLOWED = {
    "vim", "vim-runtime", "gdb", "strace", "less", "man-db",
    "build-essential", "git", "curl", "wget", "openssh-client",
}

# Packages that, if they appear in dev but not prod, are usually a problem
RUNTIME_SENSITIVE = {
    "openssl", "libssl3", "libssl1.1", "ca-certificates",
    "libc6", "libstdc++6", "zlib1g", "libcurl4",
    "python3", "python3-pip", "node", "nodejs",
    "tzdata", "locales",
}

def load(p):
    sbom = json.load(open(p))
    return {c["name"]: c.get("version", "?") for c in sbom["components"]}

dev = load(sys.argv[1])
prod = load(sys.argv[2])

dev_only = {n: v for n, v in dev.items()
            if n not in prod and n not in DEV_ONLY_ALLOWED}
prod_only = {n: v for n, v in prod.items() if n not in dev}
version_drift = {n: (dev[n], prod[n])
                 for n in dev if n in prod and dev[n] != prod[n]}

print("## Dev/prod environment drift")
print()

# Highlight runtime-sensitive divergence first
runtime_drift = {n: vs for n, vs in version_drift.items()
                 if n in RUNTIME_SENSITIVE}
if runtime_drift:
    print("### :warning: Runtime-sensitive version drift")
    print()
    print("| Package | Dev | Prod |")
    print("|---|---|---|")
    for n, (d, p) in sorted(runtime_drift.items()):
        print(f"| `{n}` | `{d}` | `{p}` |")
    print()
    print("These are packages whose version difference between dev and prod ")
    print("commonly causes 'works on my machine' failures. Confirm the prod ")
    print("version is the one you intended to ship.")
    print()

# Dev-only packages that are NOT on the allowlist
suspect_dev_only = {n: v for n, v in dev_only.items()
                    if n in RUNTIME_SENSITIVE
                    or any(t in n for t in ["lib", "ssl", "crypto"])}
if suspect_dev_only:
    print("### :warning: Packages installed in dev but missing in prod")
    print()
    print("These look like runtime libraries an agent may have installed in ")
    print("dev to make code work — but the same libraries are not in the prod ")
    print("image. Code that depends on them will crash in prod.")
    print()
    for n, v in sorted(suspect_dev_only.items()):
        print(f"- `{n}@{v}`")
    print()

# Quiet stats
print("### Summary")
print()
print(f"- Dev-only (allowlisted): "
      f"**{len([n for n in dev_only if n in DEV_ONLY_ALLOWED])}**")
print(f"- Dev-only (unexpected): "
      f"**{len(suspect_dev_only)}**")
print(f"- Prod-only: **{len(prod_only)}**")
print(f"- Version drift (all): **{len(version_drift)}**")
print(f"- Version drift (runtime-sensitive): **{len(runtime_drift)}**")

if suspect_dev_only or runtime_drift:
    sys.exit(1)

The script exits non-zero when it finds either runtime-sensitive version drift or unexpected dev-only packages — the two classes of drift most likely to bite in production.

Step 4: Detect When an Agent Drove the Drift

If the dev container image was built or pushed by a coding agent (recognised by signing identity or trailers in the build commit), the diff comment should make that explicit:

# Add to env-drift.py
import subprocess

def dev_image_authored_by_agent(dev_image_ref):
    # Pull the dev image's build provenance
    r = subprocess.run(
        ["cosign", "download", "attestation",
         "--predicate-type", "https://crashoverride.com/attestations/agent-provenance/v1",
         dev_image_ref],
        capture_output=True, text=True)
    if r.returncode != 0:
        return False
    payload = json.loads(r.stdout)
    predicate = json.loads(
        __import__("base64").b64decode(payload["payload"]).decode()
    )["predicate"]
    return len(predicate.get("agent_commits", [])) > 0

When dev_image_authored_by_agent() is true, prepend the comment with a banner explaining the drift may originate in the agent's iteration on the devcontainer, and link to the originating commits.

Step 5: Decide What Blocks and What Warns

Three tiers, by sensitivity:

• Block on: new dev-only packages with names containing ssl, crypto, tls, cert, or pki. These almost always indicate an agent worked around a TLS or signing problem in dev that prod will hit fresh.
• Block on: version drift in glibc, openssl, libssl3. ABI breakage between dev and prod is a known-bad failure mode.
• Warn on: version drift in interpreters (Python, Node, Ruby) when the major version matches and minor differs. Often safe; sometimes not.
• Ignore: dev-only packages on the allowlist, prod-only packages (those are by design).

Step 6: Close the Loop With a Drift-Aware CI

If this workflow consistently fires on agent PRs, the agent's working environment is configured wrong. Fix it at the source:

• Pin the devcontainer base image to the same registry path as the prod base image, with a dev- prefix on the tag. That way, both images inherit the same hardened base, with dev adding tooling on top.
• Express the dev-only additions as a single layer in the .devcontainer/Dockerfile. Anything not in that layer must come from the shared base.
• Make the agent rebuild its devcontainer from the latest pinned base on every session start. Stale dev images are a major source of drift.

A drift-aware CI does not eliminate divergence between dev and prod — that is impossible and undesirable. It surfaces unintentional divergence at PR time, when the cost of fixing it is minutes, rather than at incident time, when the cost is hours and the audience includes your customers.

What Good Looks Like in Practice

A team running this workflow can answer four questions on every agent-authored PR:

1. Does this PR introduce code that depends on libraries present in the dev environment but missing in prod? Answered by the dev-only diff.
2. Does prod ship a different version of a runtime-sensitive package than the agent tested against? Answered by the version drift table.
3. Is the agent's dev environment current with the latest hardened base? Answered by comparing dev base digest to prod base digest.
4. Was the dev environment itself last modified by an agent? Answered by the agent-provenance attestation on the dev image.

Each answer takes seconds at PR time. Reconstructing the same answers post-incident takes a senior engineer half a day.

Common Failure Modes

• No SBOM on the devcontainer. If you cannot attest what is in dev, you cannot diff. Make devcontainer image builds first-class CI citizens.
• Dev image rebuilt rarely. Dev images need to be rebuilt on the same cadence as the prod base image bumps, or drift accumulates silently. Renovate-driven weekly rebuilds work well.
• Codespaces or Coder workspaces with prebuild caches. A long-lived prebuild may carry an older dev image than the current .devcontainer/Dockerfile would produce. Invalidate prebuilds on every devcontainer change.
• Multiple devcontainers per repo. Some repos run a different devcontainer per service. Run the diff against the relevant one — the workflow above assumes a single devcontainer; extend it to find the closest devcontainer to the changed paths.

The Through-Line

When agents iterate inside dev containers, the dev image becomes the agent's de facto environment of record. If you do not measure how that environment differs from prod, you are trusting the agent to keep them in sync — and the data on agent-authored Dockerfile changes shows that trust is misplaced more often than it is rewarded.

A signed attestation on each side, a diff on every PR, and a small set of hard-fail rules turn dev/prod drift from a category of post-incident debugging into a category of pre-merge CI failures. The agent gets the fast feedback it needs to converge; the production cluster gets the consistency it needs to not page anyone at 3am.

References

• Devcontainer specification
• Sigstore cosign verify-attestation
• CycloneDX SBOM specification 1.6
• GitHub Codespaces prebuild lifecycle
• Anchore Syft for SBOM generation