Publish a Reference

Overview

A test case variant’s reference implementation is the authored, correct build of the case: the answer key. Publishing one is a pull flow: deploy the build, commit the lockfile, then re-ingest so the backend reads it onto the case page’s Play tab.

An asset-generation case publishes differently; see Asset-generation references. The policy for which cases get a reference is in the full guide.

Prerequisites

• wrangler on PATH, with CLOUDFLARE_API_TOKEN (Cloudflare Pages: Edit) and CLOUDFLARE_ACCOUNT_ID set, plus Node and npm for the case’s [build] commands. The command contacts no backend, so it needs no login or token.
• The repository’s npm workspace installed and its packages built (npm ci && npm run build:packages at the repository root): an engine-backed reference resolves its engine from packages/<slug>/. tcab capture-baselines needs the same.
• The target Pages project exists: test-cabinet-references for prod, or test-cabinet-references-staging for staging. See Releasing.
• The case is non-experimental and declares a [build] table, which today means an end-to-end or full-stack case.

Publish

# 1. Deploy and write the lockfile. --env selects the Pages project AND the lock key.
tcab publish-reference --env prod <slug> [<version>] --dry-run     # show the plan first
tcab publish-reference --env prod <slug>                    # newest version, all variants
tcab publish-reference --env prod <slug> <version> --variant base  # exactly one variant

# 2. Commit and push the lockfile, then re-ingest so the backend reads it.
git add test-cases/reference-builds.lock.json
git commit -m "chore(references): record <slug>"
git push
scripts/reingest-cluster.sh --env prod

--env accepts prod or staging and is required, so a publish never silently targets prod. <version> defaults to the case’s newest version. With no variant selector, every variant declaring a reference_implementation is published; --variant <slug> targets one and errors when that variant declares none, and --all-variants states the default explicitly. A multi-variant sweep reports per-variant failures and exits non-zero when any failed, after attempting them all.

A variant has one reference build per engine, and the command targets each pair. It builds the reference, re-captures its committed baseline validation media, scrubs secrets, deploys to the --env Pages project under a <slug>-<version>-<variant>-<engine> branch alias, reads the served URL back from wrangler, and writes it into test-cases/reference-builds.lock.json under the --env key. --engine <slug> narrows a run to one engine. The backends ingest that lockfile from their own git checkout, which is what lands each URL on the variant’s referenceBuilds and the case page’s Play tab.

Asset-generation references

An asset-generation case declares no [build] table and produces no site, so the same command takes a different path for it. It needs the target environment’s R2 credentials rather than Cloudflare Pages access:

# TCAB_R2_ACCOUNT_ID  TCAB_R2_BUCKET  TCAB_R2_ACCESS_KEY_ID  TCAB_R2_SECRET_ACCESS_KEY
tcab publish-reference --env prod <slug> --dry-run   # show the plan and the object keys
tcab publish-reference --env prod <slug>
scripts/reingest-cluster.sh --env prod

It seeds a scratch workspace from the case manifest, runs each variant’s reference-impl/<variant>/draw.sh with the case’s drawing binary on PATH, and uploads every produced frame image and action log to the public snapshot bucket under media/references/<slug>/<version>/<variant>/. The command echoes the bucket it is writing to.

Two differences matter:

• The frames stay out of version control. The object keys are deterministic, so the backend discovers what exists by listing that prefix at ingest. Re-running the command after editing a script overwrites the objects in place, and reingest-cluster.sh still follows.
• The drawing binary comes from your machine. It is resolved from TCAB_ASSET_BIN_DIR, then the cargo target directory’s release/, then PATH. Build it first, for example cargo build --release -p test-cabinet-draw; the command names every location it tried when it finds nothing.

To see a reference before publishing it, render it locally:

node scripts/preview-asset-reference.mjs <slug>

That writes the frames, the action logs, and a GIF per sequence to tmp/asset-previews/<slug>/<variant>/, with an index.html showing them together. It needs no credentials and uploads nothing.

Baseline validation media

A case’s baseline validation media is the expected-behavior half of a reviewer’s side-by-side, with one validation-baseline/<engine>/<variant>/ directory per reference build. It is committed to the cold-storage submodule under the version’s mirrored path (see where baselines live), so check the submodule out first. Regenerating it is its own command, needing none of the credentials above:

git submodule update --init --depth 1 cold-storage
tcab capture-baselines <slug> [<version>] [--variant base] [--engine none] [--dry-run]

Run it whenever a validator or the reference implementation it runs against changes. It exits non-zero, naming the targets, when a reference build failed or left a unit that did not run clean against it, so commit only what a clean sweep wrote. Commit the media in cold-storage, push it to that repository’s master, then commit the moved submodule pointer here:

git -C cold-storage switch -C master
git -C cold-storage add test-cases
git -C cold-storage commit -m "feat: recapture <slug> <version> baselines"
git -C cold-storage push origin master
git add cold-storage

publish-reference re-captures the same media as part of its build and refuses to deploy a target whose capture failed; --skip-baselines deploys without re-capturing when the committed media is already current.