Skip to content
The Test Cabinet The Test Cabinet

Run a Test Case

Overview

Section titled “Overview”

Launch one test case variant through an agent harness and watch it to completion. This page uses the CLI; the web console launches the same run. Every launcher enqueues the run at the backend, which executes it as a per-run driver Job.

Prerequisites

Section titled “Prerequisites”
  • TCAB_BACKEND_URL pointing at a reachable backend whose queue an in-cluster dispatcher is draining. Locally that is the k3d service stack.
  • A signed-in account (tcab login).
  • Harness credentials configured in that deployment (Set Up Authentication). The launching machine needs no container runtime and no provider key of its own.

Run it

Section titled “Run it”
Terminal window
tcab run \
  --test-case carom --version v1.0.0 --variant base \
  --harness claude --model claude-opus-4-8

--test-case, --version, --variant, --harness, and --model are all required: a run targets exactly one variant, and --model is passed to the harness unchanged. The optional flags:

  • --max-runtime <hours> overrides the case’s max_runtime_hours for this invocation. Fractional hours are accepted, for example 0.5.
  • --auth-mode <auto|subscription|api-key> selects the harness authentication mode for this run.
  • --retry-count <n> sets how many times the backend retries the run after a terminal infrastructure error. The default is one retry; 0 disables retries.
  • --orchestrator <slug> selects the orchestrator that conducts the harness sessions. It defaults to one-shot; see tcab orchestrators.
  • --engine <slug> selects the engine the produced build is written against, one the case supports. It defaults to none; see tcab engines.
  • --out-dir <dir> also writes the finished run record JSON there.

The command prints the queued job id, streams the live event stream as the driver seeds the repository, drives the harness, and validates the result, then prints the run record summary.

From a source checkout, substitute cargo run -p test-cabinet-cli -- run … for tcab run ….

Inspect the inputs without a run

Section titled “Inspect the inputs without a run”
Terminal window
tcab prompt --test-case carom --version v1.0.0 --variant base   # the rendered prompt
tcab seed   --test-case carom --version v1.0.0 --variant base   # the seeded repo, on disk
tcab harnesses                                                  # harness availability

These read the local checkout and contact nothing.

Next steps

Section titled “Next steps”