Headless CI Grading

tools/sprint-headless lets a CI job grade an already-open PR against a ticket's approved plan, using Claude Code's headless (claude -p) mode — no human present, no unattended code writing.

Prerequisites

What It Does, and Doesn't Do

Making a Ticket CI-Eligible

  1. Author plan.md/acceptance.md for the ticket as usual (by hand or via sprint start), and check - [x] Plan approved.
  2. Mark it CI-eligible: tkt ci <id> on. This force-adds and commits the ticket's docs itself (git add -f .tickets/<id>/ + a commit) — a CI checkout has nothing to grade against otherwise, and .tickets/ is gitignored by default inside canon's own repo. In a consumer project without a .tickets/ gitignore entry, the force-add is a harmless no-op (force-adding a file that isn't ignored just adds it normally). No separate manual step is needed.

Everything else about the ticket stays exactly as normal — tkt ci <id> off stops force-tracking further changes (existing commits aren't un-committed; whether the directory reverts to untracked-and-ignored depends entirely on whether your own project's .gitignore covers .tickets/, which canon's tooling never sets up for you).

Running It

tools/sprint-headless <ticket-id> --base-ref <ref>

--base-ref (or $GITHUB_BASE_REF) points at the PR's target branch. When omitted it defaults to origin/main (then main, else a clear error) — convenient for local runs; in CI always pass it explicitly (the workflow template does), so the diff base is never ambiguous.

Exit code 0 means all three gates passed; exit code 1 means a gate failed, or the invocation itself errored (auth, rate limit, dispatch failure — hard-fail, fail closed). The full grading summary, including each gate's individual verdict, prints to stdout.

Note the reviewer's binding-ness differs from interactive close: at an interactive sprint complete the reviewer gate is advisory (a NO verdict doesn't block close — see complete.md/review.md). Headless CI is stricter — a reviewer NO (like an evaluator fail or a HIGH security finding) makes the overall verdict FAIL and forces exit 1, because CI has no human in the loop to weigh an advisory verdict.

Tier: bugfix tickets — reviewer skipped (eval-only). When the committed plan.md ## Sign-off records Tier: bugfix (canon's eval-only tier), headless dispatches only the evaluator + security-review and skips the advisory reviewer entirely — matching interactive sprint complete, where bugfix drops the advisory reviewer but keeps the binding evaluator. The verdict is then PASS iff the evaluator passes and security-review finds no HIGH finding. Headless reads the tier from the Tier: field in ## Sign-off (not the type: frontmatter, which is the orthogonal job type), and it trusts the committed tier: because bugfix only skips the advisory reviewer, a wrongly-set Tier: bugfix costs at most a skipped advisory pass — the binding evaluator and security-review always run. (Mechanically detecting/proposing the tier from the diff is tracked separately; headless only honors what is committed.)

Choosing the model (cost control). By default the gates run on claude's default model (why unqualified runs land on Sonnet/Opus). To run them cheaper — e.g. Haiku for a low-risk PR — set the model one of two ways:

The model applies to the dispatched reviewer/evaluator subagents (the expensive part), not just the orchestrator — verified via the run's modelUsage.

Verdict Caching (opt-in)

Both graders can cache a verdict so an unchanged diff isn't re-graded — prime-agent's "don't rerun the same gate when the workspace hasn't changed," applied as canon's cost proportional to risk. It is off by default: canon never silently reduces gate assurance, so a cached reuse must be an explicit, auditable choice.

Enable it per-run with --cache (or CANON_GATE_CACHE=1); --no-cache always wins:

sprint-headless-eval t-1234 --base-ref origin/main --cache CANON_GATE_CACHE=1 sprint-headless t-1234 --base-ref origin/main

This only speeds the headless graders. The interactive sprint complete close is untouched — it always runs its fresh-context gates.

When a Gate Legitimately Can't Pass Headlessly

Some acceptance criteria are inherently untestable by an automated evaluator — e.g. a claim that only a real claude -p dispatch can verify, which would cost real API money on every CI run. If the evaluator correctly fails such a criterion, headless grading exits 1, and it should: the CLI never auto-waives anything.

To unblock a legitimate case like this, a human (never CI, never an agent) does the following, outside the CI run:

  1. Confirm the failure is genuinely a known, accepted limitation, not a real defect — check the evaluator's findings.
  2. Record a dated waiver directly in the ticket's acceptance.md, e.g. **Waived:** live-API-only claim, user-approved waiver, 2026-07-11: cannot re-verify without real cost per run.
  3. Hand-edit eval_override: false to eval_override: true in the ticket's ticket.md frontmatter (tkt create already seeds it as false on every new ticket). No tkt command ever sets it to true — it must be flipped by hand. An agent asked to flip it must refuse.
  4. Close the ticket interactively: sprint start <id> then sprint complete — not via tools/sprint-headless, which only grades, never closes.

_gate_eval_report's check here is deliberately coarse: it confirms the flag is set and that acceptance.md records a dated waiver, nothing more. It does not try to verify which specific failing item the waiver covers — a mechanical per-item check was attempted and abandoned after repeatedly failing open in adversarial testing (see DECISIONS.md). That judgment is the human's, made at close time, not the CI run's.

Example GitHub Actions Step

- name: Headless canon grading if: "contains(github.event.pull_request.body, 'Closes: t-')" # or however you detect a CI-eligible ticket run: tools/sprint-headless <ticket-id> --base-ref "${{ github.event.pull_request.base.ref }}" env: ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

Run this only in an isolated, secret-scoped CI environment — claude -p's own permission system genuinely restricts what the headless orchestrator and its subagents can do (--permission-mode dontAsk + a narrow --allowedTools — read/dispatch only, no unrestricted Bash), but an isolated runner is still recommended defense-in-depth, since the agent is processing PR content that could, in principle, be adversarial.

Git goes through tools/git-read (t-5ff5). Both gates grant git only as the running install's tools/git-read, by absolute path, never plain git: git diff/log/show --output=<path> writes any file, so a plain Bash(git diff:*) grant would let text in the graded PR steer a grader into writing files. git-read runs only diff, log, show and status, first (so git's global options such as -c, -C, --git-dir, --work-tree and --exec-path are refused), refuses --out*, --ext-diff, --no-index and --help, refuses any existing path outside the repository (git diff would otherwise switch to no-index mode by itself), and sets GIT_OPTIONAL_LOCKS=0 so status does not rewrite the index. The prompt tells the orchestrator and every subagent to use it. Shell redirection under that grant (git-read diff > f, >> f, | tee f) was probed with a real claude -p 2.1.309 (the CLI version the canon-gate template pins) in dontAsk mode: each was denied and no file was written. A different local CLI version is not covered by that probe.

Generating the workflow from the board

Instead of hand-adding the workflow, click ⚙ CI ("Set up CI gate") in the sprint-check board header — it writes .github/workflows/canon-gate.yml from a shipped template (refusing if the file already exists, so it never clobbers a customized one). That workflow extracts the ticket id from the PR body (Closes: t-xxxx), then dispatches by the ticket's committed gate: sprint-headless-eval for gate: eval, else the full sprint-headless. You still: add an ANTHROPIC_API_KEY repo secret, commit + push the file, and (optionally) require the grade check in branch protection to block merges. The board only writes the file — it can't set GitHub secrets or push.

What the generated workflow trusts (t-89b0):

Consumer-Project CI

The example above only works when the CI checkout is canon's own repo — tools/sprint-headless exists at that relative path because canon's tools/ directory is right there in the checkout.

If your project installed canon via ~/.canon/tools/skills.sh add sprint (per docs/setup.md), your repo has no local tools/ directory of its own — sprint, tkt, and sprint-headless are only reachable via $PATH, pointing at wherever canon was cloned. That works fine on your own machine once you've answered skills.sh add's PATH prompt, but it does not carry over to a CI runner: skills.sh add's PATH offer (tools/skills/prompts.sh's offer_tkt_path) only prompts when it detects an interactive terminal, and silently skips writing to PATH otherwise — a CI runner has no TTY, so it always takes that skip path. A fresh runner starts with neither a local tools/ checkout nor a PATH entry for it, regardless of what's set up on your laptop.

Provision both explicitly in the workflow: clone canon to a known path, then add its tools/ to the runner's PATH before grading.

- name: Checkout canon uses: actions/checkout@v4 with: repository: sunitghub/canon-skills path: canon - name: Add canon tools to PATH run: echo "${{ github.workspace }}/canon/tools" >> "$GITHUB_PATH" - name: Headless canon grading if: "contains(github.event.pull_request.body, 'Closes: t-')" run: sprint-headless <ticket-id> --base-ref "${{ github.event.pull_request.base.ref }}" env: ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

$GITHUB_PATH is GitHub Actions' own PATH-extension mechanism — appending to it persists the addition to every later step in the same job, so sprint-headless resolves bare in the grading step exactly as it would on a developer's machine after skills.sh add finishes locally. Note the grading step invokes sprint-headless, not tools/sprint-headless — with canon's tools/ on PATH rather than vendored into your own repo, the relative path from the canon-repo-only example above doesn't exist here and would fail with "No such file or directory."

Lightweight Spec-Driven Gate (sprint-headless-eval)

tools/sprint-headless-eval is the low-ceremony alternative to the full sprint-headless pipeline. It runs only the evaluator against criteria you write directly in a markdown file — no ticket, no plan approval, no reviewer, no security-review.

Use it when:

Spec File Format

Any markdown file with checklist items:

--- ci: true --- # calculateSplit() must reject split counts above 100 - [ ] Calling calculateSplit() with people > 100 returns a validation error - [ ] The UI rejects people > 100 before calling calculateSplit() - [ ] No unbounded array allocation occurs for large people values

Frontmatter is optional. The body must contain at least one - [ ] or - [x] checklist item — each becomes an independently graded acceptance criterion.

Running It

sprint-headless-eval <ticket-id|spec-file> [--base-ref <ref>] [--model <model>]

Pass a ticket id (t-xxxx) to grade that ticket's .tickets/<id>/acceptance.md and write eval-report.md into the ticket folder — every artifact stays ticket-self-contained, and its ## QA items are not graded (per eval.md, QA is the human's attestation). Or pass any markdown spec file with - [ ] criteria; its eval-report.md is written next to it.

--base-ref (or $GITHUB_BASE_REF) defaults to origin/main (then main, else a clear error) when omitted; an explicit --base-ref and $GITHUB_BASE_REF take precedence, in that order. Relative refs (HEAD~1, main~3, HEAD^) are accepted.

The evaluator grades each criterion with pass/fail/partial/not-run and file:line evidence, writes eval-report.md (in the ticket folder, or next to the spec file), and prints HEADLESS_VERDICT: PASS (exit 0) or HEADLESS_VERDICT: FAIL (exit 1).

To run the evaluator on a specific model (e.g. Haiku to save tokens), pass --model <model> — an alias (haiku/sonnet/opus) or a full model id: sprint-headless-eval t-abcd --model haiku. Alternatively set ANTHROPIC_MODEL in the environment; claude honors either. The model governs the dispatched evaluator subagent, not just the orchestrator.

Demo tickets default to Haiku. In ticket-id mode, if the ticket's frontmatter has demo: true and you pass no --model, sprint-headless-eval defaults the evaluator to Haiku (and prints a notice) — mirroring the interactive close's "Demo mode forces Haiku (evaluator only)". This is a model choice only: the (still-binding) evaluator always runs, so no gate is skipped. Precedence: an explicit --model wins over the demo default, which in turn wins over ANTHROPIC_MODEL/the claude default (it is applied by passing --model haiku). This is scoped to sprint-headless-eval; sprint-headless ignores demo and always runs its full pipeline on the default/Gate model: model. Spec-file mode has no ticket frontmatter, so it never reads demo.

Evaluator tool permissions (--allowed-tools-file)

By default the evaluator subagent runs under a least-privilege allowlist — Agent, read-only git through tools/git-read (diff/log/show/status, above), the subagent-log hook, and Read/Grep/Glob/LS. It deliberately has no general Bash, so it cannot execute code: a criterion whose only proof is running a command grades not-run and (per eval.md's fail-closed rule) forces fail.

To let the evaluator run a load-bearing test, pass --allowed-tools-file <json> — a JSON array of tool strings that replaces the default list (the script always force-adds Agent, the git-read grant and the subagent-log entry so dispatch/audit can't break). The shipped template tools/headless-eval-tools.json is exactly the least-privilege set; copy it and add only what you need:

sprint-headless-eval t-1234 --base-ref origin/main \ --allowed-tools-file my-eval-tools.json

⚠️ Security — grant narrowly. The evaluator runs under --permission-mode dontAsk against an untrusted diff, so a blanket "Bash" grant is a code-execution / prompt-injection vector. Grant the specific runner only, e.g. "Bash(bash <your-test-runner>:*)" — never bare "Bash" or "Bash(bash:*)". Broadening is an explicit, committed, auditable choice (the run prints a Notice: evaluator allowedTools (from …) line); the default stays locked so no consumer incurs this risk unintentionally. A malformed or non-array file fails closed (non-zero exit, no grading).

⚠️ A config that still grants plain git stays exposed until you change it (t-5ff5). An existing file with "Bash(git diff:*)" (or any Bash(git …)) keeps working, so it does not break CI, but it reopens the --output write path; every run prints a Warning: … grants 'Bash(git diff:*)': plain git under dontAsk can write files line. Remove those entries: git-read is force-added.

Scoping the grade to ## Criteria (--criteria-only)

By default sprint-headless-eval grades every - [ ] item — both ## Criteria and ## Test Plan (only ## QA is excluded). Pass --criteria-only to grade only items under ## Criteria, skipping ## Test Plan. This is for callers that execute the Test Plan themselves (e.g. an orchestrator that runs the tests on the host) and want the fresh evaluator scoped to the static criteria — so it stays binding without fail-closing on Test-Plan items it cannot run. With no ## Criteria heading, --criteria-only fails closed with a hint.

Differences from sprint-headless

Item sprint-headless sprint-headless-eval
Gates run reviewer + evaluator + security-review evaluator only
Input ticket ID + committed plan.md/acceptance.md a ticket id (grades its acceptance.md) or any markdown spec file with criteria
Requires tkt ci <id> on, plan approved, ticket committed just the ticket/spec and a git repo
Output review-notes.md + eval-report.md in .tickets/<id>/ eval-report.md in the ticket folder (ticket id) or next to the spec file
Cost ~100k+ tokens (three subagents) ~30-40k tokens (one subagent)

Example GitHub Actions Step

- name: Spec-driven eval run: sprint-headless-eval specs/input-validation.md --base-ref "${{ github.event.pull_request.base.ref }}" env: ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}

When to Use Which