Skip to the content.

Auto-eval: governed KIT ⇄ EVAL repair loop

Auto-eval turns a failing /eval into a bounded repair loop. The model gets the real failures of the canonical eval and fixes the code; the acceptance criteria stay locked. A REQ is promotable only when its own checks pass and it does not break what is already promoted.

/eval REQ-002 --fix
  EVAL (sandbox) ── PASS ─────────────────────────────▶ done → /gate REQ-002
     │ FAIL
     ▼
  KIT repair (cycle 1/2): failed checks + current candidate files (+ hint)
     ▼ files written by the extension
  EVAL ── PASS ──▶ done
     │ FAIL
     ▼
  KIT repair (cycle 2/2) → EVAL ── still failing ──▶ stop, report what fails

How to use it

What Command / setting
Evaluate and repair /eval REQ-002 --fix
Repair with guidance /eval REQ-002 --fix "price() must stay in pricing.py; add the discount in checkout"
Continue after the loop stopped run /eval REQ-002 --fix ["hint"] again: it restarts from the current files, with a new cycle budget
Repair a failing gate /gate REQ-002 --fix ["hint"] (see Auto-gate)
Stop an eval, a gate or a --fix loop Cancel in the chat (or cancel_run over MCP): the eval running in the sandbox, the repair agent and the next cycles stop; a cancelled agent’s files are not sent to the orchestrator
Run it automatically after every /kit clike.autoEval.afterKit: true (default false)
Number of repair cycles clike.autoEval.maxCycles (default 2, max 5)
Extra cycles while improving clike.autoEval.extraCyclesOnProgress (default 3): after maxCycles, the loop goes on as long as each cycle leaves fewer failing checks, or fewer failing checks of the REQ itself (the regression of the promoted REQs runs only once those pass: its failures are new, not a step back)
Targeted re-eval after a repair of regression failures, the next eval runs every check of the REQ and the regression of the promoted REQs that failed only (regression_reqs, minutes instead of the whole set); when it passes, a full eval confirms it in the same cycle — taking the results the targeted eval just produced on the same inputs (the REQ’s checks, the regression of those promoted REQs; marked Reused) and running the rest. A targeted eval is never the REQ’s verdict (partial, not promotable, not reused by the gate)
Agent re-runs failed checks in the sandbox clike.autoEval.agentEvalChecks (default 8 runs per repair, 0 = off)
Re-check the promoted REQs clike.eval.regression (default true)
Make warnings block the gate clike.gate.strictWarnings (default false)

Then promote as usual with /gate REQ-002.

The hint is optional. Without it the repair works from the failures alone, plus CLike’s own diagnosis (below); with it, both reach the repair. What each cycle received is written next to the report: runs/eval/<REQ>/repair_<REQ>_<ts>.md (runs/gate/... for /gate --fix), and the chat shows its path.

What the repair model receives

CLike diagnosis: intermittent failures

Before each repair CLike reads the earlier eval and gate reports of the REQ (runs/eval/<REQ>/report_*.json, runs/gate/<REQ>/report_*.json). Each report stores inputs_fingerprint, a hash of what the run used: the canonical src/, test/, tests/ and the KIT (src/, test/, tests/, ci/). Installed dependencies and CLike’s promotion leftovers (.conflict-…, .incoming-…) are not counted. A failing check that passed earlier is flagged:

The note names the test when its id appears as PASSED in an earlier output, else the check. The chat says ℹ AUTO-EVAL REQ: <check> passed in earlier runs. Why it matters: from a single failing run (assert 1 == 3) a model tends to change the value; the history is what shows that the code is not deterministic. The tests stay locked either way.

Auto-gate: /gate --fix

/gate re-runs every check and the regression of the promoted REQs, so it can fail where the last eval passed (an intermittent test, a change in the canonical tree). /gate REQ --fix ["hint"] then runs the same loop as /eval --fix, driven by the gate verdict:

/gate REQ-006 --fix
  GATE ── PASS ──▶ promotion (auto-merge) → done
    │ FAIL (checks, regression, required outputs)
    ▼
  KIT repair (same failures shape, same governance, same CLike diagnosis)
    ▼
  GATE (full) ── PASS ──▶ promotion
    │ FAIL → next cycle, or stop (same limits as /eval --fix)

Even when the gate passes after a repair of an intermittent failure, look at what the repair changed (src/ of the KIT): a passing run of a nondeterministic test may still be luck.

Unattended: /autopilot

/autopilot chains, for every REQ of the plan, /kit, /eval --fix and /gate through the same handlers (any executor), so the loop above runs for each REQ without anyone typing. A REQ whose loop stops is parked with the reason and the REQs that do not depend on it go on; a provider quota or rate limit waits and retries. After each gate it checks that the REQ reads done on the branch the run goes on with: if the merge into the default branch did not happen, the run stops and says how to merge, and a REQ the run promoted is never built again. An eval interrupted before a verdict runs once more. See commands.md.

Following a run

The chat’s run card and the status bar follow the loop step by step: each eval, repair and gate with its outcome and duration, the cycle (cycle 2 of 2 (+3 while failures drop)), an estimate from the same step’s previous duration. During an eval or a gate the sandbox reports its progress (GET /v1/eval/progress?req_id=…: setup, checks done of total and the check running, then the promoted REQs of the regression being re-run), polled every 2 s. After this step ends the loop at the next step boundary (the eval in flight finishes and keeps its report; no new repair starts); re-run /eval REQ --fix to go on. Stop now stops the check running in the sandbox, as Cancel.

Time of a cycle

A cycle is: the repair (the agent verifies its fix with eval checks, ~1 minute each), then the full eval. To keep it short and stable:

When the loop stops

Outcome Message What to do
Eval passes ✔ AUTO-EVAL REQ-002: eval PASS after N repair cycle(s) /gate REQ-002
Cycles exhausted still failing after N repair cycle(s) and the failing checks read the eval report, re-run with a hint, or fix manually
No progress no progress: the same checks still fail (same checks and same errors; different errors in the same check are progress; numbers, ids and random values such as session tokens printed by the test do not make an error different) a hint usually helps; or the failure is environmental
Repair request failed the KIT repair request failed see the error in the chat (provider, validation)
Acceptance surface changed the acceptance surface (test/ or ci/) of REQ-002 changed after the lock: ci/LTC.json modified tests/LTC of that REQ (the candidate, or a promoted REQ its regression runs) were edited outside the governed path: edited on purpose and nothing weakened → /eval REQ-002 --accept "<why>" (audited); otherwise restore them or re-run /kit
REQ not KIT-ready stopped before a repair: not KIT-ready, the plan waits for your answers: … (the plan marks it kit_ready: false; no hint given) answer in the SPEC Note section and run /spec, /plan, /kit; or /eval REQ-002 --fix "<your decisions>" to give the answers to the repair
Repair changed a locked test without proof stopped: the repair says a locked test is wrong … Its change is saved: runs/eval/<REQ>/proposed_test_fix_<ts>.patch (the lock restored the test; the eval showed no error raised in the test itself) review the patch: if the test is wrong, /eval REQ-002 --accept "<why>" applies it and records your reason (audited); if the code is wrong, /eval REQ-002 --fix "<hint for the code>"
Eval interrupted ↻ the eval was interrupted before a verdict (…); running it again (timeout, disconnection, the machine asleep, the stack restarting: no check ran) nothing: the same eval runs once more in the same cycle; if it breaks again the loop stops with eval failed without failing checks
Nothing to repair eval failed without failing checks see the eval report (e.g. integrity or setup errors)

The same outcomes apply to /gate --fix (AUTO-GATE, gate failed without failing checks).

What the repair may change (governance)

The repair prompt (cloud) and the orchestrator enforce the same rules:

The eval sandbox has Python 3.12, Node 22, Java 21, Chromium’s system libraries and the PostgreSQL client tools (psql, pg_dump, pg_restore, major 18: they also work with older servers; build argument PG_CLIENT_MAJOR). It has no container runtime and no privileges.

With a local agent (Claude Code, Codex) the repair follows the same rules:

Follow-up KIT stages (/kit REQ --integrity|--hardener|--promotion-eval) work on the current candidate: they do not start a new KIT generation, and once the tests are locked (first eval) their changes to test/ and ci/ go through the same governance as a repair.

Promotability: the acceptance criteria are delivered

Passing checks are not enough: a check can only fail for what it tests. Every KIT writes runs/kit/<REQ>/docs/ACCEPTANCE.json, one entry per acceptance criterion of the plan:

{"criteria": [
  {"id": "REQ-038-A01", "status": "done", "checks": ["tests", "unit"]},
  {"id": "REQ-038-A06", "status": "missing", "note": "the adapter probes are not implemented"},
  {"id": "REQ-038-A07", "status": "developer_evidence", "note": "a real IT+GB research on the developer machine"}
]}

The eval adds a blocking check acceptance::coverage. It fails when a criterion is partial or missing, is not in the file, or is done without a blocking check of ci/LTC.json that verifies it; the repair receives the list and implements what is left. developer_evidence (only for what the criterion itself says is proved on the developer machine or by a real external service) is reported, not blocking. Without the file, a SELF_REVIEW.md that declares the KIT incomplete (“not complete or promotable”, “remains unimplemented”) blocks the same way, and so does a file that says everything is done while the self review says otherwise. Nothing declared either way (KITs written before this rule): the checks decide, as before.

Repository-root deliverables

Only runs/kit/<REQ>/root/<repository path> reaches the repository root: scripts/, deploy/ and the other root folders, .env.<name>.example, .gitignore, ci/SOLUTION_LTC.json, project docs docs/<runbook>.md (never docs/harper/, docs/inputs/, docs/reference/, a real .env or a secret). The eval sees root/ at the workspace root. A KIT that keeps such files in a staging folder of ci/ (ci/shared/scripts/…) or writes a staging map of its own fails the blocking check structure::root-deliverables, with the move to make; /finalize gets the list of those that delivered REQs left there (staged_root_deliverables) and brings them to the root.

Promotability: regression of promoted REQs (L2)

With clike.eval.regression (default on), /eval and /gate also run the acceptance checks of every promoted REQ (status: done in plan.json) with their own LTC commands, against their source with the candidate’s composed source overlaid on top, i.e. the code as it would be after promotion — with the dependencies after promotion too: the candidate’s Python requirements are installed on top of the promoted REQ’s, and a promoted REQ’s npm manifest (package.json + lock, wherever its KIT pins one) is replaced by the candidate’s at the same path when the candidate’s holds every dependency of it at the same version (a version conflict keeps the promoted one). These cases appear as regression::<REQ>::<check>, run in runs/eval/<REQ>/regression/<OTHER>/ (the other REQ’s own eval results are not touched), and their acceptance locks are verified first.

Only behavioural checks of the promoted REQ run and block (tests, contract/e2e checks, with the installs and builds they need). Its static checks — lint, format, types, security scans, coverage, syntax, SBOM/licence — passed at its own gate and would now measure the candidate’s code, which the candidate’s own checks already cover: they are not run in the regression (one non-blocking static-checks::not-run case lists them). A test check whose only failure is the coverage threshold is a warning.

A promoted REQ’s own regression checks (a check named regression, test:backend-regression, … — they re-run tests of earlier REQs, often copies of them in its KIT) are not run in its regression either: those earlier REQs are regressed themselves, on their promoted tests (regression-checks::not-run lists them). A REQ whose only test check is called regression keeps it. The KIT prompt (cloud and local agents) asks not to copy other REQs’ tests into a KIT nor to add such checks. The same holds for the candidate’s own eval when it includes CLike’s regression: its regression checks would re-run tests of earlier REQs that are not in its KIT (“file not found”), and a locked check cannot be dropped by the repair, which then copied other REQs’ tests into the KIT; they are not run (regression-checks::not-run), CLike’s regression runs those tests on their promoted version with the candidate overlaid. An eval without CLike’s regression runs them as declared.

Speed: what is not done twice

Dependencies that are not promoted yet are not regression targets: their source is already part of the candidate’s eval (promoted src/ + dependency KITs + current KIT), and a REQ that never passed has nothing to regress from.

A candidate may carry its own version of a promoted REQ’s test file (a shared test it extends): the regression runs that version and lists it (regression::<REQ>::reconciled-tests, a warning to review). It may change or extend the test, never relax it: when an assertion of the delivered version is removed, loosened (== to <=, toEqual to toContain) or newly skipped in the candidate’s, the regression blocks (regression::<REQ>::reconciled-tests-assertions) with the lines and the remedy — make the product satisfy the test, or declare the change (/eval <REQ> --supersede REQ-x#path::name, or supersedes from /idea --extend), which lifts the block for that file. A changed value or a longer expected list stays strict: only the warning.

A MODIFIED REQ that changes a default behind a setting and keeps the old behavior selectable (a new default UI theme, the release 1 one by configuration) declares the setting its regression runs the promoted REQs’ unchanged checks under: regression_env: {"LG_UI_THEME": "classic"} in plan.json (/idea --extend writes it; plain settings only — names that look like credentials, CLIKE_*, PATH-like variables, nested or long values are ignored). Every promoted check still runs and still blocks: it proves its behavior under the setting it was written for, while the new default is proven by the declaring REQ’s own checks. Like supersedes, the declaration holds while the REQ is the candidate and once it is promoted: for the REQs delivered before it (plan order); the declaring REQ and the ones after it run without it. The report says it (env::regression-env), and it is part of what a reused regression must match.

These declarations survive a later /plan: when its rewrite of plan.json omits the release fields an existing REQ had (change, amends, supersedes, regression_env, release, origin, retires, keeps), CLike puts them back from the plan before the run and says so in the chat (ℹ /plan dropped release fields of existing REQs; kept from the plan before the run: …). A value the new plan sets is kept.

A promoted REQ whose environment does not come up in its regression (a setup step fails — a database not created, dependencies not installed — and its checks are blocked) is run once more before any verdict. If it comes up the second time the report says it (regression::<REQ>::env::setup-retried) and nothing goes to the repair; if it fails again the first verdict stands and the usual diagnosis follows. A tool or a container runtime the sandbox lacks fails the same way every time and is never re-run. Test databases are created with retries (four attempts, short backoff) when the server is slow under parallel regressions.

Declared changes of a release (/idea --extend) narrow the regression, when the REQ that declares them is promoted or is the candidate: the checks a MODIFIED REQ supersedes (REQ-x#check) are not run in the amended REQ’s regression, in whatever section of its LTC they are declared (reported as regression::<REQ>::<check>::superseded, its other checks still run), and a REQ a REMOVED REQ retires leaves the regression set. A superseded test (REQ-x#path::name) is recognised in the output of any check that runs it, whichever promoted REQ the check belongs to: a check whose failed tests are all superseded holds ([CLike EvalRunner] Superseded tests: …); one with other failed tests still fails, and the note names the superseded ones so the repair leaves them alone. The failed tests are read in the formats of the common runners (pytest, vitest/jest, Playwright, Go, Rust, .NET, Maven/Gradle, RSpec, PHPUnit), and the runner’s summary must not count more failures than those. A failing delivered test of a REQ the candidate amends, not declared, gets a note with the exact /eval <REQ> --supersede REQ-x#path::name "<why>" to run if the change is on purpose (see commands.md): a repair may not change the plan.

A REQ that passes its own tests but breaks a promoted REQ fails the eval; the gate reports GATE_BLOCKED_REGRESSION with the list in regression_failures. In an auto-eval loop the regression failures are part of what the repair receives.

A blocking test of a promoted REQ that fails is run once more on the same code. If it passes, it is nondeterministic (timing, order, asynchronous UI): it is reported as a non-blocking warning ([CLike EvalRunner] Nondeterministic: … make the test of REQ-0xx deterministic), does not block the gate and is not sent to the candidate’s repair. Failing twice, it is a regression. Unless it is known unstable: CLike remembers (7 days, any promoted state, in the eval sandbox volume) each check that failed and then passed on a re-run, with its error lines; failing twice with a later candidate while it passes on the promoted tree alone, and showing only errors seen in those runs (a timeout, a missing UI element), it is a warning (… a known unstable check). A failure with an error it never showed, or with no error line to compare, is a regression.

Promoted REQs are regressed side by side (CLIKE_EVAL_REGRESSION_WORKERS, default auto). A blocking failure that ran out of time (a test over its limit, a database connection, a suite over the check’s own timeout, a dev server that did not start in time) runs again alone once the others are done: passing alone, it was the parallel load, a non-blocking warning ([CLike EvalRunner] Timeout under parallel load: …) never sent to a repair; failing alone too, it stays blocking and the note says the slowdown comes with the candidate.

A regression already run on the same workspace is not run again. Each promoted REQ’s regression is kept, on the sandbox volume, with the digest of its composed workspace (every file its checks can read) and of what else decides it (the promoted state, CLike and its environment, the declared changes). The next eval reuses it when the digest is the same and it did not block, marked Reused: … byte-identical workspace: an eval run again after an interruption resumes, and a repair that changed only what the promoted REQs do not receive (the candidate’s own tests, CI scripts or docs) skips their regression. A change to the candidate’s source changes every workspace: everything runs again. A blocking or unstable result (passed only on a re-run) is never reused, and nothing is reused for longer than a day.

Disk. Before an eval composes its workspaces it checks the free space on the eval volume (CLIKE_EVAL_MIN_FREE_GB, default 10, and at least 15%). Below it the work directories of the project’s other evals are removed, least recently used first (reports and the shared caches stay); still below, the report carries the warning env::disk-space with the remedy, before an install fails unclearly.

A long eval is never cut by CLike’s own wait: the orchestrator waits for the sandbox up to CLIKE_EVAL_SANDBOX_TIMEOUT_S (default 6 hours, a guard against a lost sandbox); the deadline is the caller’s (clike.harperTimeout in VS Code), and closing the request stops the run.

A blocking failure of a promoted REQ that remains (a test, a service it starts, a value it provisions) is then run on the promoted tree without the candidate. If it fails there too, the candidate did not cause it (a check environment that later REQs outgrew, an earlier REQ): it is a non-blocking warning ([CLike EvalRunner] Already failing without this candidate: … REQ-0xx's check or its environment needs a fix), not sent to the candidate’s repair. Unless the candidate adds errors of its own: the error lines of the run without the candidate are kept, and a failure that shows lines not seen there (an import error where the promoted tree only missed the coverage threshold) stays blocking ([CLike EvalRunner] New errors from this candidate: …). Only what the candidate breaks blocks its gate. When a service of a promoted REQ starts on the promoted tree but not with the candidate, the failure stays blocking and says why ([CLike EvalRunner] Service broken by the candidate): a configuration the candidate makes mandatory must stay optional with a safe default, or be declared with /idea --extend.

That run without the candidate depends only on the promoted tree, so it is done once per promoted state: its results are kept in the eval base dir (`/.regression-baseline/

.json`) with a digest of the canonical tree, the promoted REQs' KITs, every LTC, `plan.json`, CLike's version and the eval environment, for at most 24 h. The next repair cycle, the gate and the next REQ's evals on the same tree reuse it: a check known to fail there is neither retried nor run again, one known to pass there is not run again either. Any change of the promoted tree computes it again. The candidate's own KIT is not part of the digest. A check that failed there on a recent promoted state (7 days) and passes there now, while it fails with the candidate and on its re-run, is **unstable**: a non-blocking warning (`[CLike EvalRunner] Nondeterministic: … an unstable check`), since its result proves nothing about the candidate. The results are kept only when the eval base dir is outside the project (the eval sandbox volume): a project directory is writable by local agents, which must not be able to turn a regression into a warning; there every eval runs it again. When a check of a promoted REQ fails and the candidate KIT replaces files that REQ also delivered (its `src/` or `test/` files, compared with the canonical copy), CLike names them on the failing check, with a short diff: `[CLike EvalRunner] Shared files changed: …`. It works on files, so in any language: a file the candidate **adds** in a test directory of that REQ or one of its parents (not a test of its own: a package marker such as `__init__.py`, a conftest, a runner config: it changes how those tests are collected and import their helpers), a test runner config whose selectors no longer match the earlier tests ("No tests found"), a compose file or manifest an earlier check pins, a component whose behaviour changed. The repair is told to keep what the promoted REQ put there and add its own part, or to put it in a separate file (a compose override, an extra test config or project) when an earlier check pins the content; the repair notes list the replaced files. The KIT rules say the same up front: test runner configs, infrastructure and build manifests are shared files (test selectors never depend on which test files exist: after promotion they all do), and checks assert what their REQ contributes, never that a shared file contains only that. In the regression a promoted REQ runs the promoted version of its tests: each test file of its KIT is replaced by its copy in the canonical test tree when that copy differs and was committed later (a later REQ reconciled it and promotion delivered it), so a reconciliation is done once, not by every following REQ. The promoted version gets the support files it imports from the canonical test tree (a helper at the test root, a package marker beside it), with the same rule as a reconciled test. The test root's `conftest.py` is shared by every REQ and merged at promotion, so the REQ keeps its own and the canonical one is not brought. A service of another promoted REQ (`services[].req` in the LTC) runs that REQ's promoted code with the candidate's source over it, as in its regression, not the original source of its KIT. When the candidate changes a promoted REQ's behaviour on purpose, it reconciles that REQ's test in its own test root, at the same path (`runs/kit//test/...`). The regression runs that version, since promotion delivers it: the candidate's copy replaces the test files the promoted REQ already has, and the candidate's new support files that version needs come with it: the modules it imports, directly or through another copied helper, from anywhere in the candidate's test tree (a helper at the test root for a test in `test/v2/`; for JS/TS its relative imports, `from './v2/x.fixtures'`, `require('../y')`, `import('./z')`, with or without extension), and a package marker (`__init__.py`) or a `conftest` in the same directory as a reconciled test. Nothing else comes: a `conftest` in another directory would be loaded by the test runner into that REQ's own tests, a module that exists in the sources is never taken from the test tree, and the candidate's tests run in its own checks. A candidate file identical to the promoted one reconciles nothing (promotion does not change it) and is not overlaid. The candidate's own tests are not added to that REQ's checks. The change is never silent: the report carries the warning `regression::::reconciled-tests` with the replaced files, for review before the gate. It is not sent to a repair. The eval workspace is composed from the KIT and the canonical tree without installs, caches and scratch (`node_modules`, `.tmp`, `.cache`, `.npm-cache`, `.next`, `local-eval-workspaces`…); a file removed while it is copied is skipped. If CLike still cannot compose it, the preparation is retried once, then `env::prepare-workspace` is environment-blocked: no check ran, `--fix` stops instead of sending it to a repair. A promoted dependency is already in the canonical tree, merged with what later REQs changed, so its KIT copy must not bring old files back. With a gate auto-merge record (`runs/kit//promotion/*/report.json`) its KIT is not overlaid at all. A REQ that is done without that record (promoted before the records existed, by an overwrite, or kept beside the tree by a folder/suffix strategy) is overlaid file by file: a KIT file wins unless its canonical copy differs and was committed later (git history of the project); without history the KIT wins, as before. This also applies to a promoted REQ's own run in the regression. Repository-root deliverables of a KIT (`runs/kit//root/`: `scripts/`, `deploy/`, root manifests) are at the eval workspace root, after the canonical ones and those of dependency KITs not promoted yet, so a check runs `./scripts/…` as it will after promotion; in the regression the candidate's root files are overlaid too, and a promoted check failing on one names it. A KIT's root `test/conftest.py` (or `tests/conftest.py`) is promoted over the canonical one that every delivered REQ shares: the eval fails it (`structure::shared-conftest::`, blocking) when it drops a fixture or helper the canonical one defines, so the repair keeps them or moves the REQ's fixtures into its own package conftest — instead of a promotion conflict after a passing gate. Removing that root conftest is the one deletion the acceptance governance accepts: a test-support file that defines no test and that the eval named is taken out of the lock (audited); a deleted test file is always restored. The same holds for every other **shared test-support file**, in any language CLike reads: a KIT that replaces a helper or fixture module of the promoted tree (not a test: `test/fixtures.ts`, a `*_support.py`, a `TestBase.java`, a Go helper) must keep the public names the promoted REQs' tests use — functions, classes and module names in Python; `export`s in JS/TS; public members in Java/C#; declarations in Kotlin; exported names in Go. A dropped one fails the eval (`structure::shared-test-support::/`, blocking) with the names and the tests that use them, before the regression would fail late (a suite that does not load, an e2e that times out). A name counts as used when a test names it and refers to the module (or sits in the same directory, as Go and Java packages do), in the version after promotion: a test the candidate reconciled is read in its version. A changed behaviour behind the same names is the regression's to find; a file in a language CLike does not read is left to the regression as before. A KIT also must not **add a test-layout file** in a test directory of the promoted tree (or one of its parents): a file that changes how the tests already there are collected or import their helpers — a package marker (`__init__.py`), a `conftest.py`, a pytest / Jest / Vitest / Playwright / Mocha / Babel / TypeScript config, a `go.mod`, `pom.xml`, `build.gradle`, `.csproj`. The eval fails (`structure::test-layout::`, blocking, no run needed) with the files, at the first eval: the regression would show it only once the REQ's own checks pass. The REQ's own tests, helpers and directories are not concerned. ## Required outputs in the eval Required outputs missing from the candidate (FILE_REQUIREMENTS, e.g. a runnable launcher) block the gate (`GATE_BLOCKED_REQUIRED_OUTPUTS_MISSING`). The eval reports them too, as failed checks `structure::` with what is accepted, so the auto-eval repair can add them. A launcher is a conventional entry file (`main.py`, `app.py`, `__main__.py`, `server.js`, `Program.cs`, …) or a source file that starts the application (`if __name__ == "__main__"`, `FastAPI(...)`, `def create_app(...)`, `.listen(...)`). ## Browser (e2e) checks The sandbox image carries Chromium's system libraries. When a Playwright check fails because the browser is not downloaded, the EvalRunner downloads Chromium in the version of the project's Playwright (`npx playwright@ install chromium`) and runs the check once more; the report says so. The download needs egress and memory (see `docs/development.md`). ## Native tools of the sandbox (`/tools`) **Language runtimes.** The sandbox image has Python 3.12 and Node 22. An LTC that declares a newer version in `"environment"` — `"python": "3.14"`, `"node": ">=24"` — gets it from the catalog (`python3.13`, `python3.14`, `nodejs24`) without a question, for that REQ's checks only: its venv is created with the declared Python, `uv` uses it (`UV_PYTHON`, nothing downloaded) in a project environment of the REQ, and the declared Node comes first on its PATH. A version the image already satisfies installs nothing. Every REQ gets a writable `uv` project environment of its own, whatever it declares (the image's default pointed at CLike's own environment, so a project's `uv sync` failed). The image has Python, Node, a Java runtime, Chromium's libraries and the PostgreSQL client tools. Any other native program a check runs — a database or messaging client (`mysqldump`, `mongodump`, `kcat`), a cloud or IaC CLI (`aws`, `az`, `gcloud`, `kubectl`, `helm`, `terraform`), a toolchain (`jdk@17`, `mvn`, `gradle`, `dotnet`, `go`) — comes from CLike's **tools catalog** (`orchestrator/eval_tools_catalog.yaml`, conda-forge, linux-64 and linux-aarch64), installed into a volume the sandbox mounts read-only and appends to the PATH of the checks (never before the image's own programs). CLike finds what an eval needs **before** running it: - *declared*: `"environment": {"tools": ["kcat", "jdk@17"]}` in `ci/LTC.json` (the KIT rules ask for it); - *used*: a catalog program the KIT's code, tests or scripts run (`"pg_dump"`, `["kcat", …]`, a script line `aws s3 ls`); - *failed*: a check that stopped on a missing program (`kcat: command not found`, `spawn kcat ENOENT`, `No such file or directory: 'kcat'`, `Cannot run program "mvn"`), or whose own wording says a catalog program is missing (`shellcheck missing`, `aws is not installed`, `requires aws`) when that program is indeed absent from the sandbox: it is environment-blocked (`missing_tools` in the report), never sent to a repair. The script linters `shellcheck` and `powershell` (`pwsh`) are default tools: the eval-tools service installs them at its start, for every project, without asking (`CLIKE_EVAL_TOOLS_DEFAULTS=0` turns this off). What else is missing is installed after the developer's confirmation (VS Code: *Install*, *Always for this project*, *No*; `clike.evalTools.install`: `ask` | `always` | `never`); tools listed in `.clike/eval-tools.yaml` (`tools:`) install without asking, so a team shares the same tools. A check that stopped on a missing tool runs again once it is installed; without an answer within 5 minutes the eval runs without it, and `--fix` stops saying how to install it. A project can add other conda-forge packages under `custom:` in the same file. Libraries of a language (psycopg, ojdbc, kafka-python…) are not tools: the project's manifests install them. `/tools` lists the catalog and what is installed with versions, `/tools search ` finds an entry, `/tools add [@version]…` installs and approves for the project, `/tools update [tool…]` rebuilds them at the latest versions and shows what changed, `/tools remove ` withdraws the approval. How the tools are installed (the `eval-tools` service: only the orchestrator reaches it, its own network, no credentials): - **One environment per tool** (`/opt/clike-tools/envs/`): installing Maven never changes the `jdk@17` already there; micromamba's package cache keeps the copies small. - **Three kinds of catalog entry**: conda-forge `packages`; `pip` for CLIs distributed on PyPI (e.g. `azure-cli`, whose conda-forge package does not resolve), installed in the tool's own Python; `download` for a vendor archive — an `https` URL per platform (`linux-64`, `linux-aarch64`) with its **sha256**, verified before it is unpacked (tar/zip, no paths outside the tool directory), never executed by CLike. A project adds its own entries of any kind under `custom:` in `.clike/eval-tools.yaml`. - **Versions are locked.** Each tool records its top-level versions and an exact lock (micromamba explicit spec with md5, `pip freeze`, archive digest); the orchestrator keeps a copy outside the volume (`/eval-tools/lock.json`). On a new volume `docker/rebuild.sh` rebuilds the same tools with the same versions (`restore`). - **Updates are deliberate.** A rebuild does not change versions, so two gates on the same code give the same result. `/tools update` or `./rebuild.sh --update-tools` rebuilds every tool at its latest version (the image's PostgreSQL client too) and prints `old → new`; a tool whose rebuild fails keeps its previous version. `/tools` and the rebuild remind you when the tools are older than 30 days (`CLIKE_EVAL_TOOLS_STALE_DAYS`). - **Evidence.** Every eval and gate report records the versions of the installed tools (`tools`), next to `missing_tools`. Verified on conda-forge for both architectures (dry-run resolution, 2026-10): every catalog entry except `kafkacat` on `linux-aarch64` (not published there). ## Eval disk Eval directories (composed workspaces, `node_modules`, venvs, npm store) are caches on the sandbox volume: CLike rebuilds them when needed. Before an eval CLike: 1. keeps the directories of the 3 REQs of the project used last (`CLIKE_EVAL_KEEP_REQS`); 2. removes whole, caches included, the projects not evaluated for 7 days (`CLIKE_EVAL_PROJECT_RETENTION_DAYS`, 0 keeps them); 3. when the volume has less than 10 GB free (`CLIKE_EVAL_MIN_FREE_GB`), removes the least recently used REQ directories of the other projects, then the other projects whole; the current project always stays. Below 2 GB free the eval does not start (`EVAL_SANDBOX_EXHAUSTED`): the REQ is parked with the reason, no repair runs. The volume shares the Podman machine disk with the container images, which CLike does not remove: `docker/rebuild.sh` warns when that disk is more than 80% full (`CLIKE_DISK_WARN_PERCENT`) and prints what can be freed. To recover: ```bash podman system df # images, containers, volumes and what is reclaimable podman image prune -a # unused images (old builds, stopped projects) podman machine stop && podman machine set --disk-size 200 && podman machine start # or enlarge it ``` then `/eval REQ-NNN --fix` on the parked REQ and `/autopilot`. The npm cache is one per project, shared by the REQ, its regression and its services. ## Gate warnings policy A *warning* is a failed check marked `blocking: false` in the LTC (style, optional scans). - Default: warnings are reported and the gate passes with `reason_code: GATE_PASS_WITH_WARNINGS`. - Strict: `clike.gate.strictWarnings: true` (per workspace) or `CLIKE_GATE_STRICT_WARNINGS=1` (orchestrator, for all projects) → `GATE_BLOCKED_WARNINGS_PRESENT`. ## API | Endpoint | Field | Meaning | |---|---|---| | `POST /v1/eval/run`, `POST /v1/gate/check` | `regression: true` (body) | add the regression stage | | `POST /v1/gate/check` | `strict: true` (body) | warnings block the gate | | `POST /v1/acceptance/amend` | `req_id`, `changes`, `previous`, `evidence` | govern the acceptance files a local agent changed: `{accepted, rejected, test_fixes, audit_id}` | | `POST /v1/harper/run` (phase `kit`) | `kit.repair: {cycle, max_cycles, failures, files, hint}` | KIT repair request; `failures[]` = `{name, code, command, output}`, `files[]` = `{path, content}` | `kit.repair: true` keeps its previous meaning (BMAD developer repair pass, `/kit REQ --repair`). Gate responses add `regression`, `regression_failures` and `strict_warnings`. ## Measuring it The benchmark runs the same loop without the extension: ```bash python benchmark/run_benchmark.py --runner cloud --projects pingboard --auto-eval 2 ``` The summary has a *Repair cycles* column; see `benchmark/README.md`. ## The whole solution (`/finalize`) The solution started as a whole is verified at `/finalize` (see `commands.md`): CLike installs, builds, configures and starts every component together in the sandbox and smoke-checks it while it runs. It does not run the REQs' tests again: the regression of each gate already did.