Auto-eval: governed KIT ⇄ EVAL repair loop
Auto-eval turns a failing /eval into a bounded repair loop. The model gets the real failures
of the canonical eval and fixes the code; the acceptance criteria stay locked. A REQ is promotable
only when its own checks pass and it does not break what is already promoted.
/eval REQ-002 --fix
EVAL (sandbox) ── PASS ─────────────────────────────▶ done → /gate REQ-002
│ FAIL
▼
KIT repair (cycle 1/2): failed checks + current candidate files (+ hint)
▼ files written by the extension
EVAL ── PASS ──▶ done
│ FAIL
▼
KIT repair (cycle 2/2) → EVAL ── still failing ──▶ stop, report what fails
How to use it
| What | Command / setting |
|---|---|
| Evaluate and repair | /eval REQ-002 --fix |
| Repair with guidance | /eval REQ-002 --fix "price() must stay in pricing.py; add the discount in checkout" |
| Continue after the loop stopped | run /eval REQ-002 --fix ["hint"] again: it restarts from the current files, with a new cycle budget |
| Repair a failing gate | /gate REQ-002 --fix ["hint"] (see Auto-gate) |
Stop an eval, a gate or a --fix loop |
Cancel in the chat (or cancel_run over MCP): the eval running in the sandbox, the repair agent and the next cycles stop; a cancelled agent’s files are not sent to the orchestrator |
Run it automatically after every /kit |
clike.autoEval.afterKit: true (default false) |
| Number of repair cycles | clike.autoEval.maxCycles (default 2, max 5) |
| Extra cycles while improving | clike.autoEval.extraCyclesOnProgress (default 3): after maxCycles, the loop goes on as long as each cycle leaves fewer failing checks, or fewer failing checks of the REQ itself (the regression of the promoted REQs runs only once those pass: its failures are new, not a step back) |
| Targeted re-eval | after a repair of regression failures, the next eval runs every check of the REQ and the regression of the promoted REQs that failed only (regression_reqs, minutes instead of the whole set); when it passes, a full eval confirms it in the same cycle — taking the results the targeted eval just produced on the same inputs (the REQ’s checks, the regression of those promoted REQs; marked Reused) and running the rest. A targeted eval is never the REQ’s verdict (partial, not promotable, not reused by the gate) |
| Agent re-runs failed checks in the sandbox | clike.autoEval.agentEvalChecks (default 8 runs per repair, 0 = off) |
| Re-check the promoted REQs | clike.eval.regression (default true) |
| Make warnings block the gate | clike.gate.strictWarnings (default false) |
Then promote as usual with /gate REQ-002.
The hint is optional. Without it the repair works from the failures alone, plus CLike’s own
diagnosis (below); with it, both reach the repair. What each cycle received is written next to the
report: runs/eval/<REQ>/repair_<REQ>_<ts>.md (runs/gate/... for /gate --fix), and the chat
shows its path.
What the repair model receives
- Every failed check with its errors, not the tail of its output: the eval runner extracts the
error blocks of the whole output (failed tests, assertions, TypeScript/lint errors, code frames,
the final summary) without ANSI codes, DOM dumps, passing-test lines and library stack frames
(
excerptin the eval report, ≤ 6000 characters per check). - Checks with the same errors (unit and coverage run the same tests) are listed once.
- Non-blocking failures of the regression of other REQs (their coverage) are left out: they do not fail the eval and this REQ cannot fix them. A blocking regression failure (this REQ broke a promoted REQ) is included.
- A test that ran and failed is never classified as environment-blocked (
element(s) not found, a module of the candidate not found): only missing tools, registries, native binaries and sandbox limits are. A missing module that is a file of the project (No module named 'req019_support',Cannot find module 'x'wherexis a project source file, outsidenode_modules/ venvs) is a wrong import or path, any language: it goes to the repair as a failure. A missing dependency stays environment-blocked. - An eval sandbox out of resources stops the loop. When the OS refuses a process, a thread,
memory or disk (
spawn node EAGAIN,pthread_create: Resource temporarily unavailable,ENOMEM,ENOSPC,EMFILE), or the kernel killed processes for lack of memory or held the sandbox at its memory limit while the check ran (read from the cgroup counters: a check under memory pressure fails with time-outs and noENOMEMin its output), the check is environment-blocked and the report saysEVAL_SANDBOX_EXHAUSTED(gate:GATE_BLOCKED_SANDBOX_EXHAUSTED):--fixstops instead of changing correct code, and a local agent’seval-checkanswersBLOCKED. An eval does not start when the sandbox already uses 60% of its process limit. The sandbox runs with an init (init: true) that reaps the processes the checks orphan, and what a check leaves running in its process group (a server, a browser) is stopped when it ends;GET /healthof the sandbox shows the processes in use, the zombies and PID 1. To recover:podman restart docker_eval-sandbox_1. - Cloud: the files named in the errors first, then tests, sources, the eval profile and CI, then
the KIT docs (160k characters). CLike’s own context files (
CLIKE_SELECTED_CAPABILITY_CONTEXT.*,AGENT_PROMPT.md,FILE_REQUIREMENTS.json…) and lockfiles are listed, not shown. - Local agent (Codex, Claude Code): the same errors in its prompt, and it can re-run the failed
checks in the real eval sandbox while it repairs:
node .clike/bin/eval-check.mjs test:unit test:e2e(composed workspace, provisioned variables, services of other REQs), so it iterates until they pass in one session. The extension serves the requests (it holds the token; the agent gets neither the token nor network access), up toclike.autoEval.agentEvalChecksruns. Such a partial run is never the REQ’s verdict: the full eval follows.regression::<REQ-ID>(e.g.regression::REQ-014) re-runs a promoted REQ’s checks with the candidate overlaid, as the regression does: a repair of a regression failure is verified before the full eval. Only the checks the repair was given decide: a failure the eval marks non-blocking (already failing without the candidate, unstable, a warning) printsWARN … not yours to fixand does not make the resultFAIL; when every given check passes the result saysDONE … finish now. On a real project an agent whose two checks passed at its second run went on for 25 more minutes chasing non-blocking coverage checks.
CLike diagnosis: intermittent failures
Before each repair CLike reads the earlier eval and gate reports of the REQ
(runs/eval/<REQ>/report_*.json, runs/gate/<REQ>/report_*.json). Each report stores
inputs_fingerprint, a hash of what the run used: the canonical src/, test/, tests/ and the
KIT (src/, test/, tests/, ci/). Installed dependencies and CLike’s promotion leftovers
(.conflict-…, .incoming-…) are not counted. A failing check that passed earlier is flagged:
-
passed on the same code (same fingerprint): its errors reach the repair behind a note such as
CLike note: test_foreign_records passed in 3 of 5 earlier eval/gate run(s) of this REQ on the same code (src/, test/ and the KIT unchanged), so the failure is nondeterministic (order of rows or jobs by random ids, timing, concurrency, unseeded randomness): make the behavior deterministic in src/, do not adjust the asserted value.
-
passed only on other code (or in a report without a fingerprint): the note says that the code changed since then and that the failure most likely comes from that change, not from nondeterminism.
The note names the test when its id appears as PASSED in an earlier output, else the check. The
chat says ℹ AUTO-EVAL REQ: <check> passed in earlier runs. Why it matters: from a single failing
run (assert 1 == 3) a model tends to change the value; the history is what shows that the code
is not deterministic. The tests stay locked either way.
Auto-gate: /gate --fix
/gate re-runs every check and the regression of the promoted REQs, so it can fail where the
last eval passed (an intermittent test, a change in the canonical tree). /gate REQ --fix ["hint"]
then runs the same loop as /eval --fix, driven by the gate verdict:
/gate REQ-006 --fix
GATE ── PASS ──▶ promotion (auto-merge) → done
│ FAIL (checks, regression, required outputs)
▼
KIT repair (same failures shape, same governance, same CLike diagnosis)
▼
GATE (full) ── PASS ──▶ promotion
│ FAIL → next cycle, or stop (same limits as /eval --fix)
- Same rules as
/eval --fix: tests locked, test/ci changes governed,clike.autoEval.maxCyclesandextraCyclesOnProgress, stop on no progress or a failed KIT request. - A repaired candidate is promoted only by a gate that ran on it: there is no eval in between, the gate runs the same checks and more.
- Missing required outputs (
GATE_BLOCKED_REQUIRED_OUTPUTS_MISSING) reach the repair as failuresrequired-output::<role>. - It stops without repairing on a tampered acceptance surface (
GATE_BLOCKED_ACCEPTANCE_TAMPERED: restore the files, accept a deliberate non-weakening edit with/eval <REQ> --accept "reason", or re-run/kit) and on a manual override. - A gate PASS whose promotion left conflicts ends the loop: the conflict is a merge with the
canonical tree (
<file>.conflict-<REQ>-<ts>), not a defect of the candidate, and a KIT repair cannot resolve it. Resolve it, commit, and run/gate REQagain. - Messages:
↻ AUTO-GATE REQ: repair cycle n/m,✔ AUTO-GATE REQ: gate PASS after n repair cycle(s),✖ AUTO-GATE REQ stopped: <reason>.
Even when the gate passes after a repair of an intermittent failure, look at what the repair
changed (src/ of the KIT): a passing run of a nondeterministic test may still be luck.
Unattended: /autopilot
/autopilot chains, for every REQ of the plan, /kit, /eval --fix and /gate through the same
handlers (any executor), so the loop above runs for each REQ without anyone typing. A REQ whose
loop stops is parked with the reason and the REQs that do not depend on it go on; a provider quota
or rate limit waits and retries. After each gate it checks that the REQ reads done on the branch the
run goes on with: if the merge into the default branch did not happen, the run stops and says how to merge,
and a REQ the run promoted is never built again. An eval interrupted before a verdict runs once more. See
commands.md.
Following a run
The chat’s run card and the status bar follow the loop step by step: each eval, repair and gate with
its outcome and duration, the cycle (cycle 2 of 2 (+3 while failures drop)), an estimate from the
same step’s previous duration. During an eval or a gate the sandbox reports its progress
(GET /v1/eval/progress?req_id=…: setup, checks done of total and the check running, then the
promoted REQs of the regression being re-run), polled every 2 s. After this step ends the loop at
the next step boundary (the eval in flight finishes and keeps its report; no new repair starts);
re-run /eval REQ --fix to go on. Stop now stops the check running in the sandbox, as Cancel.
Time of a cycle
A cycle is: the repair (the agent verifies its fix with eval checks, ~1 minute each), then the full eval. To keep it short and stable:
- the eval right after a repair runs without the local-agent pre-pass: the repair already changed and checked the candidate; a pre-pass changed it again unverified (it once undid a repair that had made the unit tests pass) and cost 3–5 minutes per cycle;
- the regression of the promoted REQs runs only once the REQ’s own blocking checks pass: while
they fail the REQ is not promotable anyway, and the regression (every other REQ’s environment,
services and suites: ~4 of the ~6 minutes of a full eval in a 5-REQ project) would only delay
the loop.
/gateis unaffected: it needs the REQ’s checks to pass, then runs the regression.
When the loop stops
| Outcome | Message | What to do |
|---|---|---|
| Eval passes | ✔ AUTO-EVAL REQ-002: eval PASS after N repair cycle(s) |
/gate REQ-002 |
| Cycles exhausted | still failing after N repair cycle(s) and the failing checks |
read the eval report, re-run with a hint, or fix manually |
| No progress | no progress: the same checks still fail (same checks and same errors; different errors in the same check are progress; numbers, ids and random values such as session tokens printed by the test do not make an error different) |
a hint usually helps; or the failure is environmental |
| Repair request failed | the KIT repair request failed |
see the error in the chat (provider, validation) |
| Acceptance surface changed | the acceptance surface (test/ or ci/) of REQ-002 changed after the lock: ci/LTC.json modified |
tests/LTC of that REQ (the candidate, or a promoted REQ its regression runs) were edited outside the governed path: edited on purpose and nothing weakened → /eval REQ-002 --accept "<why>" (audited); otherwise restore them or re-run /kit |
| REQ not KIT-ready | stopped before a repair: not KIT-ready, the plan waits for your answers: … (the plan marks it kit_ready: false; no hint given) |
answer in the SPEC Note section and run /spec, /plan, /kit; or /eval REQ-002 --fix "<your decisions>" to give the answers to the repair |
| Repair changed a locked test without proof | stopped: the repair says a locked test is wrong … Its change is saved: runs/eval/<REQ>/proposed_test_fix_<ts>.patch (the lock restored the test; the eval showed no error raised in the test itself) |
review the patch: if the test is wrong, /eval REQ-002 --accept "<why>" applies it and records your reason (audited); if the code is wrong, /eval REQ-002 --fix "<hint for the code>" |
| Eval interrupted | ↻ the eval was interrupted before a verdict (…); running it again (timeout, disconnection, the machine asleep, the stack restarting: no check ran) |
nothing: the same eval runs once more in the same cycle; if it breaks again the loop stops with eval failed without failing checks |
| Nothing to repair | eval failed without failing checks |
see the eval report (e.g. integrity or setup errors) |
The same outcomes apply to /gate --fix (AUTO-GATE, gate failed without failing checks).
What the repair may change (governance)
The repair prompt (cloud) and the orchestrator enforce the same rules:
- Source and docs (
runs/kit/<REQ>/src/**,docs/**): free to change. Only the changed files are returned; the others stay as they are. - Tests (
runs/kit/<REQ>/test/**): locked. They are the acceptance criteria taken at the first eval: the repair makes the code pass them. A test file the lock does not hold (new, or added by an earlier repair, e.g. a reconciled test of a promoted REQ) is not locked: it is kept as written, by a cloud model or a local agent alike. Other changes are rejected (repair_change_rejected:...) and never written. Two exceptions, both checked by the orchestrator on the AST and audited:- unused imports (lint failures): a Python test may only drop imports;
- the test itself is wrong: when the eval shows an error raised in the test file that is
not a failed assertion (
TypeError,AttributeError,NameError,ImportError, a missing fixture — e.g. a wrong API call; or a test that contacts a reserved name that never resolves, such asexample.test, its stack in the test file), or a type checker error reported on it (mypytest_a.py:286: error: … [arg-type], pyright, TypeScripterror TS…), the test is fixed, not the code. Every test function, everyassertand everypytest.raises/warns/approxmust stay identical and no skip/xfail may be added (JS/TS tests: every test title andexpectline). When the error is on an assertion line itself, only the API misuse the eval proves may change there, expected values kept: an option or keyword argument the type checker or the runtime reports as unknown on that line is removed (TypeScript'exact' does not exist in type 'ByRoleOptions', mypyUnexpected keyword argument "exact", pyrightNo parameter named, Pythongot an unexpected keyword argument), and a regex-shaped string that a pattern matcher (toMatch,toThrow,toHaveText, …) compared literally — the failure shows the received value matching it as a regex — becomes the same regex literal. The orchestrator applies exactly these corrections to the locked test and requires the repaired one to be identical; anything else is rejected. The fix is reported asrepair_test_fixed_review_required:<file>and the gate showsreview_required: true(VS Code: REVIEW in the gate message) so the developer checks it. - a failed assertion the repair is sure comes from the test (a helper, fixture or setup that
contradicts the SPEC/PLAN, e.g. an HTTPS-only helper on a local HTTP origin): the repair writes
the smallest correction in the REQ’s own test instead of only describing it. The lock restores
the test, CLike saves the change as
runs/eval/<REQ>/proposed_test_fix_<ts>.patchand stops the loop;/eval <REQ> --accept "<why>"applies it and records the reason.
ci/LTC.json: only the command of a check that cannot run may be fixed (wrong path, module or flag). Removing a check, making it non-blocking or weakening it is rejected, with one exception: a check the eval shows needs a container runtime (docker, podman, compose — the sandbox has none) or waits for a person (an observation, a sign-off, a manual verification: the eval output says so) may become"blocking": false; it stays runnable on the developer’s machine, described inci/HOWTO.md. The checks are read in every shape the runner accepts (checks,cases,commands). Accepted changes are recorded as audited amendments of the acceptance lock (audit/acceptance_amendments.jsonlin the CLike state directory) and reported asrepair_amendment_accepted:....- Other
ci/**(e.g.ci/requirements.txt): accepted and audited, so the repair can upgrade vulnerable dependencies. Vulnerable dependencies keep blocking the gate until upgraded. - Environment failures (network, missing system tool) are not “fixed” in code: the model
documents them in
docs/KIT_<REQ>.md. - Local agents: the changes it made to locked files are governed (the rejected ones restored) before the phase commit, so git never holds a change CLike then reverts.
The eval sandbox has Python 3.12, Node 22, Java 21, Chromium’s system libraries and the PostgreSQL
client tools (psql, pg_dump, pg_restore, major 18: they also work with older servers; build
argument PG_CLIENT_MAJOR). It has no container runtime and no privileges.
With a local agent (Claude Code, Codex) the repair follows the same rules:
- the agent package carries the failed checks, the hint and the rules
(
repair_context.auto_eval), since the canonical eval reports stay in the sandbox; - the agent writes directly, so the extension snapshots
test/andci/before the repair and submits what changed toPOST /v1/acceptance/amend(with the previous content). Accepted amendments are audited; rejected changes and deleted files are restored (↺ AUTO-EVAL ... restored).
Follow-up KIT stages (/kit REQ --integrity|--hardener|--promotion-eval) work on the current
candidate: they do not start a new KIT generation, and once the tests are locked (first eval)
their changes to test/ and ci/ go through the same governance as a repair.
Promotability: the acceptance criteria are delivered
Passing checks are not enough: a check can only fail for what it tests. Every KIT writes
runs/kit/<REQ>/docs/ACCEPTANCE.json, one entry per acceptance criterion of the plan:
{"criteria": [
{"id": "REQ-038-A01", "status": "done", "checks": ["tests", "unit"]},
{"id": "REQ-038-A06", "status": "missing", "note": "the adapter probes are not implemented"},
{"id": "REQ-038-A07", "status": "developer_evidence", "note": "a real IT+GB research on the developer machine"}
]}
The eval adds a blocking check acceptance::coverage. It fails when a criterion is partial or
missing, is not in the file, or is done without a blocking check of ci/LTC.json that verifies
it; the repair receives the list and implements what is left. developer_evidence (only for what
the criterion itself says is proved on the developer machine or by a real external service) is
reported, not blocking. Without the file, a SELF_REVIEW.md that declares the KIT incomplete
(“not complete or promotable”, “remains unimplemented”) blocks the same way, and so does a file
that says everything is done while the self review says otherwise. Nothing declared either way
(KITs written before this rule): the checks decide, as before.
Repository-root deliverables
Only runs/kit/<REQ>/root/<repository path> reaches the repository root: scripts/, deploy/ and
the other root folders, .env.<name>.example, .gitignore, ci/SOLUTION_LTC.json, project docs
docs/<runbook>.md (never docs/harper/, docs/inputs/, docs/reference/, a real .env or a
secret). The eval sees root/ at the workspace root. A KIT that keeps such files in a staging
folder of ci/ (ci/shared/scripts/…) or writes a staging map of its own fails the blocking check
structure::root-deliverables, with the move to make; /finalize gets the list of those that
delivered REQs left there (staged_root_deliverables) and brings them to the root.
Promotability: regression of promoted REQs (L2)
With clike.eval.regression (default on), /eval and /gate also run the acceptance checks of
every promoted REQ (status: done in plan.json) with their own LTC commands, against their source with the candidate’s composed source
overlaid on top, i.e. the code as it would be after promotion — with the dependencies after promotion too:
the candidate’s Python requirements are installed on top of the promoted REQ’s, and a promoted REQ’s npm
manifest (package.json + lock, wherever its KIT pins one) is replaced by the candidate’s at the same path when
the candidate’s holds every dependency of it at the same version (a version conflict keeps the promoted one).
These cases appear as
regression::<REQ>::<check>, run in runs/eval/<REQ>/regression/<OTHER>/ (the other REQ’s own
eval results are not touched), and their acceptance locks are verified first.
Only behavioural checks of the promoted REQ run and block (tests, contract/e2e checks, with the
installs and builds they need). Its static checks — lint, format, types, security scans, coverage,
syntax, SBOM/licence — passed at its own gate and would now measure the candidate’s code, which the
candidate’s own checks already cover: they are not run in the regression (one non-blocking
static-checks::not-run case lists them). A test check whose only failure is the coverage threshold
is a warning.
A promoted REQ’s own regression checks (a check named regression, test:backend-regression,
… — they re-run tests of earlier REQs, often copies of them in its KIT) are not run in its
regression either: those earlier REQs are regressed themselves, on their promoted tests
(regression-checks::not-run lists them). A REQ whose only test check is called regression keeps
it. The KIT prompt (cloud and local agents) asks not to copy other REQs’ tests into a KIT nor to add
such checks. The same holds for the candidate’s own eval when it includes CLike’s regression: its
regression checks would re-run tests of earlier REQs that are not in its KIT (“file not found”),
and a locked check cannot be dropped by the repair, which then copied other REQs’ tests into the
KIT; they are not run (regression-checks::not-run), CLike’s regression runs those tests on their
promoted version with the candidate overlaid. An eval without CLike’s regression runs them as declared.
Speed: what is not done twice
- The gate reuses the eval verdict when nothing changed. Every full
/evalrecords its verdict with a digest the orchestrator computes from the project itself (canonicalsrc/,test/, repository-root deliverables, the REQ’s KIT, every LTC,plan.json, CLike’s version), inCLIKE_STATE_DIRwhere agents cannot write./gateon the same inputs with the same regression mode takes that passing verdict instead of running everything again (executor: reused_eval, chat: “nothing changed since the passing eval”); the acceptance integrity and the required outputs are still checked by the gate. Any changed file, LTC or REQ status, a failed verdict, or a verdict older than 24 h: the gate runs the profile as before. - Installed
node_modulesare shared by package.json + lockfile digest (<eval base>/.npm-store/): a fresh work copy of the REQ or of a regression REQ links them (hard links, no extra disk) instead of running npm again. Python venvs were already kept per REQ. - Every check has its duration (
duration_sin the report cases), and the report’stimingssay where the eval spent its time: setup, the REQ’s checks, the regression and each promoted REQ in it, with the workers used. - Promoted REQs are regressed side by side:
CLIKE_EVAL_REGRESSION_WORKERS(docker/.env, defaultauto: about 2 GB and 2 CPUs of the sandbox per worker, at most 4, so a 4 GB sandbox keeps 2 and a 10 GB / 7 CPU one runs 3; or a number,1= one after the other, at most 8) of them at a time, each in its own work dir with its own venv, test databases and service ports; installs they share (a REQ’s venv that another REQ’s service also uses, the npm install store) are created by one at a time, and the report lists the cases in plan order as before. Language-independent: it runs the LTC checks, whatever they are. More workers need more sandbox memory (CLIKE_EVAL_SANDBOX_MEM_LIMIT): with a large frontend count about 1.5 GB per worker; if the sandbox runs out of memory, lower it.
Dependencies that are not promoted yet are not regression targets: their source is already part
of the candidate’s eval (promoted src/ + dependency KITs + current KIT), and a REQ that never
passed has nothing to regress from.
A candidate may carry its own version of a promoted REQ’s test file (a shared test it extends): the
regression runs that version and lists it (regression::<REQ>::reconciled-tests, a warning to
review). It may change or extend the test, never relax it: when an assertion of the delivered version
is removed, loosened (== to <=, toEqual to toContain) or newly skipped in the candidate’s, the
regression blocks (regression::<REQ>::reconciled-tests-assertions) with the lines and the remedy —
make the product satisfy the test, or declare the change (/eval <REQ> --supersede REQ-x#path::name,
or supersedes from /idea --extend), which lifts the block for that file. A changed value or a
longer expected list stays strict: only the warning.
A MODIFIED REQ that changes a default behind a setting and keeps the old behavior selectable (a
new default UI theme, the release 1 one by configuration) declares the setting its regression runs
the promoted REQs’ unchanged checks under: regression_env: {"LG_UI_THEME": "classic"} in plan.json
(/idea --extend writes it; plain settings only — names that look like credentials, CLIKE_*,
PATH-like variables, nested or long values are ignored). Every promoted check still runs and still
blocks: it proves its behavior under the setting it was written for, while the new default is proven
by the declaring REQ’s own checks. Like supersedes, the declaration holds while the REQ is the
candidate and once it is promoted: for the REQs delivered before it (plan order); the declaring REQ and
the ones after it run without it. The report says it (env::regression-env), and it is part of what a
reused regression must match.
These declarations survive a later /plan: when its rewrite of plan.json omits the release fields an
existing REQ had (change, amends, supersedes, regression_env, release, origin, retires,
keeps), CLike puts them back from the plan before the run and says so in the chat
(ℹ /plan dropped release fields of existing REQs; kept from the plan before the run: …). A value the
new plan sets is kept.
A promoted REQ whose environment does not come up in its regression (a setup step fails — a database
not created, dependencies not installed — and its checks are blocked) is run once more before any verdict.
If it comes up the second time the report says it (regression::<REQ>::env::setup-retried) and nothing
goes to the repair; if it fails again the first verdict stands and the usual diagnosis follows. A tool or a
container runtime the sandbox lacks fails the same way every time and is never re-run. Test databases are
created with retries (four attempts, short backoff) when the server is slow under parallel regressions.
Declared changes of a release (/idea --extend) narrow the regression, when the REQ that declares
them is promoted or is the candidate: the checks a MODIFIED REQ supersedes (REQ-x#check) are
not run in the amended REQ’s regression, in whatever section of its LTC they are declared (reported
as regression::<REQ>::<check>::superseded, its other checks still run), and a REQ a REMOVED REQ
retires leaves the regression set. A superseded test (REQ-x#path::name) is recognised in
the output of any check that runs it, whichever promoted REQ the check belongs to: a check whose
failed tests are all superseded holds ([CLike EvalRunner] Superseded tests: …); one with other
failed tests still fails, and the note names the superseded ones so the repair leaves them alone.
The failed tests are read in the formats of the common runners (pytest, vitest/jest, Playwright, Go,
Rust, .NET, Maven/Gradle, RSpec, PHPUnit), and the runner’s summary must not count more failures
than those. A failing delivered test of a REQ the candidate amends, not declared, gets a note with
the exact /eval <REQ> --supersede REQ-x#path::name "<why>" to run if the change is on purpose
(see commands.md): a repair may not change the plan.
A REQ that passes its own tests but breaks a promoted REQ fails the eval; the gate reports
GATE_BLOCKED_REGRESSION with the list in regression_failures. In an auto-eval loop the
regression failures are part of what the repair receives.
A blocking test of a promoted REQ that fails is run once more on the same code. If it passes,
it is nondeterministic (timing, order, asynchronous UI): it is reported as a non-blocking warning
([CLike EvalRunner] Nondeterministic: … make the test of REQ-0xx deterministic), does not block
the gate and is not sent to the candidate’s repair. Failing twice, it is a regression. Unless it is
known unstable: CLike remembers (7 days, any promoted state, in the eval sandbox volume) each
check that failed and then passed on a re-run, with its error lines; failing twice with a later
candidate while it passes on the promoted tree alone, and showing only errors seen in those runs (a
timeout, a missing UI element), it is a warning (… a known unstable check). A failure with an
error it never showed, or with no error line to compare, is a regression.
Promoted REQs are regressed side by side (CLIKE_EVAL_REGRESSION_WORKERS, default auto). A blocking
failure that ran out of time (a test over its limit, a database connection, a suite over the
check’s own timeout, a dev server that did not start in time) runs again alone once the others
are done: passing alone, it was the parallel load, a non-blocking warning ([CLike EvalRunner]
Timeout under parallel load: …) never sent to a repair; failing alone too, it stays blocking and
the note says the slowdown comes with the candidate.
A regression already run on the same workspace is not run again. Each promoted REQ’s regression is
kept, on the sandbox volume, with the digest of its composed workspace (every file its checks can read)
and of what else decides it (the promoted state, CLike and its environment, the declared changes). The
next eval reuses it when the digest is the same and it did not block, marked Reused: … byte-identical
workspace: an eval run again after an interruption resumes, and a repair that changed only what the
promoted REQs do not receive (the candidate’s own tests, CI scripts or docs) skips their regression. A
change to the candidate’s source changes every workspace: everything runs again. A blocking or unstable
result (passed only on a re-run) is never reused, and nothing is reused for longer than a day.
Disk. Before an eval composes its workspaces it checks the free space on the eval volume
(CLIKE_EVAL_MIN_FREE_GB, default 10, and at least 15%). Below it the work directories of the project’s
other evals are removed, least recently used first (reports and the shared caches stay); still below,
the report carries the warning env::disk-space with the remedy, before an install fails unclearly.
A long eval is never cut by CLike’s own wait: the orchestrator waits for the sandbox up to
CLIKE_EVAL_SANDBOX_TIMEOUT_S (default 6 hours, a guard against a lost sandbox); the deadline is the
caller’s (clike.harperTimeout in VS Code), and closing the request stops the run.
A blocking failure of a promoted REQ that remains (a test, a service it starts, a value it
provisions) is then run on the promoted tree without the candidate. If it fails there too, the
candidate did not cause it (a check environment that later REQs outgrew, an earlier REQ): it is a
non-blocking warning ([CLike EvalRunner] Already failing without this candidate: … REQ-0xx's check
or its environment needs a fix), not sent to the candidate’s repair. Unless the candidate adds errors of its
own: the error lines of the run without the candidate are kept, and a failure that shows lines
not seen there (an import error where the promoted tree only missed the coverage threshold) stays
blocking ([CLike EvalRunner] New errors from this candidate: …). Only what the candidate
breaks blocks its gate. When a service of a promoted REQ starts on the promoted tree but not with
the candidate, the failure stays blocking and says why ([CLike EvalRunner] Service broken by the
candidate): a configuration the candidate makes mandatory must stay optional with a safe default,
or be declared with /idea --extend.
That run without the candidate depends only on the promoted tree, so it is done once per
promoted state: its results are kept in the eval base dir (`