Repository reports and proof runs
Read a repository’s setup report, verify its commands, choose where to run them and require evidence for a story.
Before handing an agent a change, check whether a clean checkout builds and tests. The repository report identifies missing setup. A proof run executes the commands and records the result, the commit and where the code ran.
Read the repository report
- Connect the repository to your workflow. Open Dashboard → Repository report, or choose Repository report on the factory’s Getting started screen. Select the repository by its full name. Connect a repository
- Read the grade, commit and time. The five levels are Readable, Runnable, Checked, Proven and Guarded. Expand Show what passes and inspect What is missing and Could not check. For example, the report can find a test command in a manifest even if nobody has run it. A host permission may be needed to read branch protection.
- Use Run again to rescan after a setup or permission change. Open History → View to compare a previous report. Run the dry run asks for confirmation before executing the repository recipe on the default branch.
- If Fix it is available, a workspace owner or admin can inspect the proposed changes, then choose Open the pull request. CodeBook opens one draft in the repository. Depending on the findings it may add repository instructions, scripts, a starter test or ignore rules. A suggested CI workflow appears in the PR description for a human to add. Review the actual diff and verify the results after merge and sync.
Verify the repository recipe
Open Factory settings → Proof runs → Repository recipe. A recipe belongs to one connected repository. A proposed recipe shows the files CodeBook read. Choose Read it from the repository again after changing the repository’s setup and review the new proposal.
- Review Install, Build, Test, Lint and Type check. Each field takes one command line run from the repository folder. Use the same pinned dependency installation and scripts your maintainers use. Leave inapplicable fields empty.
- For checks that need a running app, supply Start the app and its Health URL, such as
http://localhost:3000/health. Supply Seed if the test environment needs initial data. The health URL is reached from the machine that started the app. - Save edits. Under Where to try the recipe, select an available location and choose Try it. This exercises the configured build, lint, type-check and test commands on the default branch. The runner and hosted sandbox use the install line. A local dry run uses your already-installed checkout.
- Read each command’s exit code and available test counts. Proposed means inferred from the repository. Confirmed means saved and awaiting a successful run of those commands. Verified means that revision passed its dry run. Failing means the last dry run failed. Use the failing command’s last output, correct the setup, save and repeat.
When a factory runner cannot verify the recipe
CodeBook matches the command itself, not just its name. For example, if the saved Test line is npm run test:integration and your runner’s test runs npm test, a passing unit test does not verify your integration tests.
- Open Factory settings → Runtimes → Command runners. Compare the registered name and line with each step in Proof runs → Repository recipe. Use the repository root as the working folder and leave the command-specific environment overrides empty.
- If the recipe has an Install line, register the matching step too (for example,
install→npm ci). If the runner also has a before each command line, make sure it matches the recipe’s install line. Save the intended settings on each screen. - Return to Repository recipe, select On the factory’s runner and choose Try it again. If someone edits the recipe or registration while a proof is waiting, request a fresh run. Check that every required step passed at the same commit.
Older factory-runner results can remain visible in history even when CodeBook can no longer verify which recipe they ran. If a previously verified recipe is now Confirmed, use Try it to collect fresh evidence; there is no need to delete the old log.
If an agent proposes a recipe change, read its trial status under Changes runs proposed. A later verified proof of the affected step determines whether the change is kept or dropped. Repository learning and recipe history
Choose where it runs
| Location | What you supply | What to verify |
|---|---|---|
| On my computer | An installed matching checkout and the CodeBook local plugin. A queued dry run can be claimed with /factory prove in Claude Code. |
The result is attributed to the person’s computer. By default, using it for a merge-required command also requires the repository’s own checks to pass on that commit. |
| On the factory’s runner | The registered command runner on a machine you control with repository access, isolation and the required commands. | Check the runner’s connection and allowed commands if the work waits. Install or upgrade the command runner |
| In the hosted sandbox | A CodeBook installation with an operator-provided hosted runner. | Sandbox time can be metered. HTTP acceptance checks reach the app started inside the instance. Hosted browser checks require the container driver. |
The hosted sandbox is separate from the CodeBook API server. Its availability depends on the installation. A cloud worker can use the factory’s own runner when configured. If neither a factory runner nor a hosted sandbox is available, the station proof rule applies to local work only. The cloud card reports that it lacked proof. Local checkout and proof
Require proof at a station and at merge
Under Factory settings → Proof runs → Stations that need a proof run with no new failures, switch on the code-changing or QA stations you want covered. New factories enable this for code-changing and QA stations. Check existing factories individually. At a covered station, an agent’s pass needs a verified result for the PR head. A run can pass with named failures shown to exist on the default branch. Missing proof blocks the pass, and new failures send work back. A human approval is a separate decision.
Under Proof runs and merging, decide whether a proof on a person’s computer is sufficient for a required command. Within Policies & budgets, list commands the merge must pass. If you enable A merge needs no new failures, not no failures, CodeBook runs a default-branch comparison for failing tests. Unmatched failures and failures it cannot compare still hold the merge. With this option on and no explicit command list, CodeBook uses the recipe checks.
Give a story acceptance criteria
Open the card’s Evidence area. Add a criterion describing the required behavior, then choose the evidence under What shows it. For a refund fix, you might write “A repeated refund request produces one refund” and choose the exact named test printed by the test runner.
| Evidence type | What counts |
|---|---|
| Named test | A verified proof run at the PR head reports that exact test passing. |
| Command | A verified run at the PR head exits zero for the named test, build, lint or typecheck recipe step. |
| HTTP call | The hosted sandbox makes the specified request against the app it started and gets the specified status, for example GET /health 200. |
| Browser check | A verified check of the named page holds, with its screenshot. This evidence has no commit association and persists after a push. |
| Person’s sign-off | A person checks the behavior and records sign-off on the card. |
Each row shows Met, Failed or No verified evidence yet and explains why. A new push requires fresh commit-bound evidence. To enforce criteria, add Criteria evidenced to the station contract. The workflow then rechecks criteria during merge. Look at Dashboard → Testing to see which merged PRs had verified execution and which criteria only received a tick. Station contracts · Incident checks in customer CI