Scorers and self-improvement

Two things turn a factory from "it runs" into "it gets better": a way to measure quality that is yours to define, and a way to act on what the measurements show. Both live on the factory's Dashboard.

One rule covers everything on this page: measuring and proposing never change how your factory works. Only you do.

Scorers

A station evaluator decides whether one run may pass, and can stop a card. A scorer is different. It asks a question you care about — are we over-engineering? are our tests any good? — of a sample of the runs that already passed, and answers from 0 to 5. It never stops anything. It tells you how good the whole factory is at something, over time.

Add a scorer

Open Dashboard → Scorers. Pick a ready-made one, or Write your own.

Ready-made scorer	The question it asks of a run
Over-engineering	Was it the smallest change that does what the card asked?
Test quality	Do the tests check the asked behaviour and its failure cases?
Redundant tests	Did it add tests that repeat coverage that already existed?
Invented requirements	Did it stay within what the card asked?
Pull request quality	Is the pull request single-purpose and clearly described?
Unneeded dependencies	Did it add a dependency it did not need?
Complete task compliance	Are the acceptance criteria actually met?
Unnecessary test removal	Were useful tests removed without justification?
Error handling	Are failure paths handled appropriately?
Security regression	Does the change introduce security weaknesses?
Architecture compliance	Does it follow the system's architecture and conventions?
Story clarity	Is the story clear enough to build from without asking?

For your own, write the question, what a 5 looks like and what a 0 looks like. Aim it at all runs, one workflow or one station, then optionally narrow it to an agent or matching changed-file paths.

What to know

• It scores from now on. A new scorer starts with the next run that finishes. It does not go back over history.
• It samples. By default it judges 20% of the runs that pass; you can set 1% to 100%. Sampling limits how many model judgments you pay for.
• It reads evidence. The judge receives the card, recorded work and available pull-request evidence, with a configurable view of the actual change.
• A factory has up to 20 scorers. Pause or remove one at any time.

Targeting and evidence profiles

In Config, choose Which runs, Which agent and optionally Only runs whose pull request touched. For example, src/api/**, *.tsx matches paths below the API directory or a matching filename anywhere. Separate patterns with commas. A run whose changed files are not known does not match a file-targeted scorer. If a targeted workflow or station is deleted, its scorers are re-aimed at all stations; review their configuration after structural changes.

Enable Needs the pull request's diff for a question about the code change itself. CodeBook reads patches from the linked pull request's source host. When the diff is unavailable, it records Not scored with the reason instead of asking the judge to infer the change from an agent's summary.

Gitea and Forgejo provide their patches separately from the changed-file list; CodeBook reads both. A list of filenames alone is not diff evidence. If every patch is unavailable, binary, or omitted by the evidence budget, a diff-dependent scorer records Not scored and makes no model call. A mixed change identifies files without text patches, so a judgment is limited to the material shown.

Each scorer has its own evidence profile:

• Order file groups: tests, dependency manifests, security, source, documentation and other files.
• Optionally show the first group's whole files at the pull request's head, up to four files, for questions about existing tests or surrounding code.
• Set a diff budget from 2,000 to 200,000 characters. Files that do not fit are listed by name.

For example, a test scorer can read tests first while a security scorer prioritizes security-related files. These are bounded views of the change; the judge does not necessarily see every file.

Reading a scorer

Each scorer is a card: its pass rate, how many sampled runs passed and failed, the trend, and the change from the window before.

A run passes a scorer at its pass mark — a score of 4 or higher unless you change it. So "87%" means 87 in 100 sampled runs scored 4 or 5.

Open a card for the scorer's own page.

• Details — the pass rate by day, and every run it judged, newest first: result, score, the judge's verdict in a few words ("No surplus tests", "Duplicates existing coverage"), how long the run took and what it cost. Open a row for the judge's reason, the agent, the model that judged, what judging cost, the card and its pull request — and Save as evaluation case, which turns that run into a test case in the agent's studio.
• Config — the question and scale, workflow/station/agent/file targeting, diff requirements and evidence profile, sample percentage, passing score, and the observer's evidence thresholds.

The selected dashboard tab and scorer are kept in the page address. Share that address with another authorized factory member to open the same view.

Coverage and human feedback

Read pass rate beside coverage: scored results divided by scored plus not-scored results. Not-scored rows count toward coverage, but not pass rate or average score. Coverage describes the recorded sample, not every factory run.

Expand a scored run and choose Agree or Disagree, with an optional disagreement note. The page reports human agreement, false positives (the judge failed work a person says was acceptable) and false negatives (the judge passed work a person says should have failed). Unreviewed judgments are not votes. Disputed failures are excluded from observer evidence.

Confidence reflects the number of human reviews: fewer than 10 is insufficient, 10–29 low, 30–99 medium, and 100 or more high. This is a review-count label, not proof that the judge is accurate. Inspect representative examples before relying on the percentage.

Cost of judging

Config shows measured judging tokens and, when prices exist, dollars. It projects monthly judging cost from observed usage and shows how changing the sample percentage affects that estimate. A projection is not a spending cap or bill. Missing prices remain unpriced; see Spending.

Ready-made scorers

Twelve come ready to switch on: Over-engineering, Test quality, Redundant tests, Invented requirements, Pull request quality, Unneeded dependencies, Complete task compliance, Unnecessary test removal, Error handling, Security regression, Architecture compliance and Story clarity. A factory can have up to 20 scorers.

Self-improvement

Dashboard → Self-improvement is where the factory proposes how its agents should be told to work.

Once a week, in a factory with an active scorer — and whenever you press Look now — the observer reads what went wrong: eligible scorer failures, failed station evaluations, and failed or blocked runs. Scorer failures use each scorer's passing score and evidence window; other run evidence covers the last two weeks. Budget, policy and missing-model stops are not evidence of poor agent instructions. It looks for recurring problems that an instruction or factory change could prevent.

What it finds is a proposal:

• the pattern — what keeps going wrong
• the likely cause
• the exact instruction it would add
• the runs it was seen in, so you can check it rather than believe it

A proposal has to rest on at least two runs. An agent with fewer than three runs that went wrong is not looked at at all: no pattern, no proposal, no cost.

Enough evidence first

A scorer's failures become observer evidence only when its configured thresholds are met: a minimum count, a time window and optionally a minimum failure percentage. The default is 20 failed samples in 90 days, set per scorer on Config. Disputed failures do not qualify. Until then Self-improvement shows where each scorer stands: 18 of 20 failures — not enough evidence yet. Not-scored results do not supply failure evidence, and a run waiting indefinitely for a person does not permanently stop the scorer scanning later work.

Changes that are not about wording

Sometimes a sentence in an agent's instructions is not the fix. When an agent keeps skipping a step the workflow could simply require, the observer proposes a change to the factory instead — a command that must pass before a station, a review station, a different model. Accepting one changes nothing by itself: it records that you took it on, and you make the change in the factory's settings.

Accepting a proposal

1. Read the evidence. Edit the instruction if you would say it differently.
2. Accept as a new version. The instruction is added to the agent's instructions as a new version of the agent.
3. That version runs nothing yet. Use Test before you release on the accepted proposal, choose a workspace dataset and run its evaluation. Inspect it in Studio or compare versions and models in Replay, then release deliberately — to a share of runs first, if you like.

Or Dismiss it, with a reason if you want one kept.

The observer never changes an agent, a station, or a setting. The agent that is working your cards today is the one you released.

Did the proposals help?

The improvement report follows proposals through acceptance, evaluation and release. It distinguishes improved, regressed, unchanged and unbaselined results, with average pass-rate and token-cost changes.

Comparisons use another version of the same agent on the same dataset revision, preferring the released baseline when available. A gain of at least one pass-rate point counts as improved; within one point, at least 5% fewer tokens per case also counts. The reverse counts as regressed. Without a matching baseline, the result remains unbaselined. Here, cost change means tokens per case, not a dollar saving claim.

Fixture improvement still needs validation on real work. Typed memory, skill and retrieval-change proposals are not implemented yet; see current limits.

Comparing models with the same scorers

The factory's Benchmarks replay a frozen set of finished tasks on other models and grade every answer with these scorers, so a cheaper or stronger model can be judged on this factory's own work before anyone routes to it.