Benchmarks

A benchmark answers a practical question with your own work instead of a public leaderboard: for this kind of task, which model gives the quality we need for the least money? You pick a set of tasks this factory has already finished, hand them to several models, and the factory's own scorers grade every answer the same way.

Open a factory and choose Benchmarks in the factory tabs.

Build a task set

Choose New benchmark, name it, and narrow the factory's finished work with any of these filters:

• workflow and station
• the card's feature type (for example frontend) or category
• a card label
• files the pull request changed, as a glob such as src/web/** or *.tsx
• finished within the last 7, 30, 90, 180 or 365 days

The preview lists matching runs, newest first, one per card and station. Only finished agent runs count: gates, supervisors, people and outside runtimes are left out. Tick up to 50 and choose Freeze task set.

Freezing copies each task as it is now: the card, the station's instructions, the acceptance checklist and the original answer. Later edits to the card or the station do not change the benchmark. Editing the task set freezes a new revision; results from an older revision stay readable and are marked stale, never mixed in with current ones.

Run configurations

Choose Run configurations and pick up to six models from your organization's list. A model this factory's model policy does not allow, or one that cannot call tools, is shown with the reason and cannot be picked. You can also pick a saved version of an agent's instructions to try alongside a model.

Tick Include the original runs as the baseline to have the answers the factory really gave graded too. No model is called for the baseline; the scorers judge it exactly like the replays, so the comparison is fair.

Runs wait in a queue and execute one at a time. Every replay call counts against the factory's token budget, like any other model call.

How answers are graded

Every answer is graded by the factory's active scorers, with the same questions and the same 0–5 scale as on the Scorers tab. One judge model grades every configuration of a benchmark, so no configuration is graded more kindly than another. If that judge model is also one of the configurations, the page says so: it is grading its own answers, and you should read its quality with that in mind.

Quality is the mean of the scores that were given. Coverage is the share of scorer dimensions that could be judged. Pass rate counts answers that reached each scorer's passing mark.

Text replays, and what they cannot measure

In this release, replays are text replays: each model receives the frozen task brief and writes its answer. It does not touch the repository, run tests or open a pull request. Scorers that read the pull request's diff therefore record not scored, with that reason, rather than guess. Full repository replays need the runner and are not part of this release.

Cost against quality

The chart plots each configuration's cost per task against its quality. Up and to the left is better. Points on the frontier, joined by a line, are the configurations nothing else beats on both cost and quality. The original runs are drawn as a diamond.

Cost uses each model's price per million tokens from Bookbag (Admin → AI), split into input, output and cached tokens where the price and the usage allow. A model without a price, or an answer whose token count the provider did not report, has an unknown cost: it is listed under the chart with the reason and never plotted as free.

Below the chart, the results table shows each configuration's status, quality, coverage, pass rate, cost and tokens per task and latency. Answers opens every task's replayed answer beside the original, with each score and the judge's reason, and a per-scorer breakdown compares configurations dimension by dimension.

Propose a routing change

On a finished replay, Propose this routing turns the result into a change to which model the factory uses. Choose the stations (the stations the tasks came from are ticked) or the whole factory, and add a note for the reviewer. The chosen model becomes the station's first model; any fallbacks the station already has are kept after it.

The proposal becomes a change in the factory's Changes list, citing the benchmark, revision, quality, coverage, cost and task count. It does nothing until a second person who may change the factory approves and publishes it. The proposer cannot approve their own change.

Limits

• Up to 50 tasks per benchmark, six configurations per request, and seven benchmark runs queued or running per factory.
• Text replays only; dimensions that need a diff are not scored.
• Judge calls are not included in a configuration's cost; they are metered against the factory's budget like any scorer call.