Compare versions and models with replay
Compare historical tasks with agent-version and model combinations using frozen evaluation cases, recorded usage and explicit baselines.
Open Agent marketplace → Agent studio, choose an installed agent, and find Replay. Select a dataset to compare what happened on the original tasks with completed evaluations of other instruction versions and models.
Replay compares one agent's versions and models on cases you saved by hand. To compare models across a whole factory's finished work, graded by the factory's own scorers with a cost-against-quality chart, use the factory's Benchmarks.
Start with real cases
On a finished card run or a scorer's judged-run detail, choose Save as evaluation case. Select a dataset and write checks for the behavior you expect. The saved case keeps the original run's request and reference measurements. A result recorded as not scored is excluded from the original scorer average.
Cases can also be written by hand or started from the manager/supervisor presets in Studio. Each evaluation freezes its dataset role and role guidance. See Agent studio for frozen tool responses and checks.
Read the comparison
The first row, What really happened, summarizes the original runs attached to the dataset's cases. It reports how many cases have an original, their recorded outcomes, tokens, cost when priced, time per run and human touches.
Each later row is an agent, instruction version and model with a completed evaluation. It shows:
- Cases passed and pass rate, with a difference from the original completion rate when available.
- Reported tokens and cost, using the prices configured at Bookbag SSO.
- Median evaluation latency and tool failures.
- The number of completed evaluations and the latest result; repeated results also show a mean pass rate.
Stale means the evaluation used an earlier revision of the dataset. Best ranks the displayed results by pass rate, then cost, then tokens; a known price ranks ahead of an unknown price. The ranking can include a stale result, so compare the dataset revisions before acting on it. The badge selects no production model and releases no instructions.
Tokens and cost are compared with the originals only when every case came from a real run. An unpriced model says No price or is named as unpriced; missing usage or prices are not treated as free work.
Original completion and fixture pass rate answer different questions. The original row's human touches are not tool failures, and its time per run is not the candidate's median case latency. Inspect the case results before interpreting a faster or cheaper row as better software.
Run a matrix
- Choose a studio worker and save the instruction versions you want to compare.
- In Replay, choose a dataset.
- Under Run a matrix, tick the versions and enter comma-separated SSO model IDs. Leave the model field empty to use the organization's default.
- Select Queue matrix. The table refreshes as evaluations finish.
One request can contain up to 12 version/model pairs, and each worker can have at most five pending evaluations. A pair already queued or running on the same dataset revision is skipped; the result explains what was not queued. Model requests run through SSO and consume the organization's allowance.
What replay does and does not run
Replay uses ordinary Studio evaluations against frozen inputs and tool fixtures. It does not reopen the original pull request, execute repository writes, recreate a historical production environment or change the live agent. A passing fixture comparison does not verify a real integration or prove model quality on future work.
After reviewing the evidence, use Studio's release and rollback controls. Released instructions also govern manager conversations and workflow, specialist and factory supervision. Their execution counts remain separate from station quality; frozen replay fixtures alone do not measure real manager judgment.