Measuring the context engine

The context engine changes what a run is sent, and CodeBook measures it. Every measured non-crew run keeps what it was sent, an estimate of what the path the engine replaced would have sent, and the tokens the provider reported. When a factory asks for it, a controlled comparison — a share of cards held out on the old path — compares outcomes with a written test.

CodeBook makes no claim here about which path is better. The numbers are yours, from your runs, and the panel reports whichever way they fall.

Where it is

Factory metrics has a Context engine panel under the station tables, following the page's time window. An expanded run in Run history shows the same facts for that one run. Export runs (CSV) downloads a row per run.

The panel has two parts, labelled so they cannot be confused:

• ESTIMATED — a counterfactual, not a measurement. What the engine sent per run, what the pre-engine path would have sent for the same runs, the difference between them, and the whole candidate set had every ranked item been sent. The estimate can go either way, and the panel counts the runs on which the engine sent more.
• MEASURED — the holdout: each side's value with its sample size, the difference with its 95% interval, the verdict with its p-value, or a sentence saying there is not enough yet.

Beside them, the provider-reported input tokens per run for each side, with the sentence that they are not a comparison, and How these are computed, which prints the definitions below from the API.

What is recorded per run

Every recorded run — a worker's run, an outside runtime's brief, an MCP claim — gets one measurement row, written after the context block is built. Nothing in it reaches the block: the specification builds the same run with measurement on, off and failing, and compares the prompt text byte for byte.

Recorded	What it is
Arm	engine (the engine's pack), baseline (held out: the pre-engine path's answer), or no map (no mapped repository; in no arm)
Sent	The context block the run was handed, in UTF-8 bytes and in estimated tokens for the run's model family
Candidates	For the engine: every ranked candidate, the ones admitted, the ones left out whole, and the size of the whole candidate set
Pre-engine baseline	What the pre-engine path would have sent for this run. On the engine arm it is an estimate, with its basis; on the baseline and no-map arms it is what was sent
Actual tokens	The input and output tokens the provider reported for the run's model calls, summed

The baseline is computed, not modelled. It is the pre-engine path itself, run for the same card, job and worker, on a copy of the run's token plan taken before the engine used any of it, replaying the query vectors the engine's retrieval already obtained. It adds no model call and no embedding call. Its basis says how exact it is: sent, exact, approximated with the engine's query vector, text matching only, or failed — and a failed one is empty, never zero.

Unknown is not zero. A run with any model call that reported no token count is unknown: left out of token comparisons and counted. An outside runtime's runs have no model calls of CodeBook's and are unknown.

The holdout

A holdout is off by default. A person with write access to the factory starts one from the panel:

• Share of cards held out: 10–50%, default 20%.
• Completed cards per side: 10–1,000, default 30.

One holdout runs per factory. While it runs, each run of a workflow with a mapped repository is assigned by a hash of the holdout and the card — the same hash CodeBook's worker A/B experiments use — to the baseline arm when the card's bucket is below the share, and to the engine otherwise. The same card always lands on the same side: a retry, a resume or a re-claimed outside run keeps its arm. A held-out run reads exactly what the pre-engine path returns, and the specification compares that block byte for byte.

The holdout ends by itself once both sides have their completed cards, and writes its result. Stop holdout ends it early and writes the comparison so far. Nothing is applied either way: every run goes back to the engine, and a person reads the result. A holdout is not part of the factory's definition and is saved directly.

Outcome metrics

The comparison reads, for each side, the runs that are no longer live, and the completed cards whose work started after the holdout began and whose every measured run between start and finish belongs to this holdout and to one side. Other completed cards are left out and counted.

These are the metric definitions CodeBook uses, exactly as written in the application:

Metric (key)	Definition	Test
input_tokens_per_run	provider-reported input tokens over every call of a run	Mann–Whitney
output_tokens_per_run	provider-reported output tokens of a run	Mann–Whitney
input_tokens_per_card	provider-reported input tokens over a completed card's measured runs; unknown if any run is	Mann–Whitney
cost_per_card	metered tokens of the card (CardOutcome.tokens, factoryOutcomes.js, from WorkerJob.tokens_used). CodeBook keeps no credit ledger (SSO is the meter); tokens are the unit of every cost in the product	Mann–Whitney
first_pass	workerPerformance FIRST-PASS: the task passed on the worker's first verdict and never came back to the station (no fail route, no handoff back)	two-proportion z
qa_failure	of passed tasks a later station judged, the share it failed straight back	two-proportion z
rework_per_card	CardOutcome.station_revisits: times the card came back to a station it had left	Mann–Whitney
human_rejection	share of completed cards with a WorkerJob a person rejected at an approval (before a run or at a human review station) between start and finish	two-proportion z
time_to_done	CardOutcome.finished_at − started_at	Mann–Whitney
cited_runs	share of settled runs whose final result names at least one citation (K…, N…, E…) the run was given — RunContextItem / ContextRunItem rows marked included. Only the final answer is read (the newest 300 runs per arm); tool calls and pull requests are not	two-proportion z

Runs record no other trace of which citations a model used, and nothing else is claimed.

The significance rule

The rule is written and deterministic:

• Rates use a pooled two-proportion z-test, two-sided, at α 0.05. A result is published only when each side has 20 runs or tasks (10 cards for card rates) and each side expects at least 5 successes and 5 failures at the pooled rate. The difference carries its 95% Wald interval.
• Medians use the Mann–Whitney U test, two-sided, with tie and continuity corrections, under the same thresholds. The difference of medians (engine minus pre-engine) carries a 95% bootstrap interval from 1,000 seeded resamples.
• Below a threshold the verdict is not enough, with the counts — for example "the engine arm has 7 and the baseline arm 2; each needs 20" — and no percentage and no p-value. Otherwise the verdict is no difference, engine better or engine worse, by each metric's written direction.

The CSV

Export runs (CSV) gives one row per measured run in the window. Columns that are estimates end in _est, and a row whose pre-engine figure is an estimate says so in pre_engine_is_estimate. An unknown count is an empty cell, never zero.

What is not measured

• Crew participants retain separate context manifests and model-call usage. Crews are excluded from the existing one-row-per-job savings comparison; a member's context size is not attributed to the whole crew.
• Citations in tool calls or pull requests are not read; only a run's final answer is.
• No automatic switch. A holdout never turns the engine off or on. A person reads the result, as with worker experiments.
• The baseline costs one more knowledge read per engine run — a second retrieval over the run's permitted sources, with no model call.

Next

What the worker reads — how the context pack is built. Optimization and outcomes — the factory's other measured outcomes.