Artifacts, contracts and evaluators
Define the evidence a station needs and independently judge a worker’s pass.
Artifacts attach typed evidence to work: requirements, pull requests, test reports and other outputs. Contracts say what a card must have before work starts and what it must gain before a worker can pass.
Contracts
Configure required inputs and produced outputs on the station. Supported output rules can check structure or content, such as a required heading or minimum length. Missing inputs prevent work from starting; missing outputs prevent a worker’s pass from being accepted.
These are platform checks on worker execution. A person’s own approval follows the human approval path. Contracts are not cryptographic signatures or a general external-schema compatibility system.
Independent evaluators
Evaluators run after a worker reports pass. Choose from:
| Kind | What it checks |
|---|---|
| Another agent | Reads the work against written criteria; uses metered model tokens |
| Pull request checks | Requires the linked pull request’s CI result |
| Contract output rules | Checks the station’s declared output rules |
| Human sample | Selects a stable sample of runs for a named person to review |
An agent evaluator uses its own installed-agent model policy, then the station and factory policies, then the organization default. Configure it from Marketplace → Your agents → Model. Its calls stay within the parent run’s token budget.
An agent cannot evaluate its own work. Missing CI results and unavailable models are not evidence of success.
Required and advisory
Only required evaluators decide whether the worker’s pass is accepted. Their results combine under all, majority or any. Advisory results are recorded without blocking.
An evaluator error records unsure and stays in the required consensus rather than disappearing from it. Under “all,” that prevents a pass; under “any,” another required evaluator’s pass can satisfy the configured rule. An error in the evaluation engine itself does not approve the work.
Selected required human reviews hold the run until the named reviewer decides, when that answer can still affect the outcome. Verdicts are reused for the same run instead of repeatedly asking a model or rerolling the sample.
Inspect the recorded notes before retrying. Evaluator performance is measured from later outcomes when enough settled evidence exists.
Inspect the prepared input
Open the owning run's Context usage → Prepared model inputs to see the fitted request hash, model, instruction version and evaluator identity for each attempted model call. Two criteria using the same agent remain separate entries. The same panel lists the station worker's and crew participants' own model attempts. Reusing an existing verdict does not create a new model attempt.
CodeBook rechecks current job ownership and supplied evidence before dispatch and after the reply. Changed authority or evidence, or failure to record the prepared input, blocks acceptance even under any consensus. Calls that already ran retain their metered usage. Frozen run criteria remain frozen. These records describe prepared requests; they do not prove provider completion or provide historical prompt replay.