Test runs

CodeBook does not run code. A test run pipeline is how a factory runs its tests anyway, without an agent sitting in a loop waiting for CI: CodeBook opens a run at TestRail for the right cases, asks your CI service to run the tests, waits on the repository host's events, fetches the JUnit report, writes one result per case back to TestRail, records the structured test report on the card, and judges the run from the counts. The agent is used at two points only — before the run, to turn the card's acceptance criteria into cases, and after it, to explain the failures — and both of its outputs are proposals, never the verdict.

Open a factory and choose Test runs in the factory tabs.

A pipeline

One definition says what a run is; several triggers say when it starts.

• Which cases. The cases tagged with the card's reference (the Test station's agent writes them from the acceptance criteria, as in The Test station), the cases carrying a tag you name, or a whole suite.
• Where they are recorded. Your TestRail connection, a project and, for a multi-suite project, a suite.
• Which CI service runs them. Only a service with a documented start-a-run API is offered: CircleCI, Buildkite, Semaphore or Checkly, each connected as its API under Tool connections → Testing & CI (the CircleCI API, Buildkite API, Semaphore API and Checkly API entries, beside the MCP servers agents use). You give the project — a CircleCI slug such as gh/acme/web, a Buildkite organization/pipeline, a Semaphore project id, the tag Checkly's checks carry — and the branch a run uses when it has no pull request.
• Where the results are read. The JUnit files the CI run stored, matched by a glob (**/*.xml by default), or a results dashboard: Currents is offered as a result source only — it starts nothing, the CI does.
• The status map. What a JUnit error and a skipped become at TestRail (failed or blocked; untested or blocked).
• A time limit, after which a run the CI never finished is failed with that reason.

Triggers

• A card enters a station — usually the Test station.
• A pull request is opened or updated — a run for each card linked to it, on the pull request's branch and commit.
• A release candidate is pushed — a branch or tag matching a pattern such as release/*, for a pipeline that runs a suite or a tag's cases.
• A nightly schedule — an hour (UTC) and, if you like, a weekday; once per day.
• The Run button on the Test runs tab needs no trigger; a card-scoped pipeline asks for the card.

One active run per card and pipeline: a second trigger while one is live answers the live run.

How a run moves

A run is always in one of these states, and every transition is recorded with its reason and who caused it:

State	What happened
Opened	The run exists. With an agent on the station it waits for the cases; otherwise it is prepared at once.
Triggered	The TestRail run is open for the scoped cases and the CI service was asked to run.
Running	The CI service reported the run in progress.
Results written	The report was fetched, mapped to cases, written to TestRail and recorded on the card as a structured test report.
Judged	CodeBook's verdict from the counts: pass, fail or blocked.
Failed	The run could not be completed — the CI refused, no report was found, or the time limit passed — with the reason.
Abandoned	The card moved on, a person pressed Stop, or the pipeline was removed.

The run advances on events, never on an agent: the repository host's check or status webhook about the run's commit, or the CI service reporting completion. A bounded reconciler polls runs the webhook has not reached, and ends runs past their time limit. Check now on a run asks the CI once if you do not want to wait.

Every transition is also a factory event on the card, and the card's Evidence section shows each run with its timeline.

JUnit, and how tests become cases

JUnit XML is the interchange. Each test in the report is mapped to a TestRail case:

1. by the convention C<id> in the test's name or class — C1042 refund a partial order, tests.refund.C1042, [C1042];
2. otherwise by an exact title match, whitespace folded and case ignored;
3. otherwise the test is handed to the agent, which proposes the case it believes the test verifies, or none.

The agent's answer is recorded on the run as mapping proposals a person can read; a proposal naming a free case that was offered is applied and the result written under it, the rest stay proposals. A test that failed but mapped to no case is not in TestRail's counts, so it does not vanish into a pass: the run is judged blocked and says why.

The verdict, and the agent's two points

The verdict is CodeBook's, from the counts TestRail records: failures → fail; blocked cases, a failed test with no case, or a run where nothing was exercised → blocked; otherwise pass. The structured test report the pipeline records on the card is the same one the Test station's verified-run contract checks, so the station's own rules work unchanged.

The agent appears twice:

• Before. When the card enters a Test station with an agent, the agent's run does what it did before — reads the acceptance criteria and makes sure a tagged case exists for each — and finishes. CodeBook parks that run as awaiting tests, opens the TestRail run and triggers the CI. The agent is told not to open a run, trigger CI, wait or attach anything.
• After. When the pipeline run ends, the station's run is queued again with the results in its brief: the counts, each failed test with its message, and how often that test failed in the pipeline's last runs (a test that failed before but not always is probably flaky). The agent explains the failures, says which look flaky, and writes the next instruction for the Development station. Its explanation is posted on the card as a proposal; a pass it reports against a run that did not pass is overridden, and the counts decide.

A run with no station agent — a release candidate, a schedule, the Run button on a station nobody works — gets a one-shot analysis after a verdict that is not pass, recorded on the run and the card under a proposal, not the verdict. The two model calls appear on the AI Models page as Test run mapping and Test run analysis.

The Test runs tab

The list shows every run of the factory with its state, verdict, trigger, counts, the TestRail run and the CI run with links out, filtered by pipeline, state or card. A run's page shows its timeline, both halves with their addresses, every test the report gave with the case it was written to and its history, the agent's mapping proposals, and its analysis. Stop abandons a live run with a reason; the CI service is not cancelled. The card's Cases / Runs / Results panel links each report to the run it came from.

Everything on the tab is one factory's; nothing crosses an organization outside /admin.