The Test station

CodeBook does not run code. It has no test runner and no sandbox, and it never will. What it has is a Test station: a station whose agent works your testing services through tool connections, and whose pass CodeBook checks against what those services recorded rather than against what the agent wrote.

The shape is the one testers already know from TestRail-style work items: a card carries Cases, Runs and Results. The difference is that an agent does the work through connections, and nothing of the vendor is embedded in CodeBook.

What a Test station does

A Test station is an ordinary station with a QA agent, tool connections switched on, and a contract that holds its test report to a verified run. Its agent is told the procedure in its instructions:

1. Read the story, its acceptance criteria and the pull request under test. Each criterion is at least one test case.
2. At the test management service, find the cases already written for this card (they are tagged with the card's reference), create the ones that are missing, and update the ones the story changed.
3. Execute the tests at the connected CI or test platform against the pull request's branch, wait, and read the result.
4. Record one result per case on a run at the test management service, with a note naming the CI job and the failing output.
5. Attach the run to the card as a test report with structured data: the run with its address and counts, the cases, the results.
6. Finish: pass only when every case passed.

The agent is also told what CodeBook will check, and that a sentence saying the tests passed counts for nothing.

What CodeBook verifies

Two checks run without a model, from rows.

The contract. The Test station's contract requires a pull request and produces a test report with a run from a connected testing service. That is a rule on the test report line — Has a run from a connected testing service — and it counts an artifact only when its structured report names a run whose address is at a service this factory has connected: a Testing & CI tool connection, the TestRail connection, a connection a person marked as running tests, or a CI runtime adapter (CircleCI, Buildkite, GitLab CI/CD). The run's four counts must be present. A report written in prose, a run at an address nobody connected, or a run with no counts leaves the line unmet, and the worker's pass is recorded as blocked with the reason on the card.

The numbers. When the station's own report counts failed cases, a worker's pass becomes a fail; when it counts blocked cases and no failures, a blocked. CodeBook reads the counts; the agent's verdict is not consulted. The failed cases, with their notes and links, are written on the card as the next instruction — the first thing the Development station's worker reads when the card comes back:

Next instruction: fix the test that failed at TestRail before this card comes back to Test. CodeBook read the run's counts (1 passed, 1 failed, 0 blocked); this is not the worker's verdict.

• Failed: Refund a partial order (101) — Expected 2 lines refunded, got 1 https://acme.testrail.io/index.php?/tests/view/1001

Run: https://acme.testrail.io/index.php?/runs/view/12

A person's own approval is not held to either check; contracts and verdicts govern workers.

The structured test report

A test report artifact may carry, beside its text, a report:

{
  "run": { "id": "12", "url": "https://acme.testrail.io/index.php?/runs/view/12", "service": "TestRail", "passed": 1, "failed": 1, "blocked": 0, "untested": 0 },
  "cases": [{ "id": "100", "title": "Refund a whole order", "status": "passed" }, { "id": "101", "title": "Refund a partial order", "status": "failed" }],
  "results": [{ "case_id": "101", "status": "failed", "note": "Expected 2 lines refunded, got 1", "url": "https://acme.testrail.io/index.php?/tests/view/1001" }]
}

It is validated when the artifact is recorded (statuses are passed, failed, blocked, untested or retest; at most 500 cases and 500 results; a run needs its address, its service and its four counts) and stored as rows beside the artifact. Agents give it to attach_artifact; outside runtimes send it as report on an artifact in their answer; the TestRail connection's read_run_results tool answers in exactly this shape. A test report without one — plain text, a link — keeps working as before; it just cannot satisfy the verified-run rule.

On the card, the artifact panel renders a compact Cases / Runs / Results section from these rows: the run with its counts and a link out, each case with its status, each result with its note and link. Nothing is embedded; every link opens the service.

TestRail, as a REST connection

TestRail has no MCP server. It is connected from Tool connections → Testing & CI → TestRail as CodeBook's first REST recipe: CodeBook speaks the vendor's documented API itself and offers it to agents as tools, under the same switches, grants and permissions as any MCP server. Connecting asks for your instance's address (https://yourteam.testrail.io), the email of a dedicated TestRail user and that user's API key. The two become one Authorization header stored encrypted, like an MCP token; the email and key themselves are not kept, never shown, and never logged. Connecting checks the key against your instance at once.

The tools: list_projects, list_suites, list_sections, list_cases (the cases tagged with a card's reference), create_case, update_case, open_run, add_results, read_run_results and close_run. They arrive switched off, like any connection's.

A second recipe — Qase or Zephyr, both OAuth-only as MCP — is a data addition to the same catalogue.

Setting one up

The quickest way is a built-in blueprint. Tested Delivery Factory is Development → Test → Human review → Done with a Bug response workflow; SaaS Feature Factory now carries the same Tested delivery workflow beside its parallel review. Both expect CircleCI and TestRail among their tool connections and report them at install; a connection needs a credential, so none is made for you.

By hand:

1. Give a station a QA agent (Maya) and, under its Permissions, allow tool connections.
2. Under Contract, require a pull request and produce a test report with the rule Has a run from a connected testing service.
3. Connect the CI or test platform that executes your tests, and TestRail, under Tool connections → Testing & CI, and switch their tools on.
4. Paste the station's procedure into its instructions, or start from the Tested delivery workflow template, which carries it.

Executing tests stays where it is today: at a CI or test platform CodeBook connects to. The agent triggers it there, reads the results, writes them to TestRail, and records the artifact.