Agent studio

Open Agent marketplace → Agent studio and choose an installed worker. Workspace write access is required to save versions, datasets, evaluations and releases. Readers can inspect results.

For a manager, Manager settings → Agent settings → Agent Studio opens this page with the installed manager selected. Model policies and instruction releases are separate controls; see manager settings.

Save and evaluate instructions

Save the worker's current instructions as a baseline. Create a candidate by editing the instructions and saving another named version. Versions are immutable; saving one does not change production behavior. Tool permissions remain in the existing worker permission dialog and are not granted by version text.

Create a dataset with 1–50 labeled inputs. Each case needs at least one explicit check: required text, forbidden text, valid JSON or a tool fixture. Text checks are case-sensitive; JSON checks apply to the entire output. Editing a dataset increments its revision. Previously queued evaluations keep their original case snapshots.

Choose a version, dataset and optional SSO model identifier, then Queue evaluation. An empty model field uses the organization's configured default. The background scheduler runs model conversations through SSO. Cases with tool fixtures require a tool-capable model. Tools return only the frozen responses defined in the case; they do not execute real integrations or repository actions. It rechecks workspace write access before and after model calls. Results include case output, checks, elapsed time, reported tokens across all model turns, and tool-call arguments, fixture responses and failures. Provider failures are errors, not passes.

You can cancel queued or running evaluations. Cancellation discards late responses; it does not promise that the provider stopped processing or charging tokens. An interrupted evaluation becomes an error after its lease expires; it is not automatically retried.

Test tool use with fixtures

In a dataset case, select Add tool fixture. Give the tool a unique name and description, an object input JSON Schema and a JSON response. The model sees that tool during the test. When it calls the tool with valid arguments, it receives your fixture response and can continue to a final answer.

Set Expected calls to 0–8. A value of zero tests that the model avoids a tool. Optionally supply Exact expected arguments to check the full argument object, in addition to the schema. For example, a lookup_ticket fixture can require {"id":42} and return {"priority":"high"}; a final-text check can then require high.

A case supports up to 10 tools, 16 calls and 9 model turns. Each schema, response and expected-argument payload is limited to 12,000 characters. Schemas must be self-contained and cannot use references, regular-expression patterns or formats. Wrong arguments, undeclared tools, repeated call IDs, missing or excess calls, and exhausted limits fail the case. Invalid calls do not receive fixture responses.

Queued evaluations retain their original tool schemas, responses and checks even if you later edit the dataset. Expand Case results to inspect expected and recorded call counts, arguments, returned responses and errors. These are fixture tests of agent behavior; a passing result is not a claim that a production service or credential works.

Save a real run as a test case

The best test cases are the ones your factory already met. On a card, open its timeline; under any finished agent run choose Save as evaluation case.

• The request the run was given — the station, the card's title, description and checklist — becomes the case's input.
• You say what a right answer must contain, or must not. That is the check every later run of the case is judged by.
• Choose a dataset, or name a new one.

Nothing is run again. Re-running a finished run against your repository would open real pull requests; a case in the studio touches nothing real.

The original run's own numbers are kept beside the case: the model that answered, tokens, how long it took, how it ended, its scorer average, and how many times a person had to act. When you evaluate another version of the agent, or another model, against the dataset, compare with those — the same request, cheaper, faster, better, or not.

Compare evidence

Select two completed evaluations from the same dataset revision. Compare passed cases, reported tokens and requested model. Inspect individual case output before drawing conclusions. Passing text checks does not establish correctness of tool-driven production work. Missing provider usage is not evidence that a call used no tokens.

Review conversation and manager quality

Under a finished agent or manager reply, choose Review quality. The conversation owner can mark it Acceptable, Needs correction or Unsafe. Corrections and unsafe judgments need a note. In Manager settings, the same control appears on model-made decisions; reviewing those requires workflow write access. Pending work and automatic limit decisions cannot be reviewed as model judgments.

Only your own review note is returned to you. Studio shows aggregate judgments without conversation text, reviewer identities or notes. Edits and withdrawals retain an audit history; deleting the source conversation or agent removes its associated review history. A retry that changes the source invalidates its earlier reviews until the new result is reviewed.

Human quality reviews groups the last 30 days by instruction version and role. Each source counts once using each person's latest judgment. Conflicting judgments count as Disagreement, not acceptance. The accepted share uses reviewed sources as its denominator; unreviewed work remains unknown. Live instructions without a saved version are labeled separately. Reviews do not approve work, change permissions or routing, or automatically promote a release.

Start role-specific evaluation cases

In Test datasets, choose Start manager conversation cases or Start supervisor decision cases, inspect the synthetic cases, then save. Manager cases cover existing-work reuse, human approval, other-workflow boundaries, denied sources and untrusted retrieved instructions. Supervisor cases cover unchanged failures, offered routes and budget escalation. Zero expected tool calls test actions that must be avoided.

The Evaluation role is saved with the dataset. Manager conversation evaluations include the manager's runtime guidance; supervisor evaluations use a bounded decision exercise. The role, guidance, inputs and tool fixtures are frozen when queued. Changing a dataset later does not rewrite an existing evaluation. These fixtures do not recreate a live factory, establish tool-call ordering, or measure real-provider quality by themselves.

Release and roll back

Use the Replay matrix to compare saved historical tasks across versions and models before choosing a rollout. It includes original-run baselines, current versus stale dataset revisions, usage, cost and tool failures.

Choose a baseline and optionally a candidate, then allocate 0–100% of new runs to the candidate. A candidate needs a completed evaluation with every case passing against a dataset's current revision. The baseline must match current instructions, have equivalent passing evidence, or have previously been deployed as a baseline.

Native runs, crew participants and agent evaluators pin their instruction version at first execution. Supported outside runtimes and workers briefed over MCP pin it when work is handed over or claimed and receive it in their brief. Allocation is deterministic per worker and run. Retries keep the selected version after rollback. Agent and manager conversations, workflow managers, specialist supervisors and factory supervisors also use the released instructions. A conversation selects its version before its first model call; Retry and Continue retain that version after rollback or return to live instructions. New replies follow the current release. Planning selects a released instruction version before its first model call and keeps it for that invocation. Each replan starts a separate invocation using the current release; previous plans retain their original attribution. See planning history.

To promote, select the candidate as baseline and clear the candidate selection. Restore a prior baseline to roll back. Return to live instructions removes the release configuration for future runs; existing assignments remain frozen. While a release is active, direct skill/base edits are blocked to prevent changes that would appear saved but be ignored by released runs.

Release history records the actor, selected versions and percentage. The last-30-days station table reports run status by version. Token attribution includes directly assigned runs; shared crew/evaluator job tokens are not assigned to every participant.

Conversation and manager execution reports version, role, status, run count and tokens separately. Planning also appears here as its own role. Completion means the reply, decision or plan ran; it does not establish correctness or count as a station quality pass. Private conversation contents are not shown. Coverage → Manager roles it reaches lists enabled manager roles in factories you can read.

Boundaries

Workers support up to 200 instruction versions and five pending evaluations. Workspaces support up to 50 datasets. These evaluations test instruction output; they are not sandboxed integration tests. Learning promotion requires human review, evaluation and an explicit release. SSO continues to own model credentials and organization token policy.

Review learned instructions

Learning review lists pending worker-scoped memory proposals. Expand a lesson and choose Review lesson. Review or rewrite the wording, select current instructions or an immutable version as its base, and approve it as an evaluation candidate.

This creates a new immutable instruction version with the original proposal, reviewed wording and reviewer recorded as provenance. It does not activate the memory note or change production instructions. Duplicate reviews of the same proposal wording are refused; use the resulting version as a draft for further revisions.

Evaluate the candidate, inspect its results and release it with the controls above. An unevaluated candidate cannot receive rollout traffic. Roll back by restoring a prior baseline. Workflow-specific proposals remain in workflow memory: they cannot be widened into worker-wide instructions through this review flow. Direct memory approval remains a separate operator action and does not claim evaluation evidence.