Model policies and routing

CodeBook picks among the models Bookbag SSO already offers your organization. It reads that list live, stores no provider key and keeps no model catalogue of its own. Every model call is metered through SSO. Routing measures quality, tokens and time; the Spending dashboard also reports dollars when model prices are configured.

Open Factory settings → Models for the factory's policy and a measured-performance table per model and kind of work. The same editor sits on a station's Worker tab and behind the Model button on a worker in the marketplace's In this workspace view.

For a workflow manager, open Manager settings → Agent settings → Model from the board. The same worker policy applies to its conversations and automatic interventions.

Policies

A policy can be set on a factory, a station or a worker. The narrowest one wins: worker, then station, then factory. With no policy anywhere, a run uses the organization's default model, as before.

Station runs, ordinary station conversations, crew participants and station evaluators follow that order. Each crew leader, member and reviewer uses its own installed-agent policy; they do not share one selected model. The factory planner follows worker → factory → organization default. Manager conversations and workflow, specialist and factory supervision follow worker → factory → organization default: they do not inherit the station where a card became stuck or where a conversation is stored.

• Fixed: one model. If a pinned model is switched off or removed at SSO, the run stops with an error that names it. It is never silently swapped.
• Fallback: an ordered chain of two to four models.
• Auto: the model is chosen from measured results, with an objective — quality, speed, tokens or balanced — and a cap on exploration from 0 to 50 percent (default 20).

Every model named in a policy must be an active chat model that can call tools in the organization's SSO list at the time the policy is saved, so a typo cannot be saved. The organization is read from the scope itself, never from the request.

Policies in the factory definition

A factory's policy and each station's policy are part of the factory definition. They are recorded in every new version, shown in Versions ("Factory model: …", "Model: …"), listed by Compare, merged, exported and restored by rollback. Clearing a policy is recorded as explicit inheritance. Versions recorded before this have unknown model history and say so. A worker's own policy is not a factory setting and does not travel with the factory.

Saving creates a version, or a request. In a factory that does not require review, saving or removing a policy applies from the next run and records a new factory version. In a factory that requires another person's approval, the editor shows a reason field and Request model policy change instead: the save becomes an ordinary change request that does nothing until a second writer approves and publishes it. The requester cannot approve their own request.

Shared worker policies are read-only while review applies. While any factory in the workspace requires approval, a worker's own policy cannot be edited, because worker policies are not yet versioned with factories.

A missing model is never substituted. When a definition is applied — installing a blueprint, restoring, rolling back or promoting — to an organization that does not have a model the policy names, that factory or station is left without a policy of its own, so runs inherit the wider choice. The install report, rollback outcome, change trail, blueprint-update and promotion reports name the missing models, and a rollback that hits one is not reported as exact. No other model is chosen in its place.

Eligibility

A model is skipped when it is unavailable or incompatible with the required tools, images or input size. Provider-qualified selections require SSO support. Each attempt fits its request to the model's input budget, using token estimates adjusted from reported usage and byte limits where needed. These are estimates, not a provider tokenizer. Eligibility and fitting are checked again as the conversation grows.

In a chain, a model too small for the required input can be skipped before a provider call so a later model can handle it. Chat uses the same policy during message checks and execution, and can summarize older history. A fixed model that cannot fit the required content still refuses it; shorten the request or select a larger model.

Fallback chains

After a provider call, fallback is limited to a timeout, a provider 5xx, an overloaded answer (including a rate limit SSO already retried) or SSO reporting the model unavailable. It does not bypass token-limit or budget refusals, MadeBook decisions, a caller's stop, other coded SSO refusals, provider 4xx/content refusals, an exhausted context budget, unreachable SSO or unrecognized failures. Skipping an ineligible model before sending it the request is separate from retrying a provider failure. A model that refuses in words has answered.

Each attempt is checked against the budget, metered and recorded, with its own idempotency key so a timed-out attempt cannot make the next one free. A call tries at most four models. The run's next step starts from the model that answered and never goes back to one that failed it. Each crew participant keeps its own fallback chain across its turns. The job's events record the routing decision with its reason, and each failed attempt with its failure class and status, never the provider's response body.

Performance-driven choice

One record per run and model holds calls, failed calls, tokens, model time and which model's answer the run ended on, kept per model and per kind of work (the card's work category). Quality is the Wilson lower bound of runs that passed and whose card outcome has settled at a good score; an outcome still inside its 14-day provisional window counts for nothing. The arithmetic is the same as worker routing uses.

Exploration is deterministic: the model with the fewest runs goes first, and exploration happens only while it stays under the cap. Past the cap, the best model with a record runs; if none has a record, the first model in the list runs and the reason says the data is insufficient. Every tenth decision re-measures the model used longest ago, inside the same cap. The table shows sample sizes and "Insufficient data" rather than an invented rating.

Automatic manager, planner, crew and evaluator selection uses these measured station outcomes too. Supervisor model calls remain audited, but a completed manager decision is excluded from station quality statistics. Separate manager, planner and reviewer quality evaluation is not yet available. Crew-member and evaluator calls use the parent job’s budget without replacing the station worker’s model-performance record.

Not built

• Policy coverage is not universal. Station runs, station/manager chat and workflow/specialist/factory supervision use these policies. Planner, evaluator and crew calls also use their agent policies. Domain-extraction policy integration remains open. Studio uses its explicitly requested evaluation model or the organization default.
• Worker policies are not part of the factory definition. Factory and station policies are; a worker's own policy is not in snapshots, versions, blueprints or rollback.
• SSO gives provider failures no code, so the failure class is read from the status inside SSO's error text. An SSO without provider-qualified model selection cannot report a removed model as unavailable; its documented behaviour is to fall back to the default.