Production monitoring

The System Map ends in two layers the code cannot supply: runtime (what is deployed, and which commit it is running) and incident (what broke in production). Runtime nodes are read from the repository's deployment files. Incidents and deploys arrive from the systems that watch production, through monitoring sources an organization administrator connects.

Each incident is linked to a runtime, service or module on the map by an administrator's mapping or by an exact name, and never by a guess. A linked production incident then raises the impact level of cards that touch those components, reaches the context pack while it is open, and is shown on the node sheet.

Where it is

Organization settings → System Map → Production monitoring. Members of the organization read everything on it. Only an organization administrator can connect, rotate or remove a source, or add and remove a mapping; every route checks that again.

The page has four parts: Monitoring sources, Recent deliveries, Incidents not linked to the map, and the Service → node mapping. A fold at the bottom, What each provider sends, and the generic webhook's JSON, repeats the provider notes and payloads on this page.

An organization can connect up to 20 sources.

Runtime nodes come from the repository

Runtime nodes are not typed in and do not come from monitoring. The analysis's runtime phase reads the deployment descriptors in the repository:

• Dockerfile and Containerfile, and docker-compose files
• Kubernetes manifests (Deployment, StatefulSet, DaemonSet, Job, CronJob, Pod and Rollout, with Services and Ingresses)
• a Helm Chart.yaml and the values*.yaml beside it (templates are skipped)
• fly.toml, render.yaml, vercel.json, netlify.toml, Procfile, App Engine app.yaml (only with a runtime: key) and serverless.yml
• Terraform resources of the runtime types the rule lists
• deploy steps in .github/workflows/*.yml

Each deployable becomes one runtime node, cited to its file and line. A written rule then links it:

• runs the module or service in the directory the descriptor names (a build context, a Dockerfile's directory, a serverless handler's file, a Render rootDir), or the module whose image another placed runtime builds — both certain. When only the image's last segment or the runtime's name equals exactly one module's name, the edge is possible. Otherwise there is no edge.
• exposes the endpoints of what it runs: certain under a declared route (an Ingress path, an App Engine handler, a serverless HTTP path, a Helm ingress path), possible when it only publishes a port or is a web process.
• A GitHub Actions deploy step is only evidence for a runtime (evidenced by), never a runtime of its own.

A descriptor already listed before this release is read on the next full build. Rebuild on the System Map page starts one.

Connect a source

1. Choose the provider: Sentry, Datadog, PagerDuty, Opsgenie, Grafana / Alertmanager or Generic webhook.
2. Give it a name, such as Sentry production.
3. For Sentry, paste the integration's client secret; it is required. For PagerDuty and the generic webhook a signing secret is optional.
4. Connect.

The page then shows two values once, with the sentence "Copy these now: the token is shown once and only its hash is kept."

The address the provider posts to. On a deployed CodeBook the page shows:

https://<your CodeBook address>/api/kanban/api/monitoring/inbound?source=<source id>

The same endpoint also answers at the web origin without the leading /api:

https://<your CodeBook address>/kanban/api/monitoring/inbound?source=<source id>

The source parameter only names the source. It is not a secret and authorizes nothing.

The token. A 32-byte token, sent as the header:

Authorization: Bearer <token>

CodeBook stores only its SHA-256 hash. The source list shows its last characters, how many deliveries were received and refused, and when the last one arrived.

A signing secret, when the source holds one. Then every delivery must carry a valid signature of its raw body under that secret, and the token is not needed for it:

Provider	Signature header
Sentry	Sentry-Hook-Signature — hex HMAC-SHA256 of the body with the integration's client secret. Required: Sentry cannot send a bearer header
PagerDuty	X-PagerDuty-Signature: v1=<HMAC-SHA256> — with the v3 subscription's secret
Generic webhook	X-CodeBook-Signature: sha256=<hex HMAC-SHA256 of the raw body> — with a shared secret you choose

Setting up each provider

Provider	In the provider	Authenticated by
Sentry	An internal integration's webhook, with issue, alert and metric alert events	The client secret's signature
Datadog	The webhook integration, with the token as a custom header (Authorization: Bearer <token>) and the payload below	The token
PagerDuty	A v3 webhook subscription with incident events	The token as a custom header, or the subscription's signing secret
Opsgenie	A webhook integration, with the token in the Authorization header	The token
Grafana / Alertmanager	An Alertmanager webhook_config receiver or a Grafana webhook contact point, with the token in the Authorization header	The token
Generic webhook	Your own sender, posting the JSON below	The token, or the shared secret's signature

Datadog's payload. Paste this into the webhook's payload field; the $ variables are Datadog's own:

{
  "id": "$ID",
  "alert_id": "$ALERT_ID",
  "alert_cycle_key": "$ALERT_CYCLE_KEY",
  "title": "$EVENT_TITLE",
  "transition": "$ALERT_TRANSITION",
  "priority": "$ALERT_PRIORITY",
  "alert_type": "$ALERT_TYPE",
  "date": "$DATE",
  "last_updated": "$LAST_UPDATED",
  "link": "$LINK",
  "tags": "$TAGS",
  "hostname": "$HOSTNAME",
  "scope": "$ALERT_SCOPE"
}

The generic webhook's JSON — an incident, and a deploy (which commit a runtime is running). The values show what each field holds:

{
  "incident": {
    "event": "incident",
    "id": "your stable incident id (required)",
    "title": "what broke (required)",
    "severity": "critical | high | medium | low | info",
    "status": "open | resolved",
    "started_at": "ISO 8601 or epoch ms",
    "resolved_at": "ISO 8601 or epoch ms",
    "service": "the service or runtime name",
    "services": ["more names"],
    "tags": ["key:value"],
    "environment": "production",
    "url": "https://…"
  },
  "deploy": {
    "event": "deploy",
    "id": "your deploy id (optional: service + environment + commit_sha otherwise)",
    "service": "the runtime or service name (required)",
    "environment": "production",
    "commit_sha": "the full commit sha (required)",
    "ref": "main",
    "status": "success | failure | in_progress",
    "deployed_at": "ISO 8601 or epoch ms",
    "url": "https://…",
    "repository": "owner/name"
  }
}

Send one object per delivery, with event set to incident, deploy or test.

The names each provider sends

Linking reads the names an incident carries. Each is lower-cased, and a source keeps at most 20 per incident.

Provider	Service names	Tags
Sentry	For an issue, the project's slug and name; for an alert, the event's service or app tag and its project	The event's tags
Datadog	The service: or app: tag	Every tag
PagerDuty	The PagerDuty service's name	team:<name> for each team
Opsgenie	The alert's service detail, a service: tag, and its entity	Every tag
Grafana / Alertmanager	The labels service, app, application, app_kubernetes_io_name, deployment, job and container	Every label, as key:value
Generic webhook	service and services	tags

What a delivery does

• Refused. An unknown source, a wrong token or a bad signature gets one answer, 401, and is logged under Recent deliveries as refused with the reason.
• Rate limited. More than 120 deliveries a minute to one source, or 300 a minute from one sending address, is answered 429.
• Once only. Each delivery is claimed by the provider's delivery id or, without one, the SHA-256 of its body. A replay answers duplicate and is applied once.
• Not the provider's shape. A body that is not what the provider documents is recorded as error with the reason and answered 400. A corrected retry is read again.
• Tests. Sentry's installation event, PagerDuty's ping, an Opsgenie delivery without an alert and a generic test event are acknowledged and write nothing.

The last 200 deliveries of each source are kept, with counts by outcome.

Rotate replaces the token at once, and can replace the signing secret; the provider must be given the new token. Remove stops the address working and deletes the source's delivery log; the incidents and deploys it sent stay on the map.

What an incident becomes

• Severity is one of critical, high, medium, low and info, read from each provider's written table (P1–P5, levels, urgency). A word the table does not list is medium.
• Status is open or resolved. A delivery that only acknowledges, annotates or assigns changes nothing. An incident is created, updated, resolved and reopened idempotently; the same state sent again is unchanged.
• One incident per alert for Grafana / Alertmanager: a group of alerts is up to 50 incidents, keyed by fingerprint.
• A node. Each incident is an incident node on the organization's map, with the provenance runtime, and an affects edge to what it is linked to.
• Events. Only a change of status sends incident.opened or incident.resolved (the System map group of events), to every factory with a workflow attached to the linked node's repository. An incident that is not linked reaches no factory.

Production is any environment except staging, stage, dev, development, test, testing, qa, preview, sandbox, local, uat, ci and review. An incident that names no environment counts as production. Incidents in the listed environments are kept, but impact analysis does not count them.

Linking, never guessed

An incident is linked by the first rule that applies:

1. A mapping an administrator wrote: a service name or tag → a runtime, service or module node.
2. An exact name: a name the incident carries is exactly one runtime's name; else exactly one service's; else exactly one module's. Two nodes with that name is no link, and the note says so.
3. A GitHub deployment is linked to the repository's only runtime.

Anything else is kept as it arrived and listed under Incidents not linked to the map, with the note saying why and the names it sent. A mapping always wins over a name.

The mapping table

Service → node mapping is read before any name rule. Each row is:

Match	Value	Node
Service	the service name the provider sends, e.g. checkout-api	a runtime, service or module, picked by search
Tag	key:value, e.g. kube_deployment:checkout	the same

From an incident that is not linked, one click maps the first service name it sent to the node you pick, and every incident naming it is linked at once. Removing a mapping applies the name rules again. An organization can hold up to 500 mappings.

Deploy events

A deploy records which commit a runtime is running. Deploys arrive two ways:

• The generic webhook's deploy event — service, environment and commit_sha (7 to 64 hex characters) are what it is read by.
• GitHub's deployment_status, through the repository's own webhook. CodeBook's hook subscribes to it; hooks registered before it did are brought up to date by the webhook repair.

A deploy is linked to one runtime; a mapped service is followed to its only runtime. When the repository has a map version of exactly that commit (or a prefix of at least seven characters that matches one), the deploy records it. The runtime's node sheet reads running <sha> since <time> per environment.

Each incident records the newest successful deploy of its runtime at or before the incident started, in the same environment when both name one — so an incident's sheet says which commit was running when it began. That is recomputed when the incident's link changes.

Deploys and cards: merged is not live

CodeBook does not deploy. Your own system deploys when the main branch changes, and the deploy events above tell CodeBook the result. For a production deploy, CodeBook ties that result to the cards whose merged work the deploy carried:

• It succeeded. Every merged pull request the deploy carries is marked live — the one whose merge commit was deployed and every one merged before it. Each card gets a DEPLOYED comment naming the environment and commit, and its outcome records it.
• It failed. One pull request answers for it: the one whose merge commit was being deployed. Its card goes back to the station that built it, which is started with the reason and the deploy log link. The merged pull request stays merged; the fix is a new pull request on the same card. After three returns, or when CodeBook cannot tell which station built it, a person is asked instead.

Staging, preview and other non-production environments change nothing on a card. A failed deploy of a commit CodeBook did not merge blames nobody.

Verifying a deploy

A deploy that succeeded is not the same as work that works. Under Factory settings → Policies & budgets, Commands that verify a deploy lists command runner commands — a smoke test against production, for example. After a production deploy succeeds CodeBook runs them on the deployed commit:

The card says	Meaning
DEPLOYED · verifying	The deploy succeeded; the verification commands are running.
VERIFIED	They all passed against the deployed commit.
VERIFY FAILED	One failed. The card goes back to the station that built it with the output, like a failed deploy.
DEPLOYED · not verified	They could not be run — no runner, or the command is not on its list. Nobody has checked.

With no commands listed, a card stops at DEPLOYED.

Agents can also look at the deployed application themselves: with browser checks switched on for a runner, an agent can open a production page, act on it, and report what it saw with a screenshot.

Coolify

Coolify is a source of its own. Connect a Coolify source, enter the repository the application deploys, and paste the address CodeBook shows into Coolify under Notifications → Webhook, with Deployment Success and Deployment Failure ticked.

Two things differ from the other providers, because of what Coolify's webhook is:

• The token is part of the address. Coolify cannot send a header. Treat the address like a password; make a new token if it leaks.
• It names no commit. A successful deploy carries everything merged before it; a failed one is answered for by the most recent merge not yet live.

GitHub deployments — Vercel, Netlify, Render, a GitHub Actions environment — need no source at all: they arrive through the repository's own webhook.

Incidents become cards

An incident on the map is information. A card is work. A factory can open a card for every production incident it should fix.

Open Factory settings → Events → Production incidents, choose the station the card opens at and the lowest severity that opens one, and switch it on.

The card is written as a brief an agent can start from: what production reported, the service, when it started, what the System Map links it to, the deploy that was running at the time, and what is asked — reproduce it, find the cause starting from that deploy, fix it with a test that fails first, and say what the cause was. At a station that starts by itself, the agent starts.

The rules lean towards fewer cards:

• One incident, one card. A provider repeating or reopening an incident is noted on the card; it does not open another. Once that card is finished, the incident coming back is new work.
• The same problem under another name is attached. An incident that looks like work already open in the factory is added to that card instead.
• Production only, and only from the severity you chose.
• An alert storm does not bury the board. At most ten new incident cards an hour per factory; the card says once that more are arriving, and the rest stay on the map.
• A resolve is said on the card. Nothing moves or closes the card for you: the fix and the cause still matter.

Investigate before fixing

Tick Investigate before fixing and a new incident's card is not handed straight to a developer. First, up to three of the workflow's agents look into it at once, each on one line of inquiry:

• What changed — the deploy that was running, the pull requests merged before it.
• Can it be reproduced — the smallest way to make it happen again.
• Where it comes from — where in the code the error is raised, and what makes that path fail.

They answer on the card, in parallel, and change no code. When the answers are in — or after twenty minutes — the workflow's manager reads them, says what they add up to, and takes the next step: sends the card to the station that fixes it, asks one more question, or asks a person. You see one card and one conclusion, not three separate chats.

A workflow with no manager starts the card at its station with the findings above it.

Where production incidents count

• Impact analysis. Production incidents in the last 180 days on the affected and dependent nodes — and on their modules, their services and the runtimes that run any of them — add to the same Incidents count a card's impact already reads, with their own reason: how many, how many in the last 90 days, how many are open and how many critical. There is no second rule: an incident in the last 90 days with five or more dependents is CRITICAL, and risk routing reads that level unchanged.
• The context pack. An open production incident on a seed's component is a candidate in the context pack, weighted by its severity; at most 10 are offered.
• Ask your system. The runtime_status tool answers what a runtime runs per environment, a service's runtimes, an incident's facts and the known incidents on a node. See Ask your system.
• The node sheet. A runtime shows what it is Running; a service or module shows where it Runs in; an incident shows its link and the deploy at its start; Known incidents list severity, status and the provider's link. A card node shows Incidents on components this card touches.
• The Architecture tab has two more sections: Runtimes and Production incidents.

Limits

• No model links, scores or places anything here. Links, severities, levels and runtime nodes are rules.
• Each adapter is written against the provider's documented webhook payload, one pure function per provider, not against what one delivery happened to contain.
• Incidents exist only where a source is connected. A system nobody connected has no production incidents, and its cards score lower than the truth.
• MadeBook policy violations are incident nodes on the map, sent by a linked workspace, but impact analysis does not count them.
• Deploys come only from the generic webhook's deploy event and GitHub's deployment_status.
• Removing a repository removes its deploys and the mappings onto its nodes. Incidents linked to it stay, not linked, with the sentence "The repository it was linked to was removed."

Next

System Map — the map these incidents are linked into. System impact and risk routing — what a level changes.