Incidents

An incident is what WireWatch opens when a monitor stops passing. It is the object every alert, every escalation and every status-page entry is about, and it is the thing you acknowledge, assign, silence and eventually close.

Two axes, one state

An incident is written by two hands, and they never touch the same columns.

Detection writes the machine status, which takes exactly two values: open while the monitor is failing, resolved once detection has seen it recover. Nothing you do in the dashboard changes it. You cannot mark an incident resolved by hand, and you cannot un-resolve one.

You write the human overlay — acknowledged, assigned, snoozed, closed, marked a false positive. None of those touch status.

The state you see is derived from both, every time it is read. It is not stored anywhere, so the two axes have no third place in which to disagree.

The five states, in the order that decides them

The derived state is the first of these that applies, left to right:

false_positive > closed > resolved > acknowledged > open

State	What it means	Terminal
false_positive	somebody marked it a false alarm	yes — nothing in the product undoes it
closed	somebody archived a resolved incident	no — reopen puts it back
resolved	detection saw the monitor recover	no — it stays until somebody closes it
acknowledged	somebody said they have it, and the monitor is still failing	no
open	nobody has touched it, and the monitor is still failing	no

Precedence is what makes an incident that is both resolved and marked a false positive read as false_positive: the flag wins over everything below it. The same rule decides what a filter returns, so a list filtered by one state and the state shown on the incident itself can never disagree.

Severity

An incident carries a severity, and there are two stored values:

Severity	What it means
partial	some of the vantages that checked see the target failing and at least one does not; an absence (no_data) incident is also partial
full	every location that checked agrees the target is down, including a monitor checked from a single location or agent

A third value, high, is legacy: incidents from before WireWatch had cross-region quorum carry it. It is read two different ways, and the difference can cost you an alert.

• In a Service’s derived status it counts as a full outage, so one high member reads down.
• For notification routing it ranks as partial, not as full. A channel whose min_severity is full is therefore not notified about a high incident.

If you filter channels on min_severity, reason about high as a partial outage.

What you can do to an incident

Eight write actions, plus reading:

Action	Route	Permission	What it changes
Read	GET /orgs/:orgId/incidents	incident:read	nothing
Acknowledge	POST /orgs/:orgId/incidents/:incidentId/ack	incident:ack	records who took it and when; a second one does nothing
Comment	POST /orgs/:orgId/incidents/:incidentId/comment	incident:ack	appends a note to the timeline
Snooze	POST /orgs/:orgId/incidents/:incidentId/snooze	incident:ack	silences notifications until an instant you choose
Unsnooze	POST /orgs/:orgId/incidents/:incidentId/unsnooze	incident:ack	clears the snooze
Assign	POST /orgs/:orgId/incidents/:incidentId/assign	incident:manage	names an owner, who has to be a member of the organisation
Close	POST /orgs/:orgId/incidents/:incidentId/close	incident:manage	archives a resolved incident
Reopen	POST /orgs/:orgId/incidents/:incidentId/reopen	incident:manage	un-archives a closed one
Mark false positive	POST /orgs/:orgId/incidents/:incidentId/false-positive	incident:manage	flags the whole incident a false alarm

Each of them appends an entry to the timeline attributed to you, except where it has nothing to do: a second acknowledgement and a reopen on an incident that is not closed both leave the timeline alone. None of them touches status.

Who may do what

Permission	Held by
incident:read	Owner · Admin · Editor · Viewer · Billing
incident:ack	Owner · Admin · Editor
incident:manage	Owner · Admin

Reading is held by every role, Billing included: a Billing user sees your incidents.

The dashboard shows every action to every role. It does not hide or grey out the ones your role lacks, because it does not know your role — the API is the gate. So a Viewer sees the same toolbar an Owner does, and finds out by pressing: the request comes back 403 forbidden and the message is shown inline above the actions. That is working as intended, not a bug in the page you are on, and it is the one refusal on this page that is about your permissions.

Why an action was refused

Close acts only on a resolved incident. Ask to close one the dashboard still shows as open and you get 409 — not 403, and there is nothing wrong with your permissions. Close archives something detection has already seen recover. For an incident that is still failing, acknowledge it instead: that is the action that says a human has it.

Reopen on an incident that is not closed is a no-op, not an error. It returns the incident unchanged, appends nothing to the timeline and does not fail. Do not code against an error here; there is none to catch.

Mark false positive applies from any state, and cannot be undone. It carries no status guard, so it can be recorded on an open, acknowledged, resolved or closed incident alike — and nothing in the product clears the flag afterwards. It also removes the incident from your response metrics. Treat it as permanent, because it is.

A snooze expires, and then the incident notifies again. Snoozing sets an instant; until that instant passes, nothing is created for the incident. Once it passes, alerts resume as before — a snooze defers. A maintenance window is the other kind of silence: it discards, for good. The whole list of reasons an alert did not arrive is on Suppression.

The timeline

Every incident carries an append-only timeline. Nothing on it is edited and nothing is removed: an entry you regret is corrected by a later entry, never by rewriting the earlier one. That is what makes it usable as a post-incident record.

It holds both kinds of event side by side, and a human entry names the user who caused it while a machine entry names nobody, because nobody did.

The whole set, and there are no others: acknowledged, assigned, closed, commented, escalated, false_positive, opened, reopened, resolved, severity_changed, snoozed, unsnoozed.

Written by	Events
detection	opened, severity_changed, resolved
the escalation sweep	escalated
you	acknowledged, assigned, commented, snoozed, unsnoozed, false_positive, closed, reopened

The escalation entry is the one people are surprised by. It is a machine entry — it carries no actor — but detection does not write it: it is the notification ladder moving the incident to its next level. You will see it on the timeline of any incident that got that far, and the screen filters nothing out.

MTTA and MTTR

Three figures over the incidents opened inside a window you choose:

• MTTA — the mean number of seconds from an incident opening to its first acknowledgement.
• MTTR — the mean number of seconds from an incident opening to detection resolving it.
• Sample count — how many incidents the window holds.

Two things about them are easy to misread, so both are stated together:

All 3 figures exclude a false positive; mttaSeconds and mttrSeconds are null with no sample.

That null is not a perfect score, and a dashboard that draws it as a zero is lying to you. It means there was no sample: nothing in the window was ever acknowledged, or nothing in it was ever resolved. An incident that opened inside the window and has not been acknowledged is counted in the sample and left out of MTTA, so the two numbers are means over different sets and are meant to be.

Deleting a monitor resolves its open incidents

Delete a monitor and every one of its open incidents is resolved in the same operation. Each one gains a resolved event on its timeline carrying the cause monitor_deleted, so the record says why it ended rather than leaving you to guess. The incident then behaves like any other resolved one: its state reads resolved, and Close accepts it.

Only that monitor’s incidents move, and only the ones that were open. An incident that had already resolved keeps its original resolvedAt, and another monitor’s open incident is untouched.

Nothing else is deleted. The monitor row, its results and its incident history are all preserved — deleted_at is set, and the incidents stay readable in the list with resolved state.

You no longer have to mark anything a false positive to get rid of it. That flag records a false alarm, and using it on an incident where detection was right is a lie your history keeps. Reserve it for incidents that really were false alarms.

Its resolvedAt is the moment you deleted the monitor, so it enters MTTR as a time-to-delete rather than a time-to-recover.

The alert a deletion sends says monitoring stopped

If an alert channel had already received the “opened” alert for that incident, the same channel hears about the close on the notifier’s next pass — and what it hears is that monitoring stopped, never that the service recovered. The subject reads [Wirewatch] <monitor> — MONITORING STOPPED and the body’s first line says the monitor was deleted while the incident was open, so nobody on call reads a recovery for a service that may still be down.

The audience is the channels your notification policy routes to — e-mail, Slack, webhook, PagerDuty, Opsgenie. Status-page subscribers are not involved: they receive the updates you post, not incident events, so a deletion is invisible to them.

The alert is still sent, and that is deliberate rather than incidental. PagerDuty and Opsgenie close their alert on this event and on nothing else — the alert is keyed by the incident’s id, the monitor is gone, and no later event will ever carry that key. Withholding the close to spare somebody a misleading word would leave an alert nobody can ever close.

Two details worth knowing:

• A channel that never heard the opening hears nothing now. The close only reaches a channel that already had a notification for that incident, so deleting a monitor never introduces you to an alert you had not already been receiving.
• A webhook channel’s JSON body is unchanged. It carries the event and the incident’s facts and renders no wording at all, so there is nothing there to correct — see Payload.