Incidents, MTTR and MTBF: formulas and example
Incidents, MTTR, and MTBF are the core set of SRE reliability metrics. The number of incidents reflects the frequency of failures, MTTR (Mean Time to Restore) the speed of service recovery, and MTBF (Mean Time Between Failures) the average time between failures. Together, these metrics show platform stability, the maturity of incident management processes, and compliance with SLO/SLA.
What the Metric Measures
The volume and quality of incidents by priority (P0–P3), the speed of detection/response/recovery, and service resilience between incidents. It is advisable to track MTTD (time to detect) and MTTA (time to acknowledge) separately, as well as the share of repeat incidents and SLA breaches.
Calculation
- Incident Count = Number of confirmed incidents for the period (separately for P0/P1/P2/P3)
- Incident Rate = Incidents / Period (or per 1 million requests, per 1,000 users)
- MTTD = avg(Detection time − Incident start time)
- MTTA = avg(On-call acknowledgment time − Detection time)
- MTTR = avg(Recovery time − Start time) — within the definition of "service recovery"
- MTBF = avg(Start time of the next incident − Recovery time of the previous one)
Alternative: (Time window − Σ Downtime) / Number of incidents - Breach Rate, % = Incidents with an SLA breach / All incidents × 100
Define a policy: what counts as the "start/end" of an incident, calendar or business hours, accounting for partial degradations (weighted by traffic/users), deduplication of related events, and P0–P3 criteria.
Calculation example
In a month (720 hours) there were 4 incidents with 6 hours of recovery in total. MTTR = 6 / 4 = 1.5 hours, MTBF = (720 − 6) / 4 = 178.5 hours — about one failure a week. Cutting MTTR to 30 minutes would reduce downtime threefold.
Data Sources
Monitoring/APM/logging (metrics/traces/logs), synthetic checks, PagerDuty/On-call, ITSM/incident management (JSM/ServiceNow), CI/CD (deployments/rollbacks), status page, postmortem logs.
Interpretation
- Incident Count↑ — technical debt, weak tests/checks, high change failure rate
- MTTR↑ — slow diagnostics/escalation, lack of runbook scripts, poor-quality alerts
- MTBF↓ — fragile architecture, unstable dependencies, risky releases
- Check the impact on SLO/SLA, Error Budget, and customer metrics (CSAT, conversion)
Useful Breakdowns
- Service/endpoint; region/zone; environment (prod/stage)
- Incident type: infrastructure/deployment/config/dependency/security
- Root cause (RCA): code/changes/capacity/network/DB/cache/queues
- Team/on-call role; time of day/shift; release/canary/rollback
Dashboard Visualizations
- Trends of Incident Count, MTTR, MTBF with target corridors
- Incident timeline with deployment annotations and SLA breaches
- Heatmap by priority/service/region
- Pareto of RCAs and recurring problems
- Relationship with Uptime, Error Budget burn, latency, and 5xx errors
How to Set Goals
Examples: P0 ≤ 1/quarter, MTTR P0 ≤ 30–60 min, MTBF for the core ≥ 30 days. Implement an error budget policy, canary/blue-green releases, automated diagnostics and runbook scripts, SLO alerts on burn rate (1/6 h), and regular postmortems with CAPA.
Common Mistakes
- Double-counting related events/alerts as separate incidents
- Incorrect definition of start/stop; mixing calendar and business hours
- Ignoring partial degradations and weighting by traffic/users
- Focusing only on the average without P50/P90 and P0–P3 segments
- No link to deployments, tests, and Capacity/SLI
Questions for Discussion
- Which 3 RCAs account for 80% of incidents, and how can they be eliminated?
- Where can MTTR be reduced: detection, escalations, runbook scripts?
- What is the effect of canary releases/feature flags on the Incident Rate?
- Do alerts match SLO/SLI, and is burn rate configured?