On-Call Engineering & Alert Management
On-call engineering is the practice of having engineers available to respond to production incidents around the clock. Done poorly, it leads to engineer burnout, poor reliability decisions, and high attrition. Done well - with clear severity definitions, escalation paths, and alert hygiene - it creates a sustainable feedback loop that continuously improves system reliability.
Sustainable On-Call Practices
On-call sustainability is measured by alert volume. The Google SRE standard: no more than 2 pages per 12-hour shift. More than this and on-call engineers are too interrupted to investigate properly, incidents compound, and burnout follows.
Alert hygiene is the most impactful reliability improvement most teams can make. Most on-call alert fatigue comes from three sources: alerts that fire when nothing actionable is required (monitoring noise - alert when CPU briefly spikes to 90% during cron jobs), alerts that fire but auto-resolve without impact (transient flapping), and alerts for problems that always resolve themselves.
For each alert, ask: 'Is this urgent? Is it actionable? Will a human response make a difference?' If the answer to any of these is no, the alert should be removed, converted to a ticket, or made a dashboard metric rather than a page. Eliminating one useless alert that fires 5 times a night gives every on-call engineer 35 minutes of sleep back per week.
On-call rotations: typically weekly per team, with primary and secondary responders. Primary handles incoming pages and escalates to secondary if needed. Schedule should account for time zones - don't make engineers take on-call during their night when a team in another timezone can handle it. Track on-call burden per person and ensure equitable distribution.
Each model shifts more responsibility from you to the cloud provider
On-Call Best Practices
- Severity levels - P1: production down, customer-impacting, page immediately. P2: degraded service, workaround exists, respond within 15 min. P3: minor issue, respond next business day. P4: cosmetic, handle in sprint planning
- Runbooks - Step-by-step response procedures for every alert. Include: what triggered it, immediate mitigation steps, escalation contacts, links to dashboards. Runbooks reduce MTTR from hours to minutes
- Escalation policy - Clear chain: on-call engineer → team lead → engineering manager → director. Each level has a defined response window. Escalate when blocked for more than 15 minutes on P1
- Postincident metrics - Track per on-call rotation: number of pages, MTTR by severity, percentage of alerts that required human action. Report these monthly to drive improvements
- Alert ownership - Every alert has an owner responsible for maintenance. Review alert utility quarterly - silence alerts that haven't triggered in 90 days and evaluate whether they're still relevant
- On-call compensation - Many organizations provide on-call pay or compensatory time off. Document the policy clearly; uncertainty about compensation creates resentment
Quick Quiz
Tip
Tip
Practice OnCall Engineering Alert Management in small, isolated examples before integrating into larger projects. Breaking concepts into small experiments builds genuine understanding faster than reading alone.
Practice Task
Note
Practice Task - (1) Write a working example of OnCall Engineering Alert Management from scratch without looking at notes. (2) Modify it to handle an edge case (empty input, null value, or error state). (3) Share your solution in the Priygop community for feedback.
Common Mistake
Warning
A common mistake with OnCall Engineering Alert Management is skipping edge case testing - empty inputs, null values, and unexpected data types. Always validate boundary conditions to write robust, production-ready cloud code.
Key Takeaways
- On-call engineering is the practice of having engineers available to respond to production incidents around the clock.
- Severity levels - P1: production down, customer-impacting, page immediately. P2: degraded service, workaround exists, respond within 15 min. P3: minor issue, respond next business day. P4: cosmetic, handle in sprint planning
- Runbooks - Step-by-step response procedures for every alert. Include: what triggered it, immediate mitigation steps, escalation contacts, links to dashboards. Runbooks reduce MTTR from hours to minutes
- Escalation policy - Clear chain: on-call engineer → team lead → engineering manager → director. Each level has a defined response window. Escalate when blocked for more than 15 minutes on P1