From 3:00 AM Panic to Confidence: How I Use AI During On-Call Incidents

In this blog post, we will see how I use AI to speed up incident investigation without letting it take over the decisions that need a human. It is 3:00 AM. Your phone starts making that familiar PagerDuty noise. You open the alert with half-open eyes. Error rates are climbing. Slack is already active. The incident commander wants an update. Depending on the severity, your director or CTO may also join the call. Every developer who goes on call will face this situation at some point. I have faced it a few times a year. The first time, panic is normal. You do not know where to start, which dashboard to open, or how to explain the issue while you are still investigating it. Experience teaches you how to stay prepared. AI can reduce some of that early morning panic too. It will not bring the panic factor down to zero, and it should not replace the engineer. But it can remove the first few minutes of searching, tab switching, and collecting context. The goal is simple: move from panic mode to confidence mode faster. I Started With a Prompt in My Notes I started with something small, before custom skills became common in coding harnesses. I kept one incident prompt at the top of my notes folder. I also pinned it in my clipboard manager. When an alert arrived, I filled in the blanks and launched the investigation: I received this alert: <PagerDuty or Slack alert link>. Context: - Service: <service name> - Environment: <environment> - Region: <region> - Error or symptom: <error details> - Investigation window: past <n> hours - Runbook: <runbook link> Start investigating the issue. 1. Analyze the relevant Splunk logs and dashboards. 2. Check recent deployments, configuration changes, and feature-flag changes. 3. Check upstream and downstream dependencies. 4. Check cloud-provider status pages and internal maintenance announcements. 5. Search PagerDuty history and incident records for similar symptoms. 6. Use parallel agents for independent investigation tracks where useful. R