The Production Incident Post-Mortem Prompt: Turning Raw Slack Threads and Datadog Logs into RCA Reports
When a severe production outage strikes, the immediate priority is restoring system availability. But once the alarms quiet down and traffic normalizes, Site Reliability Engineers and engineering leads face a tedious chore: writing the blameless post-mortem report. Sifting through hundreds of frantic Slack war-room messages, correlating disparate Datadog alert timestamps, and reconstructing an accurate chronological timeline often takes 3 to 5 hours of senior engineering time.
Because post-mortems must remain objective, rigorous, and blameless, dumping chat transcripts into an unconstrained LLM produces shallow, finger-pointing summaries that fail executive and engineering review. Below is a production-tested prompt architecture designed to transform messy incident chat logs and alert metrics into an industry-standard Root Cause Analysis (RCA) report.
The Anatomy of an Industry-Standard Blameless Post-Mortem
High-maturity engineering organizations (such as Google, Stripe, and Netflix) structure post-mortems around five foundational sections:
- Executive Incident Summary: Duration, customer impact (e.g., error rate percentages, affected regions), and severity tier (P0/P1).
- Strict Chronological Timeline: T0 (bug introduced) → T1 (outage detected) → T2 (incident declared) → T3 (mitigation applied) → T4 (full recovery).
- Root Cause Analysis (5 Whys): Drilling down from surface symptom to architectural or procedural root cause without assigning personal fault.
- What Went Well vs What Went Poorly: Candid operational reflection on monitoring efficacy, alerting latency, and runbook accuracy.
- Preventive Action Items: Concrete Jira tickets categorized into Prevent, Detect, and Mitigate buckets with assigned owners.
The Production Post-Mortem Synthesis Prompt
Copy and use this system prompt to process incident transcripts, Datadog alerts, and deployment logs:
You are a Principal Site Reliability Engineer (SRE) and Incident Commander.
Your Task:
Synthesize the provided raw Slack incident war room messages, Datadog alerts, and deployment logs into a formal, blameless Incident Post-Mortem Report.
Strict Guidelines:
1. Blameless Culture: Focus on system architectures, procedural gaps, automation failures, and test coverage. Never use personal names to assign blame; refer to roles (e.g., "the on-call engineer", "the deploy pipeline").
2. Chronological Precision: Reconstruct timestamps into a clear, standardized UTC timeline. Highlight the Time to Detect (TTD) and Time to Mitigate (TTM).
3. Root Cause Analysis: Conduct a rigorous "5 Whys" analysis starting from the customer-facing error down to the latent architectural condition.
4. Action Items: Every action item must follow the SMART criteria (Specific, Measurable, Assignable, Realistic, Time-bound) and be categorized into PREVENT, DETECT, or MITIGATE.
Input Context:
- Incident Metadata: Service Name, Severity (P0/P1), Affected Regions.
- Chronological Chat Logs & Metrics:
{{INSERT_SLACK_THREAD_AND_DATADOG_DATA_HERE}}
Output Format:
Return a fully structured Markdown report matching the template below.
The Standard Post-Mortem Markdown Template Output
Here is the standard output generated by the prompt when fed incident transcripts:
# Incident Post-Mortem: Core Payment Gateway Outage (P0)
**Date:** 2026-11-17
**Incident Commander:** On-Call Infra Lead
**Severity:** P0
**Impact:** 42 minutes total degradation; 34% of checkout requests returned HTTP 504. ~2,400 customer transactions failed.
**Time to Detect (TTD):** 4 minutes
**Time to Mitigate (TTM):** 38 minutes
---
## 1. Executive Summary
On November 17, 2026, between 14:12 UTC and 14:54 UTC, our primary payment processing service experienced connection pool exhaustion following the deployment of release `v3.14.2`. Customers attempting to complete purchases received gateway timeout errors (HTTP 504). Traffic was fully restored after rolling back to release `v3.14.1` and recycling the connection pools across all Kubernetes backend pods.
---
## 2. Chronological Timeline (All Times in UTC)
| Timestamp | Event / Observation | Actor / System |
| :--- | :--- | :--- |
| **14:10** | Release `v3.14.2` deployed to production cluster via ArgoCD. | CI/CD Pipeline |
| **14:12** | Datadog P0 Alert: `payment-service-api 5xx rate > 5%`. | PagerDuty / Monitoring |
| **14:14** | Incident Commander opens Slack war room `#inc-20261117-payments`. | On-Call Lead |
| **14:22** | SRE identifies PostgreSQL connection spikes reaching the 500-client ceiling. | SRE Team |
| **14:31** | Database query analysis reveals un-indexed query on `stripe_customer_tokens`. | Database Engineer |
| **14:40** | Rollback triggered: ArgoCD synced back to `v3.14.1`. | DevOps Engineer |
| **14:48** | Active connection count drops to baseline (65 active connections). | Datadog Telemetry |
| **14:54** | Error rates return to < 0.01%. Incident officially resolved. | Incident Commander |
---
## 3. Root Cause Analysis (The 5 Whys)
1. **Why did customer payments fail?**
The payment API pods timed out waiting for available PostgreSQL database connections.
2. **Why were database connections exhausted?**
Average query latency spiked from 12ms to 4,800ms, causing worker threads to hold database connections open indefinitely.
3. **Why did query latency spike?**
Release `v3.14.2` introduced a token verification query that performed a sequential table scan across 18 million rows.
4. **Why was a sequential scan performed?**
The database migration script adding the index on `stripe_customer_tokens(token_hash)` failed in staging due to a lock timeout, but the deployment pipeline did not halt.
5. **Why did the deployment pipeline proceed after a migration failure?**
The CI/CD migration step lacked a strict non-zero exit status validation check, allowing application rollout despite database script failures.
---
## 4. What Went Well vs What Went Poorly
### What Went Well:
- Automated alerts triggered within 120 seconds of customer degradation.
- Rollback execution via ArgoCD completed cleanly in under 8 minutes once authorized.
- Clear role delegation in the incident channel prevented duplicate operational commands.
### What Went Poorly:
- Staging migration failure logs were not surfaced prominently in the GitHub PR check status.
- Pods lacked circuit-breaker timeouts: backend threads blocked forever rather than failing fast.
---
## 5. Preventative Action Items
| Action Item | Category | Priority | Owner | Target Date |
| :--- | :--- | :--- | :--- | :--- |
| Add strict non-zero exit check to ArgoCD migration pre-sync hooks | PREVENT | P0 | DevOps Squad | 2026-11-20 |
| Configure database connection pool query timeout limit to 2,000ms | MITIGATE | P1 | Backend Team | 2026-11-22 |
| Add composite index `idx_tokens_hash` on `stripe_customer_tokens` | PREVENT | P0 | DBA Squad | 2026-11-18 |
| Implement automated synthetic canary checkout test in CI/CD pipeline | DETECT | P2 | QA Squad | 2026-11-27 |
Operational Checklist for War-Room Log Scrubbing
Before pasting raw chat logs into any prompt, apply these data sanitation safeguards:
- Mask PII and Authentication Secrets: Ensure session tokens, internal IP addresses, database passwords, and customer emails posted during debugging are purged.
- Include Telemetry Timestamps: Always paste exact Datadog alert firing and acknowledgment timestamps alongside the chat transcript so the model can construct an accurate timeline.
- Keep Raw Transcripts Chronological: Avoid pasting disjointed conversation snippets out of order, which can cause the model to confuse detection with mitigation phases.
Frequently Asked Questions (FAQ)
How do I extract clean Slack threads without copy-pasting hundreds of messages?
You can use a simple Slack CLI or Python script using the conversations.replies API endpoint to pull the entire JSON thread from your incident channel and format it with timestamps and usernames automatically.
Can this prompt synthesize logs from Datadog or CloudWatch directly?
Yes. If you paste Datadog alert transition events (e.g., [14:12:05 UTC] Alert Triggered: Critical 5xx Spike (Value: 34.2%)), the prompt correlates metric inflection points with developer comments in Slack.
Why is the "5 Whys" structure mandatory for blameless culture?
The 5 Whys framework prevents superficial conclusions like "Engineer X pushed a bad commit." By persistently asking why the error occurred, the investigation inevitably surfaces architectural omissions, missing automated test gates, or unclear procedural runbooks.
What model produces the highest quality post-mortems?
Claude 3.5 Sonnet and OpenAI o1 excel at timeline reconstruction and technical root cause deduction because they maintain high attention fidelity across long conversational contexts without hallucinating event sequences.
Conclusion
Post-mortems are the single most valuable mechanism for improving engineering reliability over time. By transforming messy incident war-room discussions into structured, blameless post-mortem reports with actionable preventative items, engineering teams turn operational outages into permanent systemic resilience.
Join the conversation