A Systematic Workflow for Debugging Live Production Incidents

A production incident is not a bug ticket. Customers are affected right now, and the cost of a wrong guess is a second outage stacked on the first. This page is the workflow: triage first, hypothesize before touching code, separate the hotfix from the root-cause fix, reproduce before patching, and verify against the original failure before declaring victory.

Step 1 — Triage before touching code

The first five minutes are for understanding scope, not for editing files. Answer three questions before anything else:

1. What is actually broken? A spike in 5xx responses, a failing health check, a customer-reported outage, a background job stuck in a retry loop — these are different problems with different urgency.
2. How many users are affected, and how badly? A checkout failure on 100% of traffic is not the same incident as an image thumbnail failing to render for 2% of users.
3. Is it getting worse? A flat error rate can wait for a careful fix. A climbing error rate means you need to stop the bleeding now, even with an imperfect fix.

Business — the reason triage comes first: twenty minutes spent hunting for the "correct" fix while the error rate keeps climbing costs more in lost orders than an ugly-but-safe hotfix shipped in three minutes and cleaned up later. Severity determines whether "correct" or "fast" wins. Treating every incident as an opportunity for a thorough fix is the most common mistake an agent makes under pressure.

Technical — establish severity mechanically: pull the last 15 minutes of the relevant metric (error rate, p99 latency, health-check status) and compare it to the same window 24 hours and 7 days ago. A 3x deviation from both baselines with an upward slope is SEV-1 territory regardless of absolute traffic volume. Do not trust a single dashboard snapshot — a flat one-minute average can hide a spike that started 90 seconds ago and is still accelerating.

Step 2 — Read logs and stack traces to form a hypothesis, not a guess

The difference between resolving an incident in ten minutes and thrashing for an hour is almost always this step. Do not open the code and start changing things that "look wrong" — read the evidence first.

Start with the stack trace a monitoring alert might attach:

TypeError: Cannot read properties of undefined (reading 'totalCents')
    at calculateOrderTotal (order-pricing.ts:42:18)
    at applyDiscountCode (order-pricing.ts:71:9)
    at POST /api/checkout/finalize (checkout-handler.ts:118:22)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)

And the surrounding log lines:

2026-09-07T14:02:11Z ERROR checkout-handler order_id=None discount_code=WELCOME10 msg="finalize failed"
2026-09-07T14:02:14Z ERROR checkout-handler order_id=None discount_code=SAVE20 msg="finalize failed"
2026-09-07T14:02:19Z ERROR checkout-handler order_id=None discount_code=WELCOME10 msg="finalize failed"

From this, state a falsifiable hypothesis: "applyDiscountCode receives an undefined pricing object whenever a discount code is present, because calculateOrderTotal was refactored to return early on a new pricing branch and the caller wasn't updated." That's testable. "Something's wrong with checkout" is not.

Business — why guessing is expensive: a wrong guess deployed during a live incident is not neutral, it is a second unverified change stacked on an already-broken system. If it doesn't fix the problem, you've burned a deploy cycle and muddied the timeline. If it fixes the symptom by coincidence, the eventual root-cause fix will look unnecessary and nobody will trust it. A hypothesis, by contrast, is falsifiable — check it against the evidence before writing any code.

Technical — mechanically forming the hypothesis: (1) find the exact line and function where the exception originated — not the handler at the top, the deepest frame in your own code; (2) grep the surrounding log lines for what's common across every failure (here: order_id=None and a discount code present on every failing request, absent on succeeding ones); (3) check recent commits touching that file — git log --oneline -- order-pricing.ts — and correlate the incident start with a deploy timestamp; (4) only once all three line up, write the one-sentence hypothesis above. If the deploy timestamp doesn't correlate, the hypothesis is probably wrong — look for a different trigger (traffic pattern, expired credential, upstream dependency).

Step 3 — Hotfix vs. root-cause fix

These are two different deliverables. Conflating them is how incidents drag on.

• Hotfix: the smallest, safest change that stops user-visible harm right now. It is allowed to be inelegant. It is allowed to disable a feature rather than fix it. Its job is to buy time.
• Root-cause fix: the change that makes the failure mode structurally impossible to recur, not just this specific trigger. It goes through normal review, has a test, and does not need to ship under incident pressure.

For the example above, a hotfix might be a guard clause that prevents the crash:

 function applyDiscountCode(pricing: OrderPricing, code: string) {
+  if (!pricing) {
+    logger.error("applyDiscountCode called with undefined pricing", { code })
+    throw new CheckoutError("PRICING_UNAVAILABLE", { retryable: true })
+  }
   const discounted = pricing.totalCents - lookupDiscount(code)
   return discounted
 }

This stops the crash — checkout now fails cleanly with a retryable error instead of throwing — but doesn't explain why pricing was undefined. The root-cause fix is separate: fix the early-return branch in calculateOrderTotal so it always returns a valid OrderPricing object, then add a test covering the discount-code path specifically, since that's the path the early return skipped.

Business — why the two are never the same commit: shipping a hotfix as the final answer means the underlying bug ships again the next time a slightly different trigger hits it. Treating the root-cause fix as urgent as the hotfix means rushing a structural change through review under incident pressure — exactly when mistakes compound. Ship the hotfix to stop the bleeding, then use the time it bought you to fix the root cause properly.

Technical — sequencing the two: deploy the hotfix, confirm the metric from Step 1 has recovered, then open the root-cause fix as a normal follow-up with its own review and test. Don't bundle it into the same deploy as the hotfix — a bundled deploy makes it harder to tell which change fixed what if something still isn't right.

Step 4 — Write a minimal reproduction before you patch

Before touching calculateOrderTotal, write the smallest test that reproduces the failure:

test("applyDiscountCode does not crash when totals include a discount code", () => {
  const pricing = calculateOrderTotal({ items: [{ priceCents: 1999, qty: 1 }], discountCode: "WELCOME10" })
  expect(() => applyDiscountCode(pricing, "WELCOME10")).not.toThrow()
})

Confirm it fails with the same TypeError, at the same line. If it fails differently, the hypothesis from Step 2 is wrong or incomplete — go back before writing the fix.

Step 5 — Verify against the original failure mode, not just the new test

After patching, re-run the reproduction from Step 4 and confirm it passes. Then go one step further: re-check the original evidence from Step 2 — the log pattern, the stack trace — against a request shaped like the ones that failed in production. A fix that only satisfies a newly written test can still miss the real-world trigger if the reproduction was slightly off. Closing an incident means the original failure mode is gone, not that a test is green.

Postmortem template

Every SEV-1 or SEV-2 incident gets a short postmortem before it's considered closed:

## Incident: [one-line description]
- Detected: [timestamp, how — alert / customer report / health check]
- Severity: [SEV-1 / SEV-2 / SEV-3]
- Duration: [start → hotfix deployed → resolved]
- Impact: [who / how many / what they experienced]
- Root cause: [one paragraph, the actual mechanism]
- Hotfix: [what shipped first, when]
- Root-cause fix: [what shipped after, link to PR]
- Follow-ups: [monitoring/alerting gaps, test coverage added]

Keep it short enough that the next reader — human or agent — understands the mechanism in under a minute. The follow-ups section matters most: an incident without a follow-up action (a new alert, a new test, a guard clause elsewhere) is one that's likely to recur under a different trigger.