Designing a Safe, Automatic Deploy Rollback Strategy

A rollback plan decided during an incident is not a plan — it is a guess made under pressure, by whoever is awake, using whatever commands they remember. Design the rollback path at the same time as the deploy path, before either one ships. This page covers what "decided in advance" means: what a rollback actually reverts, what it cannot revert, what should trigger it automatically, and when rolling forward beats rolling back.

Decide the rollback plan before the deploy, not during the incident

Business — Incident cost is dominated by time-to-mitigate, not time-to-diagnose. A team that improvises rollback mid-outage spends its first minutes deciding whether to roll back, which version was last known good, and who can run the command. Every one of those minutes is customer-facing downtime. Deciding the plan in advance turns a judgment call into a lookup.

Technical — Before merging a deploy, the pipeline should already have answers to: what is the last known-good artifact, what exact command reverts to it, what triggers that command automatically versus needs a human to confirm, and how long the rollback takes end to end (redeploy, cache warm-up, load-balancer propagation). If any answer is "we'd figure it out at the time," the deploy isn't ready. Record the last-good artifact reference as part of the deploy record — never rely on someone remembering which tag was stable.

Code rollback vs. data rollback

These are not the same operation, and treating them as one is the most common rollback failure.

Business — A code rollback undoes a bad decision. A data rollback undoes a bad decision after other data has already been built on top of it — new rows, orders processed under the new schema. Undoing a decision is cheap; undoing consequences is expensive. The data side needs to be designed for reversibility long before an incident happens.

Technical — The specific danger is a forward-only migration: a schema change (dropping a column, renaming a column, changing a type in place) applied in the same deploy as code that the old artifact cannot run against. If that migration ran and the code is then rolled back, the previous artifact starts erroring immediately — it queries a column that no longer exists. The code rollback "succeeded" and the incident got worse.

The fix is the expand/contract pattern, applied across at least two separate deploys:

1. Expand. Add the new schema element alongside the old one (new column, new table, new enum value). The old code still works — it ignores the new element. Deploy this. It is safe to roll back on its own, because old code never depended on it.
2. Migrate. Deploy the code that writes to both the old and new elements (dual-write), and backfill historical data into the new element. Old code paths still function unchanged.
3. Contract. Only after the new code has been running successfully for a full deploy cycle — and a rollback to the previous step is still possible — remove the old schema element in its own deploy.

At every step, the currently-deployed code and the currently-deployed schema must both work with whatever the previous deploy's code and schema looked like. That is the definition of a safe migration: it does not require the rollback to also revert data, because there is no window where old code encounters new-only structure.

What a good automatic rollback trigger looks like

A rollback that requires a human to notice a problem, then decide, then act, is slower than one that fires on its own. Three common trigger signals, and why each needs care:

• Error-rate threshold — the fraction of requests returning 5xx (or the app's own error class) crosses a set percentage. Good for catching broad breakage; noisy if the threshold is too tight relative to normal baseline variance.
• Latency threshold — p95 or p99 response time crosses a set ceiling. Good for catching resource exhaustion or a slow query introduced by the new code; can false-positive during unrelated load spikes.
• Failed health check — a dedicated endpoint (or synthetic transaction) starts failing. Good for catching total breakage or dependency failure; blind to partial degradation that doesn't touch the health-check path.

Business — No single signal is sufficient on its own, and none should fire a rollback the instant it crosses the line. One slow request or one dropped connection is noise, not evidence. Rolling back on noise trains the team to distrust the automation, which defeats the point of having it.

Technical — Pair every threshold with a fixed observation window: the condition must hold for N consecutive checks before the rollback fires, not on the first sample past the line — "error rate above 5% for 3 consecutive 1-minute windows," not "error rate above 5%." The window should be long enough to filter a transient blip but short enough that time-to-rollback stays under the incident's acceptable damage budget. Combine signals with OR for triggering (any one sustained breach is enough to roll back), but require the same signal to clear for the same window length before considering the system recovered.

Rolling back vs. rolling forward

Rollback is not always the right move. Rolling forward — shipping a new, corrective deploy on top of the broken one — is sometimes faster and safer.

Business — Roll back when the fix is unknown, unclear, or would itself need testing under pressure — a state already proven stable in production is the lower-risk path. Roll forward when the previous version is also broken in a way that matters (missing a security patch, or the incident sits in a migration step a rollback can't cleanly undo per expand/contract) and the fix is small, well-understood, and already reviewed.

Technical — If the fix is a one-line config change or reverting a feature flag rather than the whole deploy, roll forward — it's often faster than a full artifact redeploy. If the fix requires new code untested against production traffic, roll back first to stop the bleeding, then roll forward with the real fix after normal review. Never treat "roll forward with an untested fix" as the incident response itself — that just deploys a second unknown on top of the first.

A canary deploy with an automatic abort condition

A canary shifts a small percentage of traffic to the new version, watches it against the trigger conditions above, and either promotes or aborts automatically — no human has to be paged just to advance a healthy deploy.

canary:
  steps:
    - traffic_percent: 5
      hold_minutes: 10
    - traffic_percent: 25
      hold_minutes: 10
    - traffic_percent: 50
      hold_minutes: 15
    - traffic_percent: 100

  abort_conditions:
    - metric: error_rate
      threshold: 0.05          # 5%
      consecutive_windows: 3
      window_minutes: 1
    - metric: p99_latency_ms
      threshold: 800
      consecutive_windows: 3
      window_minutes: 1
    - metric: health_check
      status: failing
      consecutive_windows: 2
      window_minutes: 1

  on_abort:
    action: rollback_to_previous_artifact
    notify: on-call

Each step holds traffic at that percentage for the full window before advancing — the fixed observation window from the trigger design above, applied per step. If any abort condition is sustained for its consecutive_windows, the pipeline halts the rollout and executes the rollback automatically; it notifies on-call rather than waiting for approval.

Common pitfalls

• No documented last-known-good reference. If the rollback target has to be figured out during the incident, the plan doesn't exist.
• A forward-only migration shipped in the same deploy as the code that depends on it. This is the single most common way a "successful" code rollback still leaves the system broken. Use expand/contract.
• Thresholds with no observation window. A trigger that fires on one bad sample will roll back healthy deploys and erode trust in the automation.
• Treating rollback and roll-forward as interchangeable. Pick based on whether the previous version is actually safe to return to and whether the fix is trusted enough to ship without a full review cycle.
• A canary that requires a human to advance each healthy step. That reintroduces the exact 3am scramble automatic rollback is meant to remove — automate both the promotion and the abort.