What Seven Years of Production Incidents Taught Me

Every production outage teaches roughly the same handful of lessons, just wearing a different costume each time. The database that falls over looks different from the API that starts timing out, which looks different from the deploy that silently breaks a payment webhook — but if you've been on call long enough, you start recognizing the underlying pattern before the specific symptom even finishes rendering on the dashboard. After seven years of carrying a pager for production systems, a small number of lessons kept resurfacing enough times that they've genuinely changed how I architect software, not just how I respond to incidents.

1. Alerts Without Owners Are Noise

This is the single most common failure mode I've seen in monitoring setups, and it's almost never about the monitoring tooling itself. If a page or alert doesn't have a specific, named human responsible for acting on it, it gets ignored within a month — not out of negligence, but because ambiguous ownership is functionally the same as no ownership. Everyone assumes someone else is watching it, so eventually nobody is.

The fix isn't more alerts or better dashboards. It's fewer alerts, each with an explicit owner and a clear action attached. An alert that doesn't map to "this specific person does this specific thing when it fires" should either be deleted or redesigned, because an ignored alert is worse than no alert at all — it creates a false sense that the system is being watched.

2. Rollback Speed Matters More Than Deploy Speed

Teams spend enormous effort optimizing how fast they can ship a good change, and comparatively little thinking about how fast they can undo a bad one. This is backwards. Every team eventually ships a bad change — that's not a hypothetical, it's a near-certainty over a long enough timeline — and when that happens, the thing that actually determines the blast radius of the incident is how quickly you can get back to a known-good state, not how fast you deployed the bad change in the first place.

In practice, this means treating rollback as a first-class, tested capability, not an afterthought you discover works (or doesn't) for the first time during a live incident. Feature flags that can be flipped off instantly, database migrations designed to be reversible, and deploy pipelines that support a one-command revert all matter more to actual system reliability than shaving minutes off deploy time.

3. Most "Database Issues" Are Actually Query Issues

When a database starts struggling under load, the instinctive response is almost always "we need a bigger instance." Sometimes that's genuinely the right call. Far more often, profiling reveals one or two expensive queries — a missing index, an N+1 query pattern, a report running an unbounded scan — that are responsible for the majority of the load. Scaling the infrastructure in that situation doesn't fix the problem; it just buys a few more months before the same inefficient query outgrows the bigger instance too.

Before reaching for a scaling solution, profile the slow query. It's almost always cheaper, it's usually the actual fix, and unlike a hardware upgrade, it doesn't just delay the same incident from recurring later at a higher cost.

4. The Incident That Should Have Been a Non-Event Teaches the Most

The most instructive incidents aren't always the biggest outages — they're often the ones that should have been contained to a single, boring failure but instead cascaded into something much worse. A single server going down shouldn't take the whole product offline. A third-party API timing out shouldn't block unrelated functionality. When a small, expected failure turns into a large, unexpected one, the real lesson isn't about the failure itself — it's about a missing boundary somewhere in the system that let a contained problem spread.

After enough of these, you start architecting with the explicit assumption that components will fail, and designing the boundaries between them so that a failure in one place stays in one place: timeouts on every external call, circuit breakers around dependencies that aren't fully trusted, and graceful degradation instead of a hard dependency on every service being up simultaneously.

5. Postmortems Without Changed Behavior Are Theater

Writing a blameless postmortem is good practice, but a postmortem that identifies a root cause and doesn't result in an actual change — to the code, the process, or the monitoring — is just documentation of a problem that will happen again. The real value of a postmortem isn't the document; it's the specific, assigned follow-up action that closes the gap the incident revealed. If a postmortem's action items routinely don't get done, the team doesn't have a postmortem problem — it has a prioritization problem, and that's worth naming directly rather than writing more thorough postmortems that still don't change anything.

Watch for one specific pattern here: if a postmortem's recurring recommendation is "we should rebuild this," that's worth stress-testing before committing to it — see how to actually decide between a rebuild and a refactor.

6. The Riskiest Deploys Are the Ones That Feel Routine

Teams are naturally cautious around big, obviously risky changes — a major version upgrade, a schema migration, a new payment provider. Careful review, staging environments, rollback plans. The incidents that catch teams off guard are more often the "small, routine" changes that nobody thought needed the same scrutiny — a config tweak, a dependency bump, a "trivial" refactor. Treating every production change with a baseline level of caution, rather than reserving caution for changes that look scary, catches a surprising number of incidents before they happen.

What These Lessons Change in Practice

None of these lessons are exotic or require sophisticated tooling. They show up as concrete architectural habits: every alert has a named owner and a clear action, every significant change has a tested rollback path, every slow system gets profiled before it gets scaled, every external dependency has a timeout and a fallback, and every postmortem produces at least one completed action item, not just a document. Individually they're small disciplines. Compounded over years of production systems, they're the difference between incidents that are brief and contained, and incidents that spiral.

Frequently Asked Questions

What's the single highest-leverage reliability practice for a small team?

Making rollback fast and tested. It's cheaper to implement than most reliability investments, and it directly limits the damage of the mistakes every team inevitably makes, regardless of how careful the engineering process is.

How do you decide which alerts are worth keeping?

If an alert fired in the last quarter and nobody with clear ownership acted on it specifically because of that alert, it's a candidate for deletion or redesign. Alert volume should track real, actionable signal, not theoretical coverage.

Is more monitoring always better for reliability?

No — past a certain point, more monitoring without clear ownership just adds noise that makes real signals harder to spot. Fewer, well-owned alerts consistently outperform comprehensive but ambiguous dashboards.

Thinking About Reliability for Your Systems?

If your team is dealing with recurring incidents, slow rollbacks, or monitoring that generates noise instead of action, I help companies build the operational discipline that turns incidents into brief, contained events instead of spiraling ones. Explore my Cloud Infrastructure & DevOps and Technical Consulting & Architecture services, or see the full list on my services page.

Book a free 30-minute technical consultation →