Week 3. "The deploy is done. Everything's green. Now what am I actually supposed to be looking at?"
Previously
Runner
↓
Cache
↓
Artifact
↓
Deployment
Today
↓
Monitoring
Junior Engineer: The canary rolled out fine yesterday. 100% traffic, all healthy. I closed my laptop. Was that wrong?
Senior Engineer: Not wrong, exactly. But let me ask you something first. Your service is running on a server somewhere. Right now, this second — is it healthy?
Junior Engineer: I mean... I assume so? Nobody's messaged me.
Senior Engineer: "Nobody's messaged me" isn't an answer. It's the absence of one. That's the entire problem monitoring exists to solve.
The Thing Nobody Says Out Loud
Senior Engineer: Here's an uncomfortable fact about production systems: you cannot see them. Not directly. You're not standing next to the server, watching electricity move through it. Everything you know about whether it's healthy is a claim — something a piece of software told you, that you're choosing to trust.
Junior Engineer: That sounds obvious when you say it, but I don't think I've ever actually thought about it that way.
Senior Engineer: Most engineers don't, until the gap between "the system told me it's fine" and "the system is actually fine" bites them. Monitoring is the discipline of shrinking that gap — of making sure what you're told is close to what's actually true, and told to you fast enough to matter.
📒 Senior Engineer's Notebook
You don't monitor a system because you don't trust it. You monitor it because you can't see it. Trust isn't the issue — visibility is.
The Car Dashboard Analogy
Junior Engineer: Can you make this concrete?
Senior Engineer: Think about driving a car. You can't see the engine. You can't see the oil level, the coolant temperature, how much fuel is actually left in the tank, mid-drive. All of that is invisible to you, sealed inside metal, while you're doing 100 km/h.
So the car gives you a dashboard. Speed, fuel, engine temperature, warning lights. You're not watching the engine. You're watching proxies for the engine — numbers and lights standing in for things you can't directly observe.
Junior Engineer: And a production system is the engine. Monitoring is the dashboard.
Senior Engineer: Exactly that. And just like a car, the dangerous failure mode isn't "the dashboard shows a problem." It's "the dashboard shows everything's fine, and it's wrong." A dashboard that lies is worse than no dashboard — because no dashboard, at least, you know you're flying blind.
What Actually Gets Measured
Junior Engineer: Okay, so — what goes on the dashboard? What do you actually watch?
Senior Engineer: There's a well-known starting set, sometimes called the four golden signals. Not the only things worth tracking, but the ones that catch the most real problems the fastest.
Latency — how long requests take. Not just the average — the average can look perfectly healthy while 5% of your users wait eight seconds. You want percentiles: p50, p95, p99. What's typical, and what's the bad end of typical.
Traffic — how many requests you're actually getting. Without this number, every other number is meaningless. An error rate of 2% means something completely different at ten requests a minute versus ten thousand.
Errors — the rate of requests failing. Not just 500s — a request that "succeeds" with the wrong data is a failure your server doesn't know to report as one.
Saturation — how close your system is to its limit. CPU, memory, database connections, queue depth. A system can have zero errors right now and still be forty seconds from falling over.
Junior Engineer: That last one feels different from the others. The others are about what's happening. Saturation is about what's about to happen.
Senior Engineer: That's a sharp distinction, and it's exactly right. Latency, traffic, and errors tell you the present. Saturation is close to the only one of the four that gives you a warning before the present turns bad.
🪞 If I asked you this in an interview
"What are the four golden signals, and why those four specifically?"
Latency, traffic, errors, and saturation. Together they answer the questions that matter most during an incident: is it slow, how much load is it under, is it actually failing, and how close is it to running out of capacity. Most production problems show up in at least one of these four before they show up anywhere else — which is why they're the default starting point rather than an exhaustive list.
Monitoring vs Alerting
Junior Engineer: Is a dashboard the same thing as monitoring, then? Just — numbers on a screen?
Senior Engineer: A dashboard is monitoring's visible half. But a dashboard only helps if someone's staring at it at the exact moment something breaks. Nobody's staring at a dashboard at 3 AM.
Junior Engineer: So how does anyone find out at 3 AM?
Senior Engineer: That's alerting — the other half. You define a threshold: error rate above 5% for two minutes, latency p99 above 3 seconds, saturation above 90%. Cross that threshold, and instead of waiting for a human to notice a graph, the system pages someone. This is where PagerDuty from Episode 1 actually connects to everything we've built since — the pager doesn't go off because a human is watching. It goes off because a machine was, continuously, and a human wasn't.
Junior Engineer: So monitoring is the measuring. Alerting is the "go wake somebody up" part.
Senior Engineer: Precisely. Monitoring without alerting is a dashboard nobody's watching. Alerting without monitoring doesn't exist — there's nothing to alert on.
And dashboards aren't only useful during an outage, worth saying explicitly. A healthy-looking dashboard, watched over days and weeks, is how teams catch trends before they become incidents at all — latency creeping up 5% every release, memory usage climbing slightly with each deploy, error rate drifting from 0.1% to 0.4% over a month with nothing dramatic enough to trip an alert. Nothing there pages anyone. But someone glancing at trends, not just thresholds, catches the slow version of the same story the fast version tells at 3 AM.
📝 Production Note
Remember correlation IDs and structured logs, from Episode 1? That's not a separate topic from monitoring — it's the raw material monitoring is built from. A dashboard showing "errors spiked at 2:14 PM" tells you that something broke. Structured logs, searchable by correlation ID, are what let you find out which request, for which user, touching which service — the difference between knowing something's wrong and knowing what to actually fix.
In modern observability, engineers usually lean on three sources of truth together, not one alone: metrics tell you something is wrong — a number crossed a line. Logs help explain what happened — the specific events around it. Traces show where the request spent its time as it moved across multiple services. You'll meet traces properly once we're deep into distributed systems — for now, just know the three aren't competing tools, they're three different angles on the same question.
The Alert Fatigue Trap
Junior Engineer: This seems easy to over-do, though. Why not just alert on everything, so nothing slips through?
Senior Engineer: Because a human being can only take so many 3 AM pages before they stop trusting them. That's called alert fatigue, and it's one of the most common ways monitoring quietly fails — not by missing a real incident, but by burying it under noise.
Junior Engineer: How does that actually play out?
Senior Engineer: Say your team gets paged fifteen times a week. Twelve of those are noise — a brief blip that self-resolved, a threshold set too aggressively, a known flaky check. After a few weeks of that, the on-call engineer starts silencing pages before fully reading them. Reflex, not laziness — self-preservation.
Then, one night, page sixteen is the real one. And it gets the same half-second of attention as the twelve before it that didn't matter.
Junior Engineer: So too many alerts is almost worse than too few.
Senior Engineer: In a very specific sense, yes — because too few means a known gap. Too many means a false sense of coverage while the actual signal quietly drowns.
Two terms are worth having exact language for here, because you'll hear them constantly. A false positive is an alert that fires when nothing's actually wrong — that's the noise causing fatigue. A false negative is the opposite and more dangerous failure: something is wrong, and no alert fires at all. Tuning alerts is really just a constant tradeoff between the two — tighten thresholds to catch more real problems, and you risk more false positives; loosen them to reduce noise, and you risk missing something real. There's no setting that eliminates both.
👦 Junior Framing / 👨🦳 Senior Framing
Junior framing: "More alerts means better monitoring."
Senior framing: More alerts means more noise, unless each one is something a human genuinely needs to act on right now. A good alert is a promise: if this fires, it matters, and someone needs to do something about it immediately. Break that promise enough times and the alert stops meaning anything.
SLOs and Error Budgets
Junior Engineer: How do teams decide what "healthy" even means numerically? Is 99% uptime good? 99.9%?
Senior Engineer: This is where SLOs — service level objectives — come in. A team explicitly decides: "99.9% of requests should succeed, measured over 30 days." That's not aspirational marketing language. It's a number the team commits to, and measures against.
Junior Engineer: Why not just aim for 100%?
Senior Engineer: Because 100% is enormously expensive, and past a certain point, chasing it stops making users happier and starts just slowing the team down. 99.9% still means about 43 minutes of downtime a month are considered acceptable — not desired, but acceptable, a deliberately budgeted amount of failure.
Junior Engineer: Budgeted failure. That's a strange phrase.
Senior Engineer: It's called an error budget, and the phrase is supposed to feel strange — it's meant to change behavior. If you've used up this month's error budget on a risky deploy that caused an outage, that's a signal: slow down, stabilize, stop taking risky bets for a while. If you've got budget left, you have room to ship faster and take more chances. It turns "is it safe to deploy" from a gut feeling into an actual number.
🚨 Beginner Mistakes
"I'll alert on CPU usage being high." High CPU isn't inherently bad — sometimes it means the system is efficiently using the resources it has. Alert on the thing that actually matters to users: is latency degrading, are requests failing. CPU is a diagnostic detail, not the actual problem.
"I'll set every alert threshold as tight as possible, to catch things early." This is how alert fatigue starts. A threshold that fires on normal, healthy variation trains people to ignore it — right up until the day it fires for something real.
"The dashboard looks fine, so we're done watching." A dashboard is a snapshot of now. The four golden signals can look perfectly healthy one minute before a slow memory leak crosses a threshold. Monitoring is a continuous practice, not a one-time check after deploying.
🏢 Office Reality
- "We're flying blind on this service." No meaningful monitoring exists for it — nobody would know if it broke until a user complained.
- "That alert is noisy." It fires often enough, for things that don't actually matter, that people have started ignoring it. A polite way of saying "this alert has stopped doing its job."
- "We burned our error budget." The team exceeded their allowed failure rate for the period — often the trigger for pausing risky launches until things stabilize.
- "Check the dashboards before you page anyone." Standard on-call instinct — confirm what's actually happening before waking up a second person for something that might resolve on its own.
The Alert That Almost Wasn't
Junior Engineer: Has an alert actually caught something real for you — not in theory, an actual time?
Senior Engineer: More than once, but here's a clean one. A saturation alert — database connection pool climbing toward its limit — fired quietly on a Tuesday afternoon. No errors yet. No user complaints. Latency barely moved.
Easy to ignore. Nothing was actually broken yet.
Junior Engineer: But someone looked anyway.
Senior Engineer: Someone looked anyway, because the alert existed and the team trusted it enough not to reflexively dismiss it — which, notice, is the entire alert fatigue conversation paying off in real time. Turned out a recent deploy had a connection leak — every request opened a database connection and a rare error path failed to close it. Slow leak. Would've taken maybe three more hours to actually exhaust the pool and start failing every single request, in the middle of the evening peak.
Junior Engineer: So the alert fired before anything was actually broken.
Senior Engineer: That's the entire point of saturation as a signal. It's not telling you what's wrong right now. It's telling you what's about to be wrong, while there's still time to do something other than panic.
🎯 Interview Perspective
Interviewer: How would you decide what to put an alert on?
Weak answer: Alert on anything that could go wrong.
Strong answer: Alert on symptoms that directly affect users or clearly predict an imminent failure — elevated error rate, degraded latency, saturation approaching a limit — not on every internal metric that could theoretically be interesting. Every alert should represent something a human genuinely needs to act on right now; anything less specific belongs on a dashboard for investigation, not in someone's pocket at 3 AM.
🎤 Explain It In One Minute
Imagine explaining this to a teammate — without using the words monitoring, alerting, or SLO.
Why is "the deploy went fine" not the same claim as "the system is healthy"?
If you can answer that without reaching for the vocabulary, you understand the idea driving this entire episode.
Whiteboard Moment
Production system running (invisible to you directly)
↓
Metrics collected continuously
(latency, traffic, errors, saturation)
↓
Dashboards — visible, but nobody's always watching
↓
Thresholds defined against SLOs
↓
Threshold crossed → alert fires → someone paged
↓
Structured logs + correlation IDs → find the specific cause
↓
Fixed, or escalated toward an incident
Junior Engineer: So this whole episode is really just: you can't see it, so you measure it, and you don't wait for a human to notice the measurement.
Senior Engineer: That's the entire discipline in one sentence. Everything else — golden signals, SLOs, error budgets — is just making that sentence precise enough to actually act on.
What You Should Be Able to Explain Now
(Without looking at Google)
Can you explain:
- Why "the deploy succeeded" and "the system is healthy" are different claims?
- What the four golden signals are, and why saturation is different from the other three?
- The difference between monitoring and alerting?
- Why too many alerts can be more dangerous than too few?
- What an SLO and an error budget actually are, in plain terms?
- Why saturation alerts can fire before anything is technically broken yet — and why that's the point?
If yes — you understand production visibility the way it's actually practiced, not just as a dashboard you glance at after a deploy.
Junior Engineer: Okay, so say an alert does fire. For real, not a false alarm. What happens next?
Senior Engineer: Now we're back exactly where Episode 1 started — Friday, 7:02 PM, PagerDuty going off. Except this time, you're not watching from the outside. You already know what a webhook is, what a runner is, what an artifact is, how it got deployed, and what the dashboard was supposed to be telling someone the whole time.
Junior Engineer: So next time, I'm not just shadowing. I'm actually part of figuring it out.
Senior Engineer: That's the idea. Incident response, properly this time — from the inside.
(End of Episode 6)
Top comments (0)