I've been building Codenames AI, a small web game where an LLM plays Codenames with you. Like most solo products, I glance at a Product Health dashboard when I want a quick read on whether anyone is actually playing.
One morning in June, three weeks after launching the site, the Active players tile said 64. Next to it sat 122 starts and restores. The number looked like traction. My first instinct was to treat it as confirmation and keep shipping.
That instinct did not survive the next question: which sessions were actually in that count?
The dashboard answered a wider question than I asked
I was reading Product Health as if every event in the project came from real players on the production site. The tile did not lie about its math. It counted distinct people who started or restored a game. What it could not tell me, from the chart alone, was which runtime those people were in.
I had reasons to trust the number:
- PostHog init only ran when
VITE_POSTHOG_KEYwas set. Local Vite and Playwright runs did not ship that key, so I treated laptop and E2E traffic as silent by configuration. There was noanalytics_environmentproperty yet, and no environment-conditional init path. "Do not put the key in this build" was one guardrail. - Returning users looked safe too. On production, game state restores from origin-scoped
localStorage, and PostHog keeps an anonymous ID on that same origin. Come back later and you still count as one Active player viagame_restored. We do not callidentify; continuity is browser storage on that host. I assumed testing on review URLs worked the same way: me again, already counted.
Outside PostHog, the acquisition picture did not match. Real arrivals were mostly organic Google Search. In Search Console, we had not yet hit the first “30 clicks from Google Search in the past 28 days” milestone. We had only just started posting on dev.to, so that channel was not a material source either.
Sixty-four unique players on a site that young, against a search funnel that had not cleared thirty clicks in a month, and early publishing that barely existed, was already a little suspicious. The starts/restores volume next to it made it worse. My working note was blunt: investigate further; something was minting unique players that real arrivals could not explain.
Review deploys were the hole
Review deploys (for us, Vercel preview URLs) look like the real app, often share the same analytics project key, and show up whenever you click a pull-request review link. They were not "local without a key," and they were not the same origin as codenames-ai.com.
A review hostname gets its own empty save store and its own anonymous PostHog identity, so a click-through during review can land as a new unique player (game_started) instead of folding into the production self I already knew. Without a way to separate those runtimes, that 64 was still a hypothesis about whether review-deploy traffic, and new identities on those hosts, were in the count.
That investigation became a concrete plan: stop treating every capture in the project as if it were production traffic.
An early cut disabled PostHog for E2E. Silencing one runtime would still leave review deploys sharing the key; we needed an explicit boundary instead of relying on some environments staying silent.
What should count as production?
Two fixes landed together.
Client tagging. On PostHog init, the frontend resolves an analytics_environment of production, preview, local, or e2e, then attaches it to every event and to the user profile. Hostname and the host’s build-time environment distinguish the runtimes.
Non-production traffic is excluded by dashboard filters, not by skipping PostHog init. Tagging every runtime, including ones we used to silence by omitting the key, is what makes the filter meaningful.
Dashboard filters. Product Health keeps events where analytics_environment = production OR not set, so older production events from before tagging remain visible. Newer views can use an exact production filter once tagging coverage is trusted.
The missing dimension wasn't another metric. It was the production boundary. Once that existed, Product Health could filter on it.
Healthy numbers resist questions
The harder lesson wasn't that the dashboard was wrong. It was that healthy-looking numbers are the least likely ones to get questioned.
While working on model experiments, failure exposed hidden assumptions. Here nothing looked broken, so curiosity had to do the same job: notice that the system was faithfully answering a different question than the one I thought I was asking.
How we ask the dashboard questions is a separate story. This post stays on the quieter failure mode: one project key, a review runtime that looked like production, and a number that looked clean until we asked.
What I'd check on the next dashboard
- Start by asking what question the metric actually answers, not the one you hope it answers.
- Compare it against an independent signal. If the numbers do not fit together, investigate before celebrating.
- Look for missing dimensions that collapse different kinds of traffic into one KPI: environment, internal users, bots, staging, or another hidden segment.
- Only then decide whether the fix is better tagging, better filtering, or a different metric altogether.
You do not need our dashboards or our app code to apply the pattern. Review deploys were the incident that exposed the gap here.
I cannot put a clean contamination percentage, from today’s data alone, on the period before we added tagging; the point is the missing question, not a guessed share of noise.
Takeaway: Production engineering isn't just responding to broken signals. It's occasionally distrusting reassuring ones. Metrics answer exactly the question you instrumented, not necessarily the one you think you asked.
If you'd like to see the project that inspired these lessons, you can try Codenames AI.
Top comments (16)
"Metrics answer exactly the question you instrumented, not necessarily the one you think you asked" — this is the line I keep relearning.
I'm not a developer by trade; I build internal tools for the org I work at, and our health monitor used to trust a green status code. Turned out a service could return a perfectly healthy 302 while its database was dead behind it — the shell was alive, the signal was reassuring, and it was lying.
What fixed it was exactly your instinct: stop trusting the environment's self-report. The monitor no longer reads the app's own "db: true" flag — it opens the same database and runs its own SELECT 1. Same idea as your environment tag: the number only means something once you can prove which sessions it's actually counting.
The healthy-looking metric really is the one that never gets questioned. Clean catch, and a clean fix.
That's a great example. I like the idea of making the monitor prove the dependency instead of asking the service if it's healthy. It also reduces the risk of a healthy-looking metric quietly drifting away from the reality it's meant to represent.
The drift point is the one that took me longest to respect. A self-reported "healthy" doesn't fail loudly — it just slowly stops meaning what it used to, and nobody's watching at the moment it crosses the line.
The next question that bit me was the obvious one: who proves the monitor? A monitor that quietly dies looks identical to one reporting all-clear — same silence either way. So the last layer I added requires evidence the check actually ran today (a fresh log row), not just the absence of alarms. No alarm can mean "all good" or "the thing that raises alarms is dead," and those two need to look different. Otherwise the monitor itself drifts away from reality exactly the way you're describing, just one level down.
That's a great extension. I like the distinction between "no alarms" and "evidence the check actually ran." It's the same principle one layer down: once the monitor becomes another system, it needs observable evidence of its own rather than relying on silence.
That's also why operational dashboards often track the health of the monitoring pipeline itself, not just the systems being monitored.
Exactly — and that's the point where I had to decide how far down the turtles go. If every monitor needs a monitor, you never stop.
What let me stop wasn't adding another identical layer, it was changing the axis. The watchdog that proves my monitor ran lives on the same box, so it can't survive that box dying. So the final layer isn't another watchdog there — it's a check on completely separate infrastructure that only sees the public endpoint. Its failure mode is uncorrelated with everything below it: if the whole site is down, silence there actually means something, precisely because it shares nothing with the stack it's watching.
So I didn't beat the recursion, I just made each layer fail differently from the one beneath it. Tracking the health of the monitoring pipeline works right up until the pipeline and the thing watching it can die from the same cause.
I really like "I didn't beat the recursion, I just made each layer fail differently." That's a much better framing than thinking about an infinite stack of monitors. Independence between failure modes is really what lets the recursion stop.
It's also what allows observability systems to stay trustworthy at scale, because they can provide evidence that's independent of the systems they're observing.
Right — and the one thing I've learned to watch is that independence isn't permanent. It's true the day you set it up, then shared dependencies quietly creep back in: same DNS, same cloud account, same billing card, and two "independent" layers can suddenly die from one cause without anyone changing them on purpose. Independence drifts too. So it turns into the same discipline all the way up — the thing you proved once still has to keep proving it. Really enjoyed this thread.
"Independence drifts too" is a great way of putting it. I hadn't thought about it in quite those terms, but it comes back to the same lesson again: assumptions don't just need to be designed well, they need to keep earning our trust over time.
That's an interesting consequence of AI-assisted development too. Independence is one of those properties that's easy to optimize away if the reasoning behind it only lives in people's heads. The more systems evolve through fresh eyes, whether human or AI, the more those architectural invariants probably need to be explicit so they survive well-intentioned simplifications.
This is exactly the failure mode I live in — I build almost everything with AI as a non-developer, and the most dangerous moment is when it looks at something load-bearing-but-weird and helpfully offers to "clean it up." Independence, a retry, a deliberately duplicated check — from the outside they read as cruft, and a fresh pair of eyes optimizing for simplicity will remove them with the best intentions and a green test suite.
What's helped is treating the invariant and its reason as a single unit that travels together: a short convention file the AI is told to read first, and inline notes that say not just "don't touch this" but "this is here because X; remove it and Y breaks." A fence with no sign explaining it is the first thing anyone tears down. The explicit "why" is what turns an invariant from removable-looking clutter into something a simplification pass actually respects.
Chesterton's Fence, except now the person walking up to the fence is sometimes an AI that's very fast and very confident.
The line that does the real work is "healthy numbers resist questions," because it names an asymmetry most dashboards never correct for: a metric that looks broken gets audited immediately, a metric that looks like modest, plausible traction gets shipped on. 64 active players against a search funnel that hadn't cleared 30 clicks is the kind of mismatch that's only visible if you're already cross-checking against a second, independently sourced signal, and that cross-check is what actually caught this, not a sharper question in the abstract.
Which is the part I'd want to see carried forward past this specific fix. Environment tagging closes the review-deploy class of contamination for good, but it's closing a known hole, not building the habit that found it. The next untagged runtime, an internal QA harness, a bot crawling the game, a staging environment nobody thought to tag, will fold into production the same way review deploys did, unless the acquisition-funnel cross-check becomes a standing practice rather than a one-time catch. Tagging fixes this incident. A routine second-signal comparison is what catches the next one you haven't thought to tag yet.
The honesty about not being able to back-compute a clean contamination percentage is worth keeping too, since the tempting move after a fix like this is to retroactively estimate how bad it was and publish a tidier number than the data actually supports.
Mike, this is exactly the kind of takeaway I was hoping someone would pull out of the story.
You're right that "healthy numbers resist questions" is doing the real work. I deliberately didn't lead with that, because I wanted readers to experience the investigation first. If I'd started with the principle, it becomes an essay that the incident illustrates. By letting the dashboard look believable first, the lesson only arrives after you've lived through why I trusted it.
You're also right that environment tagging solves the hole we could name, while the habit that found it is broader. Since then I've codified the tagging and built a recurring analytics review around environment splits and acquisition signals, but the more durable practice is exactly what you described: comparing against an independent signal that fails differently. The next issue probably won't be preview deployments; it'll be something nobody thought to tag.
I also left the contamination percentage out on purpose. I didn't have enough evidence to put a number on it with confidence, and even a ballpark estimate from the preview hosts would have made the story about this incident rather than the habit that found it. That noise level is also workflow-dependent: a frontend-heavy sprint, a backend-focused week, or a larger team would all produce very different levels of contamination. The transferable part wasn't a percentage. It was learning that healthy numbers still need to earn our trust through independent signals.
The choice to let the dashboard look believable first is the right call structurally, not just narratively, since a reader who's told the lesson upfront nods along without ever feeling the specific moment plausible and true came apart, which is exactly the gap that makes healthy numbers dangerous in practice. Living through the 64-looks-like-traction moment before the reveal is what makes the lesson transferable instead of just agreeable.
Glad the recurring review landed on environment splits plus acquisition signals specifically, since that's the two-signal check doing double duty, one dimension you can now tag directly and one you can't fake by mistagging. The next gap probably shows up as a third signal nobody thought to compare against yet, which is the part no amount of tagging discipline fully closes.
The approach of 'label the noise instead of hiding it' is super clean! Do you run into any issues with PostHog event quota or costs by capturing all test/preview traffic and filtering it out at the dashboard layer, or is the volume low enough that full coverage is worth the clarity?
Great question. Dashboard filters don't reduce PostHog's event quota, so "label then filter" is a clarity decision rather than a cost optimization.
For this project the trade-off is pretty easy because the volume is still small. We only send a small set of explicit product events (autocapture, pageviews and session recording are all disabled), so we're at roughly 12k events/month against PostHog's 1M free product analytics events.
Local and E2E environments are usually silent because they don't initialize PostHog unless a key is explicitly configured. Vercel preview deployments do send events and count towards quota, but there are relatively few events from them. The problem wasn't the event volume, it was that preview deployments kept minting new "players", so they could disproportionately inflate an active-player metric even while representing only a small share of total events.
If preview traffic ever became material, I'd probably solve it at ingestion (or move it into a separate project) rather than rely on dashboard filters. Until then, I'd rather keep the labels, because being able to see that "noise" is exactly what made the active-player contamination diagnosable in the first place.
The review-deploy contamination is such a clean example of something I keep running into with AI eval. You look at a number, it looks good, you almost move on — but the number is answering a different question than you think. The fix here (tag everything, don't silence anything at source) is basically what I tell myself about eval data: don't hide the noise, label it, then decide what counts.
I like that framing: "don't hide the noise, label it, then decide what counts." That's essentially where I ended up as well. Once the environment became an explicit property of the event instead of an implicit assumption, the dashboard became much easier to reason about.