Your RAG copilot can't count — stop letting it try
A user asked our document-search copilot a completely reasonable question: how many d...
For further actions, you may consider blocking this person and/or reporting abuse
The inline approach is the right call, and the asymmetry between the two failure modes is the reason. A dedicated count tool fails silently when the model simply doesn't call it; the inline number fails silently only when the model ignores it, which is catchable with a structured output assertion: extract any number the model surfaces and diff it against total_matching_records in a post-processing step. The second failure mode is easier to detect and cheaper to instrument than the first. The total_matching_records sentinel for content-only searches is doing more work than it looks like — it tells downstream steps that no crisp cardinality exists, which is a different epistemic state from "zero results" and prevents the model from substituting a cosine-similarity count for a membership count.
You read the sentinel exactly as intended — 0 means "no crisp cardinality exists here", which is a different epistemic state from "zero matches", and the count note never fires on those turns, so the model is never handed a similarity-ranking size dressed up as a membership count. And yes, the asymmetry is the whole argument for inline: an ignored inline number is detectable after the fact by diffing prose against the authoritative field; a tool call that never happened leaves nothing to diff.
84 is the pre-permission number though. If the user is cleared for a fraction of those, the honest total quietly tells them how many controlled documents exist that they cannot see, which is an audit finding rather than a bug. Did you end up scoping the count to their view, or is the headline still the service-level total?
Fair push — and my table earned the question, because it makes the view gate look like the only permission step. It isn't. The count comes from the documents service queried with the requesting user's own auth token, and that query applies view permission in the service's SQL — so 84 is "documents this user can view that match the filter", not the tenant-wide total. The downstream gate is defense-in-depth: it re-confirms view on the rows actually surfaced (it exists mainly for the content lane, where the vector index is account-scoped but not view-scoped). One more guard that matters for your scenario: content-only searches never emit a count at all — the authoritative total only exists on the filtered path, which is exactly the path that's permission-scoped. You're right that a service-level total would be an audit finding rather than a bug; that's why the count is computed as the user, not about the corpus.
The pattern feels familiar. We’ve run into the same issue with RAG systems where the model ends up "reasoning" over a retrieval artifact instead of the source of truth. Counts, latest timestamps, rankings, averages they're all properties of the datastore, not the context window. Once retrieval starts doing top-k, reranking, deduplication, or permission filtering, you've already changed the dataset the LLM sees. At that point, asking it to aggregate is asking it to invent precision. The design that has held up best for us is treating retrieval as evidence and deterministic services as authority: let SQL/search engines compute facts, let the LLM explain them. It's a surprisingly useful architectural rule beyond RAG too any value that can be derived deterministically shouldn't be regenerated probabilistically. The model is the narrator, not the calculator.
"The model is the narrator, not the calculator" — you compressed my whole post into nine words and I'm keeping it. The generalization is the right one too: it's not a RAG rule, it's a rule about any value that's derivable deterministically. Retrieval as evidence, services as authority.
This is the right boundary. RAG is good at bringing the source into view, not at becoming a tiny analytics engine on top of it. I usually want counts, latest dates, and rankings to come from a deterministic query path, then let the model explain the result and cite what it used.
That's the boundary exactly — the deterministic path computes, the model explains and cites. The one thing I'd add from the trenches: make sure the deterministic number and the model's prose can't drift apart silently; that's where our bug lived.
the rename fixes this one but nothing stops the next one. in langgraph a plain state key is a shared mutable slot whose default reducer is last-write-wins, so any node can legally clobber another node's field and you only find out when a number looks wrong. what actually holds is a custom reducer on that channel that raises when two nodes write it in the same run. that turns a silent overwrite into a loud failure at graph level, instead of a naming convention everyone has to keep remembering.
No argument — as shipped it's a convention, not an invariant. The authoritative field is written by exactly one node today, but nothing at graph level enforces that; a reducer that raises on a second write in the same run would turn the next clobber into a loud failure instead of a wrong number three sprints later. One wrinkle worth naming: the original field (the surfaced count) is legitimately written twice per run — search writes it, the gate deliberately overwrites it — so a write-once reducer on that channel would fire on the designed flow. Which is the post's lesson restated at graph level: two meanings need two channels, and the invariant belongs on the authoritative one. Taking the suggestion.
The distinction between "model as narrator" vs "model as calculator" is the cleanest heuristic I've seen for RAG architecture. The date middleware point is also worth underlining — writing temporal context into persisted history (rather than injecting per-call) is a surprisingly common footgun that surfaces weeks later when a conversation resumes with a stale date.
On the eval gap you mentioned — one lightweight approach is a structured output assertion. If the model's answer includes a number from the authoritative field, extract it with a regex and diff against total_matching_records in a post-processing step. It won't catch every phrasing, but it catches the common case where the model substitutes its own count — and adds zero latency.
On the tool-vs-inline question: I'd argue you made the right call. A dedicated count_documents tool adds a round trip whose failure mode (model doesn't call it) is silent and architecturally harder to detect than "the model ignored the inline number." The inline approach, paired with the UI headline backstop, gets 95% of the way with half the moving parts.
One dimension the article hints at but doesn't quite name: retrieval artifacts are a lossy compression of the datastore, and every lossy step (top-k, dedup, permission gate) silently changes what "all" means. The fix isn't just separating state fields — it's recognizing that the retrieval pipeline and the aggregation pipeline have different correctness requirements and probably shouldn't share the same data path at all.
The regex-extract-and-diff assertion is the eval I was missing, and you're right that it's basically free — it slots into an offline eval harness as a post-processing check on real trajectories, zero runtime latency. On "different data path": partially with you. Here the authoritative count rides the same service query the search already paid for, so sharing the query is free — what the two pipelines must not share, and what actually bit us, is the state slot. "Lossy compression of the datastore" is a framing I'll be reusing.
I ran a memory and retrieval benchmark (BEAM 1M) and pulled apart every interval question my own system lost badly, the "how many days between A and B" kind. Seven of them. One retrieved nothing, one abstained. The other five were answered confidently and wrong, and here is the part that surprised me: in all five, the arithmetic was correct on the operands the model had chosen.
The clearest case, quoted verbatim from my run:
Planned to prepare for your inclusion-exclusion quiz: January 26, 2024
Started preparing for your Bayes' theorem test: April 18, 2024
Calculation: Jan 26–31 = 5 days; Feb = 29; Mar = 31; Apr 1–18 = 18; total = 83.
Flawless. It even got 2024's leap year February right. The gold answer was 84 days, from February 1 to April 25. Both endpoints were a different study session from the one the question named.
So "your copilot can't count" may be understating your own case. Mine counted perfectly. It counted over the wrong rows. That is worse news for the "just use a bigger model" reflex, and better news for your conclusion: if arithmetic competence was never the bottleneck, then nothing done to the model fixes it. Only the system of record does.
Where I would push: your caveat that this covers counts but not sums or rankings reads to me like the sharp edge rather than a footnote. COUNT(*) WHERE author = X has a system of record that defines the answer. "How many days between A and B" often does not, because the hard part is not the subtraction, it is which row is A. When I classified those five failures by the mechanism a fix would have to implement, five questions produced four different ones. Two were "several similar events, and the question names one". One was a value that had been revised, where the correct instance turned out to be the older one, which kills every recency heuristic. One was a field's value against the time it was asserted. One was event time against utterance time, where my answer quoted the right date and then computed from the date the turn was said.
That last one connects to your date middleware, and I think it generalises. Injecting "today" fixes the model not knowing now. It does not fix the model not knowing which time a retrieved record refers to. Same class of bug, one layer down.
Bounding my own numbers before anyone else has to: BEAM is adversarially constructed, so its figures are an upper bound on difficulty rather than typical behaviour. Five items is a list, not a rate. And the mechanism labels are a hand classification made from my own run artifacts. I did not have the corpus cached locally, so the evidence for each label is the wording of my own answer. Treat it as a shape, not a measurement.
One genuine open question, and it threatens my result rather than yours. When you inject total_matching_records, have you hit a case where the authoritative count is itself contestable, where the SQL predicate is a defensible reading of the user's question rather than the reading? That is where I would expect this to get hard, and my data suggests it is the durable failure mode.
"Mine counted perfectly, over the wrong rows" is the sharpest version of the point I was circling: arithmetic competence was never the bottleneck, operand selection is. To your open question — yes, and I'd call it the soft underbelly of the whole pattern. The count is authoritative for the predicate, not for the intent. Upstream of the SQL, a classifier maps the question to structured filters, and a defensible-but-wrong mapping (author vs. contributor, "created" vs. "effective" dates) produces a flawless count of the wrong set. Two things keep it survivable in practice: the cards render right next to the count, so a wrong predicate is usually visible to the user, and the phrasing ties the number to the filter ("matching documents"), not to the question. But I don't have an eval that catches a plausible predicate returning a plausible count — and your five-questions-four-mechanisms breakdown suggests that's the hard eval. Your event-time vs. assertion-time point is fair too: injecting "today" fixes the model's clock, not the record's.
Good coverage of ML patterns. I'd stress that monitoring data drift and model staleness is as important as the initial training — a model that was accurate at launch can silently degrade without proper observability.