Your RAG copilot can't count — stop letting it try
A user asked our document-search copilot a completely reasonable question: how many documents a specific person authored. The copilot answered with a number. Confidently. With a nice sentence around it.
The real answer, computed by the database, was 84. The copilot said — because it did what every RAG system does when you ask it an aggregation question: it counted what it could see.
Here's the claim I want to defend, because I think a lot of us are shipping this bug right now:
💡 An LLM should never compute an aggregate. Not counts, not sums, not "most recent". If the question is arithmetic over the corpus, the answer comes from the system of record — the model's only job is to phrase it.
🔢 Why the model's count was doomed before it started
This is a multi-agent document-search copilot for a quality-management product, built on LangGraph and Bedrock. When a metadata question comes in ("documents authored by X"), a search agent asks the documents service for matching rows, reranks and permission-gates them, collapses multiple versions of the same document into one card, and surfaces at most 30 results.
Every one of those steps is correct for search. Every one of them destroys the count:
| Stage | What it does | What "how many" means here |
|---|---|---|
| Documents service (SQL) | runs the filter against the database | true total: 84 |
| Retrieval | returns the top-k rows | capped |
| View-permission gate | drops rows this user can't see | fewer |
| Version collapse | 5 versions of one doc → 1 card | fewer still |
| Surfaced set | at most 30 cards | ≤ 30, always |
The model sits at the bottom of that funnel. When it "counts", it counts the last row of the table. The user asked about the first row. Everything in between is retrieval hygiene that silently turned a database question into a context-window question.
And this is the part that makes it worse than a crash: the answer is plausible. A count that's capped at 30 and deduped looks like a real number. Nobody squints at "you have 27 documents" the way they'd squint at a stack trace. In a quality-management context, someone might put that number in a report.
🪤 The twist: we had the right number the whole time — and overwrote it
This is the bit that stung. The documents service returns the true total with every filtered query. The search node even stored it in agent state, in a field called total_available_results.
Then, a few nodes downstream, the permission-gate node overwrote that same field with the surfaced count — deliberately, and for a good reason:
return {
"sources": diversified,
# Honest "documents found": distinct logical documents after the view gate and
# version-collapse, so the headline count matches the cards (5 versions → 4 docs),
# not the service's version-row total.
"total_available_results": len(diversified_all),
...
}
That comment is right! If the UI says "found 12" and shows 12 cards, the numbers should agree. The bug wasn't either write in isolation — it was one state field carrying two different meanings at two different points in the pipeline. Upstream it meant "matches in the database"; downstream it meant "cards on screen". Whoever read it last got whichever meaning was freshest.
💡 A state field whose meaning mutates mid-pipeline is a bug factory. If a value is authoritative, give it its own field and never let anything downstream touch it.
🔧 The fix: the database counts, the model narrates
So that's exactly what I did — preserved the service's true total in its own state field, threaded it to the two places that answer the user, and left the surfaced count alone for the card list and pager.
First, the state gets an untouchable field:
class DocumentsSearchState(OrchestratorState, total=False):
...
total_available_results: int # surfaced: view-gated, version-collapsed, capped at 30
# The documents service's true match total for the filter, before the 30-cap and
# version-collapse. Authoritative for a "how many" count; 0 on a content-only
# search, which has no exact total.
total_matching_records: int
Then the outcome node — the one that turns search state into both the user-facing status line and the message the agent reasons over — uses the authoritative number in both:
total_matching = state.get("total_matching_records") or 0
headline_total = total_matching if total_matching > total else total
found_msg = (
f"Showing {len(sources)} of {headline_total} matching documents"
if headline_total > len(sources)
else f"Found {len(sources)} matching documents"
)
...
count_note = (
f" {total_matching} documents match in total "
"(authoritative count; use this for a count question)."
if total_matching > len(sources)
else ""
)
return f"Returned {len(sources)} document cards to the user.{count_note} ..."
And the agent's prompt closes the loop with an explicit instruction:
For a "how many" question, answer with the authoritative total the tool
reports ("N documents match in total"), never the number of cards shown,
which is capped.
Notice the belt-and-braces structure, because it matters: the number goes into the agent's context (so the model can phrase the answer around it) and into the user-facing headline ("Showing 5 of 84 matching documents"). Even on a turn where the model ignores the note and free-styles a count, the UI is showing the true total right next to it. The prompt asks; the headline doesn't have to.
Three small tests pin the behavior: the count note appears when there are more matches than cards, disappears when everything is shown, and never appears for a content-only semantic search — because a similarity ranking has no crisp "matches" set, so pretending it has an exact total would be the same lie in a new costume.
📅 The sibling bug: "the last 10 days" of what?
Same release, same root cause wearing a different hat. Users asked for "documents created in the last 10 days" and the copilot had to turn that into a date filter — without knowing what day it is. A model doesn't know today's date. It will happily guess one from its training data, which is exactly as wrong as counting its context window.
The fix is a tiny LangGraph middleware that injects the current date into the system message on every model call:
def current_date_context(now: datetime | None = None) -> str:
day = (now or datetime.now(UTC)).date()
return (
f"Today's date is {day.isoformat()} (UTC). Resolve any relative date "
'("today", "yesterday", "the last 10 days", "since March") '
"to concrete ISO-8601 dates relative to it before building a date filter."
)
class CurrentDateMiddleware(AgentMiddleware):
async def awrap_model_call(self, request, handler):
base = request.system_message
line = current_date_context()
merged = line if base is None else f"{base.content}\n\n{line}"
return await handler(request.override(system_message=SystemMessage(content=merged)))
One non-obvious detail: it's injected per call and never appended to the persisted conversation. Write "today is 2026-07-30" into the thread history and a conversation resumed next week carries a stale date the model will trust completely.
Both fixes are the same principle: the model knows nothing you don't hand it, and it will answer anyway. Counts, dates, totals — anything that's a fact about your system rather than about language has to arrive as context, computed by something that can't hallucinate.
🤔 Where I'd push back on myself
Honest limits of what shipped:
- Nothing enforces that the model uses the number. The prompt instructs, the tool message supplies, the headline backstops — but there's no output check asserting the count in the model's prose equals the authoritative one. That's an eval I'd like to have and don't yet.
- Content-only searches still have no real count. A semantic ranking over chunks genuinely doesn't have an exact match total, so those turns fall back to the surfaced count. Arguably the honest move there is to refuse to give a number at all. I didn't go that far.
- This only covers counts. Sums, averages, "which author has the most" — any of those would need real aggregation endpoints on the service side. The pattern extends; the implementation doesn't yet.
And the design choice I keep turning over: I handed the model the number inline, as prose in a tool result. The agentic-purist alternative is a dedicated count_documents tool the model calls when it detects an aggregation question — cleaner separation, one more round trip, and a new failure mode where the model doesn't call it. I went with inline because the search already paid for the count; a tool call felt like latency for ideology.
So: where do you draw that line — hand the model the number, or hand it a tool to fetch the number? If your copilot answers "how many" questions today, I'd genuinely like to know which side you picked and whether it's held up.
Thanks for making it to the end 🙏 If your assistant has ever announced a total that made you go "wait, that can't be right", I'd love to swap notes on how you fixed it — come find me on LinkedIn.
Top comments (16)
The inline approach is the right call, and the asymmetry between the two failure modes is the reason. A dedicated count tool fails silently when the model simply doesn't call it; the inline number fails silently only when the model ignores it, which is catchable with a structured output assertion: extract any number the model surfaces and diff it against total_matching_records in a post-processing step. The second failure mode is easier to detect and cheaper to instrument than the first. The total_matching_records sentinel for content-only searches is doing more work than it looks like — it tells downstream steps that no crisp cardinality exists, which is a different epistemic state from "zero results" and prevents the model from substituting a cosine-similarity count for a membership count.
You read the sentinel exactly as intended — 0 means "no crisp cardinality exists here", which is a different epistemic state from "zero matches", and the count note never fires on those turns, so the model is never handed a similarity-ranking size dressed up as a membership count. And yes, the asymmetry is the whole argument for inline: an ignored inline number is detectable after the fact by diffing prose against the authoritative field; a tool call that never happened leaves nothing to diff.
84 is the pre-permission number though. If the user is cleared for a fraction of those, the honest total quietly tells them how many controlled documents exist that they cannot see, which is an audit finding rather than a bug. Did you end up scoping the count to their view, or is the headline still the service-level total?
Fair push — and my table earned the question, because it makes the view gate look like the only permission step. It isn't. The count comes from the documents service queried with the requesting user's own auth token, and that query applies view permission in the service's SQL — so 84 is "documents this user can view that match the filter", not the tenant-wide total. The downstream gate is defense-in-depth: it re-confirms view on the rows actually surfaced (it exists mainly for the content lane, where the vector index is account-scoped but not view-scoped). One more guard that matters for your scenario: content-only searches never emit a count at all — the authoritative total only exists on the filtered path, which is exactly the path that's permission-scoped. You're right that a service-level total would be an audit finding rather than a bug; that's why the count is computed as the user, not about the corpus.
The pattern feels familiar. We’ve run into the same issue with RAG systems where the model ends up "reasoning" over a retrieval artifact instead of the source of truth. Counts, latest timestamps, rankings, averages they're all properties of the datastore, not the context window. Once retrieval starts doing top-k, reranking, deduplication, or permission filtering, you've already changed the dataset the LLM sees. At that point, asking it to aggregate is asking it to invent precision. The design that has held up best for us is treating retrieval as evidence and deterministic services as authority: let SQL/search engines compute facts, let the LLM explain them. It's a surprisingly useful architectural rule beyond RAG too any value that can be derived deterministically shouldn't be regenerated probabilistically. The model is the narrator, not the calculator.
"The model is the narrator, not the calculator" — you compressed my whole post into nine words and I'm keeping it. The generalization is the right one too: it's not a RAG rule, it's a rule about any value that's derivable deterministically. Retrieval as evidence, services as authority.
This is the right boundary. RAG is good at bringing the source into view, not at becoming a tiny analytics engine on top of it. I usually want counts, latest dates, and rankings to come from a deterministic query path, then let the model explain the result and cite what it used.
That's the boundary exactly — the deterministic path computes, the model explains and cites. The one thing I'd add from the trenches: make sure the deterministic number and the model's prose can't drift apart silently; that's where our bug lived.
the rename fixes this one but nothing stops the next one. in langgraph a plain state key is a shared mutable slot whose default reducer is last-write-wins, so any node can legally clobber another node's field and you only find out when a number looks wrong. what actually holds is a custom reducer on that channel that raises when two nodes write it in the same run. that turns a silent overwrite into a loud failure at graph level, instead of a naming convention everyone has to keep remembering.
No argument — as shipped it's a convention, not an invariant. The authoritative field is written by exactly one node today, but nothing at graph level enforces that; a reducer that raises on a second write in the same run would turn the next clobber into a loud failure instead of a wrong number three sprints later. One wrinkle worth naming: the original field (the surfaced count) is legitimately written twice per run — search writes it, the gate deliberately overwrites it — so a write-once reducer on that channel would fire on the designed flow. Which is the post's lesson restated at graph level: two meanings need two channels, and the invariant belongs on the authoritative one. Taking the suggestion.
The distinction between "model as narrator" vs "model as calculator" is the cleanest heuristic I've seen for RAG architecture. The date middleware point is also worth underlining — writing temporal context into persisted history (rather than injecting per-call) is a surprisingly common footgun that surfaces weeks later when a conversation resumes with a stale date.
On the eval gap you mentioned — one lightweight approach is a structured output assertion. If the model's answer includes a number from the authoritative field, extract it with a regex and diff against total_matching_records in a post-processing step. It won't catch every phrasing, but it catches the common case where the model substitutes its own count — and adds zero latency.
On the tool-vs-inline question: I'd argue you made the right call. A dedicated count_documents tool adds a round trip whose failure mode (model doesn't call it) is silent and architecturally harder to detect than "the model ignored the inline number." The inline approach, paired with the UI headline backstop, gets 95% of the way with half the moving parts.
One dimension the article hints at but doesn't quite name: retrieval artifacts are a lossy compression of the datastore, and every lossy step (top-k, dedup, permission gate) silently changes what "all" means. The fix isn't just separating state fields — it's recognizing that the retrieval pipeline and the aggregation pipeline have different correctness requirements and probably shouldn't share the same data path at all.
The regex-extract-and-diff assertion is the eval I was missing, and you're right that it's basically free — it slots into an offline eval harness as a post-processing check on real trajectories, zero runtime latency. On "different data path": partially with you. Here the authoritative count rides the same service query the search already paid for, so sharing the query is free — what the two pipelines must not share, and what actually bit us, is the state slot. "Lossy compression of the datastore" is a framing I'll be reusing.
I ran a memory and retrieval benchmark (BEAM 1M) and pulled apart every interval question my own system lost badly, the "how many days between A and B" kind. Seven of them. One retrieved nothing, one abstained. The other five were answered confidently and wrong, and here is the part that surprised me: in all five, the arithmetic was correct on the operands the model had chosen.
The clearest case, quoted verbatim from my run:
Planned to prepare for your inclusion-exclusion quiz: January 26, 2024
Started preparing for your Bayes' theorem test: April 18, 2024
Calculation: Jan 26–31 = 5 days; Feb = 29; Mar = 31; Apr 1–18 = 18; total = 83.
Flawless. It even got 2024's leap year February right. The gold answer was 84 days, from February 1 to April 25. Both endpoints were a different study session from the one the question named.
So "your copilot can't count" may be understating your own case. Mine counted perfectly. It counted over the wrong rows. That is worse news for the "just use a bigger model" reflex, and better news for your conclusion: if arithmetic competence was never the bottleneck, then nothing done to the model fixes it. Only the system of record does.
Where I would push: your caveat that this covers counts but not sums or rankings reads to me like the sharp edge rather than a footnote. COUNT(*) WHERE author = X has a system of record that defines the answer. "How many days between A and B" often does not, because the hard part is not the subtraction, it is which row is A. When I classified those five failures by the mechanism a fix would have to implement, five questions produced four different ones. Two were "several similar events, and the question names one". One was a value that had been revised, where the correct instance turned out to be the older one, which kills every recency heuristic. One was a field's value against the time it was asserted. One was event time against utterance time, where my answer quoted the right date and then computed from the date the turn was said.
That last one connects to your date middleware, and I think it generalises. Injecting "today" fixes the model not knowing now. It does not fix the model not knowing which time a retrieved record refers to. Same class of bug, one layer down.
Bounding my own numbers before anyone else has to: BEAM is adversarially constructed, so its figures are an upper bound on difficulty rather than typical behaviour. Five items is a list, not a rate. And the mechanism labels are a hand classification made from my own run artifacts. I did not have the corpus cached locally, so the evidence for each label is the wording of my own answer. Treat it as a shape, not a measurement.
One genuine open question, and it threatens my result rather than yours. When you inject total_matching_records, have you hit a case where the authoritative count is itself contestable, where the SQL predicate is a defensible reading of the user's question rather than the reading? That is where I would expect this to get hard, and my data suggests it is the durable failure mode.
"Mine counted perfectly, over the wrong rows" is the sharpest version of the point I was circling: arithmetic competence was never the bottleneck, operand selection is. To your open question — yes, and I'd call it the soft underbelly of the whole pattern. The count is authoritative for the predicate, not for the intent. Upstream of the SQL, a classifier maps the question to structured filters, and a defensible-but-wrong mapping (author vs. contributor, "created" vs. "effective" dates) produces a flawless count of the wrong set. Two things keep it survivable in practice: the cards render right next to the count, so a wrong predicate is usually visible to the user, and the phrasing ties the number to the filter ("matching documents"), not to the question. But I don't have an eval that catches a plausible predicate returning a plausible count — and your five-questions-four-mechanisms breakdown suggests that's the hard eval. Your event-time vs. assertion-time point is fair too: injecting "today" fixes the model's clock, not the record's.
Good coverage of ML patterns. I'd stress that monitoring data drift and model staleness is as important as the initial training — a model that was accurate at launch can silently degrade without proper observability.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.