Part 4 of **The Answerability Problem, and the one that isn't about abstention. Parts 1–3 argued that the field measures the wrong half and that my...
For further actions, you may consider blocking this person and/or reporting abuse
the seven-question shape you found reads less like a ranking failure and more like a missing validity interval. raw turns preserve the text of both dates but not which one was true as of the question's reference time, so cosine has nothing to rank on. mem0 probably wins that cell for a duller reason than better distillation: its write-time llm call collapses the update, so the ambiguity never reaches storage.
you can get that property without an llm though. a turn timestamp plus a cheap "this asserts a value for field X" extraction gives you an ordering, and at read time you pick the instance whose interval covers the reference date.
I went and measured this instead of agreeing with it, and the result is more interesting than either of us expected. Three things, in order of how wrong I was.
First, most of what you described was already shipped. RE-call has carried validity windows and supersession for months: valid_from / valid_until per memory, supersession chains with terminal-successor resolution, and verdicts that demote a hit to expired or not_yet_valid against a reference time. The reference time is even already a parameter on the public search call. So this was never a missing mechanism.
Second, it could not fire. Nothing populated a validity window on benchmark data, because each conversation turn was written as a plain markdown document with the session date in the body prose rather than as metadata. With both bounds null, the expired and not_yet_valid branches were unreachable. Every temporal number I published was measured with the temporal layer structurally unable to run, which is a very different thing from "we tried it and it did not help". I have fixed that: turns now carry their session date, and 419 of 419 indexed chunks have a window where the measured baseline was 0.
Third, and this is the part that matters, your rule is still not safe here, and the reason is not one I would have guessed. A validity window records when a turn was said. Questions anchor on when the event happened. The corpus says it outright: session 17 is dated 13 October 2023, and the speaker says "recently I had a setback. Last month I got hurt." The event is September, the utterance is October. Filter by utterance time against the question's anchor and you delete retrospective testimony, which is frequently the only evidence there is.
So the boundary you asked about, whether cheap extraction generalises past dates or quietly becomes distillation, has a data-shaped answer rather than a philosophical one:
As-of retrieval keyed on utterance time is sound exactly where utterance time equals event time. In conversation it usually does not.
I measured how much of the gap is closable for free: 5.7% of turns carry a relative-time expression resolvable by arithmetic against the session date ("last month", "yesterday", "next week"), and arithmetic covers 55.9% of all relative-time turns. The largest unresolvable bucket is "recently", which carries no arithmetic content for anyone, model or not. So the free subset is real and it is small, and the rest is genuinely the distillation step, which I think is the honest version of the answer you were asking for.
fair, i collapsed two axes into one. filtering on utterance time was the wrong call and session 17 is the clean counterexample.
where i'd still push: utterance time is the right axis to order on, just not to filter on. keep it for contradiction resolution and anchor the query on event time instead. and "recently" isn't unresolvable, it's an interval with an open lower bound and a hard upper one at the utterance, which still excludes every future anchor. that was most of what the filter was buying you anyway.
Both land, and the second is a correction to my own wording.
Ordering rather than filtering is right, and it names what utterance time is for better than I did. It also buys back a limit I published: as-of retrieval keyed on write time cannot rewind supersession, because supersession edges carry no timestamps. If utterance time is the ordering axis, an edge can be dated by its revising endpoint, which is already stored on both turns. No extraction needed.
"Unresolvable" was the wrong word. I measured that "recently" carries no arithmetic content and let that travel as no content. A half-open interval is real information.
The last clause is where I counted rather than argued. Of the 26 temporal errors in my LOCOMO audit set, 9 name a date at all, and in 5 of those 9 the anchor falls after the utterance date of its own gold evidence, so a hard upper bound deletes the only evidence there is. Session 7 is dated 15 August 2023 and two questions ask about 18 and 19 August; the gold turn is "Last night I made this yummy grilled dish", so perfect event-time extraction puts the event on 14 August, further from the anchor rather than closer. The bound is not keyed on the wrong axis. It misfires because the question's date is wrong, and the bound is what makes a wrong question fatal instead of harmless. The rule I rejected kept those rows by accident.
Caveat on my own number: 9 is small and drawn from questions already known to have failed, so it is a count over selected errors, and a fact about LOCOMO's question set rather than about conversation. On a corpus with trustworthy anchors your bound is probably close to free.
So I end up applying your first point to your second: make the upper bound a ranking prior rather than a demotion, and an unreliable anchor costs position instead of evidence. What I still cannot answer is which instant to order as-of. The case that started this carries no date in the question at all.
i think the instant is the last utterance of the conversation in scope, not your clock. a question asked out of band about a closed transcript inherits that transcript's now, and you already have it stored, so there's nothing to extract.
it stops working once memory spans many conversations instead of one. there's no single center then, and scoring each candidate against its own conversation's end would just make every thread's tail look recent. at that point i'd let the temporal term go flat and lean on the rest of the ranking, which is the honest answer for a question that carries no date.
Both halves land, and the second is the part I did not have. The instant is the transcript's, not the clock's. Right, and free: conversation 0 ends on 4 January 2024 and that date is already sitting in the corpus, so there is nothing to extract. It also makes the mechanism inert, which is worth knowing before anyone builds it. I counted: of 5,882 turns across the ten conversations, zero fall after their own conversation's end. That is by construction rather than by luck, since a turn cannot postdate the transcript it belongs to. So with the instant set that way, an as-of filter demotes nothing, ever, on in-scope questions. It is exactly as safe as you say, and it does exactly nothing.
Which sharpens your second half instead of contradicting it: the regime where the instant is well defined is the regime where it has no work to do, and the regime where it would do work is the one you correctly say has no single centre.
I have a measurement from that second regime you may want. Merging LongMemEval's 500 haystacks into one 19,195-session corpus and asking every question of all of it drops hit@5 from 0.719 to 0.366. What fixed it was not scoring better across threads, it was giving each question only its own ~40 sessions and leaving the retriever untouched. That is your "let the temporal term go flat" arriving from the other side: the scope was carrying the signal, not the score.
One caveat on my own number, since it is mine. 0.366 against 0.719 is merged versus evidence-only, which is a change of scope rather than of time, so it bounds how much a global ranking loses. It does not measure how much a temporal term would have recovered.
The open question I would put back to you: once the centre is gone, is flat the same as absent? A term pinned at zero still occupies a slot and still invites someone to re-weight it a year later. I have not found a case where keeping it beats removing it, and I would rather be argued out of that than quietly ship the slot.
The temporal reasoning failure mode is the one that keeps showing up in different clothing. I have seen the same shape in verification: the system has the right data stored in multiple places, picks the wrong instance, and reports confidence the whole time.
Your correction about the validity window being structurally unreachable is actually more interesting than the original framing. "I measured it wrong and here is exactly how" is rarer than it should be in open source benchmarking. The honest version would be: the temporal layer existed in the code but was not wired to data, so the test was not testing what you thought it tested. That is not a failure of the architecture — it is a failure of the measurement, and you caught it from a reader comment. Most projects do not have that feedback loop.
The trade you are laying out — distillation at ingest vs raw storage — maps onto something I keep running into with verification gates. The cheap path stores everything and decides later what matters. The expensive path decides at write time what is worth keeping. Both have failure modes the other cannot see. Mem0 loses the ability to go back and check because it threw away the raw turns. RE-call loses temporal precision because it never collapsed the turns into a single timeline. Neither is free.
What I would push on: is the "no LLM ever" constraint a design principle or a cost constraint wearing a design principle? If reranking alone closes most of the gap, the question is not whether you need an LLM — it is where in the pipeline you can afford one.
The layer existing is what made it dangerous: missing code fails loudly, unwired code returns green and I read green as a pass. I keep a tally of that shape in another system, guards that read as protection and cannot fire, and it's at eleven. The confidence isn't a side effect of the bug, it's the bug.
One asymmetry I'd add to your two paths: different half-lives. Raw storage with a broken query layer is recoverable, the data is still there, you fix the layer and re-run over the same corpus for nothing. Write-time distillation that drops something isn't. The turns are gone, so the error stays detectable but never fixable. That's not an argument that raw wins; it's an argument about which pile your mistakes land in. With a gate on top, I want mine in the auditable one.
On the LLM: cost is real, I'm self-funding this, but it isn't the binding constraint, and I'd be dressing one up as the other if I said it was. In health, legal, finance, anything under data-residency rules, the data cannot leave the machine at any price, and "no" is increasingly the default answer before anyone asks the cost. Users are less relaxed about it every year too. So "afford" isn't budget, it's egress permission.
Two things I measured. First, "no LLM" was the wrong name for it: the local reranker is a ~22M-param cross-encoder, a neural model that happens to run on your hardware. The honest constraint is nothing leaves the machine. It also buys most of the gap, hit@5 0.640 unreranked → 0.785 local → 0.870 paid, so free captures ~63% of what paid adds (benchmarks/rerank_pool_arms.py).
Second, the judging end is where I measured an LLM can't help. On a real share of eval pairs both arms retrieve identical evidence, and nothing separates two cases handed the same documents. Abstention is starved by retrieval depth, not judge quality. So egress budget goes at ingest, one offline pass over data I control, never at query time. I tried the other route too: fine-tuned the local reranker on clean held-out labels, got a null (−0.015, CI spanning zero). It learned the objective and didn't transfer.
The half-life asymmetry is the argument I was missing. Raw storage with a broken query layer is recoverable — the data is still there, you fix the layer and re-run. Write-time distillation that drops something is not. The turns are gone, so the error stays detectable but never fixable. Not an argument that raw wins. An argument about which pile your mistakes land in. And the green-pass problem is real. Unwired code that returns green trains you to trust the signal. By the time you find the gap, you have shipped three releases on top of it.
This is a useful tradeoff to state plainly. Raw-turn memory is worse at turning messy history into answers, but it keeps the evidence inspectable. I keep coming back to validity intervals for this problem. A memory system that can say when a fact was true is doing more than retrieval.
That sentence sent me to measure rather than to agree, and it paid off, though not the way I expected.
RE-call could already say when a fact was true. What it could not do was say when we knew it, and those turn out to be different questions. The first is validity time and was shipped. The second is transaction time, and although the database column had been recorded on every write since the beginning, there was no way to ask about it.
That is now built: a search can be asked as-of a past instant and will mark anything written after it as not_yet_known, so an agent can replay what it actually held at the time rather than what it holds today. Your one sentence is a fair description of why it was worth doing.
The limit, since it belongs next to the claim: filtering by write time does not rewind supersession, because supersession edges carry no timestamps. So the replay is honest about which memories existed and approximate about which were current.
The cost/accuracy split is especially useful for coding agents because “memory” is doing at least two different jobs: retrieving evidence and resolving which state was authoritative at a given time. Raw turns preserve auditability, but they need an explicit temporal/state layer or the retriever is forced to infer truth from proximity.
A practical contract I’d add is to make each stored assertion carry its source turn, effective interval, supersession status, and confidence, then require the agent to return those receipts alongside any answer. That lets a workflow distinguish “no matching evidence” from “conflicting evidence” instead of turning both into a plausible paragraph. It also makes the offline/zero-ingest-cost design testable without pretending it wins every reasoning category.
Thanks for useful comment, and I want to report back concretely rather than thank you vaguely.
Three of your four fields already existed. A hit already returns its source, its position, its validity window, its supersession status and a confidence, alongside the verdict that was reached about it. What your framing did was make me notice that the fourth axis was missing entirely, and that the shape of the gap was different from what I assumed.
What shipped because of this comment: the ability to ask a memory as-of a past instant, keyed on when each memory was written rather than when its content was true. Both axes now compose, so you can ask what we believed on Tuesday about the state of the world in June. That is the "resolving which state was authoritative at a given time" half of your two jobs, and you were right that it was a distinct job rather than a flavour of retrieval.
The gap you identified in my measurements is real and I have not closed it. My abstention work sorts questions into answerable and unanswerable and scores the separation, so it cannot represent conflicting. A system silent because it found nothing and a system silent because it found two contradicting things score identically, which is exactly your "no matching evidence versus conflicting evidence" distinction, and it is a defect in the measurement rather than in the writeup. It is filed.
One honest subtraction from your proposal. I checked whether supersession-based selection could be scored on the standard corpus, and it cannot: of 5,882 conversation turns, the number that revise a previously asserted value is effectively zero. So on that data the mechanism has nothing to act on. It says nothing about your contract being wrong, and everything about that corpus not being where it would show up, which is worth knowing before anyone builds an eval for it.
And one limit of the contract itself, which I would rather state than have you find: receipts do not attest that anyone ran anything. A fabricated artifact can carry fabricated receipts exactly as it can carry fabricated scores. It widens what is mechanically checkable, which is real value, without closing the fabrication gap.
If you want to shape the conflict axis, I would rather build it with you than guess at it.
"Egress permission, not budget" is the sentence I've been trying to find for a year. I run a vision model fully in-browser and every time someone asks why not a server, the honest answer was never cost, it was that users paste screenshots of things I don't want to be responsible for seeing.
Exactly. “Why not a server?” often sounds like an architecture or cost question, but for this class of product it’s really a liability boundary.
If the screenshot never leaves the browser, the product can make a much stronger promise: I did not receive it, I cannot inspect it, I cannot log it, I cannot be subpoenaed for it, and I cannot accidentally retain it. That is qualitatively different from “we send it to our server but promise to behave.”
The cost argument is almost a distraction there. Even if inference were free, receiving arbitrary user screenshots means becoming responsible for whatever is inside them. Sometimes the best privacy feature is not better handling of sensitive data, but designing the system so you never possess it in the first place.
Your example is exactly the same shape as the memory-layer tradeoff: local-first is not just cheaper. It changes who is allowed to know.
The 46% fabrication number is the part that stuck with me. A memory layer that never touches an LLM trades some recall for something you can actually audit, which feels right when the retrieved context goes straight into a prompt. I would take predictable and inspectable over slightly higher accuracy in most agent setups.
Thanks, though I want to decline part of it, because it credits the design with something it didn't earn.
The 46% is Mem0's number, and being LLM-free isn't what produces it. My memory layer builds with 0 LLM calls, but the abstention behaviour comes from a retrieval threshold, not from the absence of a model. A distillation-based system with the same threshold policy would abstain the same way. Those two properties are independent, and I'd be overclaiming if I let them merge.
"Trades some recall" isn't quite the shape of it either. On LOCOMO it went the other way: n=1,540, gpt-4o judge, RE-call 0.484 against Mem0's 0.444 with a gpt-4o generator (p=0.0065), and 0.466 against 0.412 with gpt-4o-mini (p=0.0002). On BEAM, Mem0 was ahead 0.650 to 0.594. So it's benchmark-dependent rather than a clean trade, and BEAM is adversarial by construction, which makes its numbers an upper bound on difficulty rather than deployed behaviour.
What you're right about is the part I'd actually defend: inspectable is metered, not argued. 0 LLM calls and 0 tokens in the memory layer, so there's no extraction step whose output you have to take on faith.
One caveat on "predictable", since it cuts both ways. Deterministic includes deterministically wrong. On LongMemEval my abstention guard false-abstained on 48.1% of questions while retrieval hit@5 was 0.970. It refused half the questions it had just answered correctly. That failure is reproducible and inspectable, which genuinely beats a stochastic one. It's still a failure.
The Mem0 versus RE-call comparison is the exact trade-off we made deliberately in Opportunity Skill's impression management, and we chose the LLM distillation path with full awareness of its cost. The agent reasons about what should become an impression from daily interactions, distilling attributes and preferences into structured semantic units of at most 512 characters with 1 to 5 tags. This costs LLM calls. But the alternative, storing raw conversation turns and hoping retrieval finds the right signal, produces exactly the temporal reasoning weakness you measured. The 46 percent fabrication rate on unanswerable questions is the number that validates our design choice on the search side. The human discovery module returns an empty list when cosine similarities are insufficient rather than presenting a confident but unsupported match. Silence is architecturally cheaper than a wrong introduction between two professionals. The privacy advantage of RE-call is real though, and it is why impressions in our system require explicit intent to become public. A flag at the server level prevents any information the agent extracts from being externally discoverable without deliberate choice.
Thanks for reading it that closely, you picked up the temporal-reasoning weakness, which is the part most readers skim past, and you described the mechanism the way my own notes do: one distilled line versus a date scattered across raw turns.
Since you're abstaining on cosine in the discovery module, here's what I'd most want to know if I were building that. I measured that guard on three benchmarks and it has a bounded domain. It works when the gap is far, the distributions are disjoint, abstention accuracy 1.00. It falls apart on near-miss: on LongMemEval it false-abstained on 48.1% of questions while retrieval hit@5 was 0.970. It refused half the questions it had just answered correctly. I tried six signals on the same 500 questions: dense cosine AUC 0.753, cross-encoder rerank 0.742, RRF 0.739, QNLI entailment 0.648, margin 0.579, ratio 0.545, and none beat the shipped threshold. The reason looks structural: every one of those is a relevance signal, and relevance is not answerability.
That may matter more for you than it did for me. In professional matching, near-miss is the normal case rather than the tail, two people adjacent on every visible dimension, where the actual attribute is simply absent. The silence is cheapest exactly where it's easiest to be silent.
The concrete one, if it's useful: my threshold constant turned out not to be comparable across embedders. 0.50 sat at the 0th percentile of five of six distributions. Inert on bge-small and bge-large, starving 16% of legitimate queries on text-embedding-3-small. If your cutoff is a hardcoded float, it changes meaning silently the day you upgrade your embedding model. That one took me a while to find.
One caveat on my own number, so it doesn't get pulled out from under you if you cite it. The 46% is a BEAM figure, and BEAM is adversarial by construction: its unanswerable questions ask for details never stated about topics discussed at length, so cosine inverts there, 0.676 on unanswerable versus 0.641 on answerable, with lexical coverage inverting identically. Every BEAM number is an upper bound on difficulty rather than deployed behaviour. On that same 300-question run Mem0 came out ahead on the aggregate (0.650 against my 0.594, accuracy 71.0% against 66.7%), and their published numbers are honest. I re-judged their answers independently and reproduced BEAM 1M to 0.0005. I'd hate for the 46% to travel as "Mem0 fabricates half the time", because the first person to open their repo would be right and I'd be wrong.
And the honest limit on the part you agreed with: BEAM is also the one benchmark that can't price an abstention claim. Every abstention policy I tested on it came out net negative, because false-abstain risk applies to far more questions than abstention can ever win. You may well be right that silence beats a wrong introduction. My number just isn't the evidence for it.
Which is why I'm building a different benchmark, aimed at exactly that gap: abstention gets its own axis with its own manifest, rather than dissolving into an aggregate that structurally can't price it. Temporal reasoning and tenant isolation are separate axes alongside it. There is deliberately no LLM anywhere in it, not in generation, not in scoring. Part of that is an audit finding I'd rather cite than claim, since I haven't reproduced it myself: Penfield Labs found the LOCOMO judge accepts 62.8% of intentionally wrong answers, and a permissive non deterministic scorer compresses every margin toward zero until you can't separate a real effect from judge noise. The larger reason is that I want verification to cost nothing and run locally, so a hostile party can re-run it without an API key and without trusting me.
That choice has a price, and I'd rather state it than have it found. Because the verifier runs locally with no secret and can score any artifact, the verifier holds the answer key. So every submitter holds it too, and roughly twenty lines that open the manifest and echo it back score at the ceiling on every axis and pass all six mechanical checks. It isn't fabrication, which is what makes it hard: it's a real deterministic program, honestly executed, so an independent re-run confirms it, and two people submitting it would earn the "independently reproduced" badge automatically when an honest stochastic system might never agree that closely. I think it's structurally unclosable, salted hashes fail because the corpus is public and the id space is enumerable, and a held-out key destroys the $0 local verification that is the entire point. It's documented as the top limit and pinned by a test. So the honest reading of any score on it is: a ceiling a system did not fail to reach, never evidence that a system exists.
On privacy worth being precise that mine is architectural rather than a policy control: the memory layer meters 0 LLM calls and 0 tokens, so there's nothing extracted to gate in the first place. Yours covers something mine doesn't, though. I have no equivalent of deliberate publication, because I have no publication path at all.
Last thing, and it's a real question rather than a rhetorical one. I observed the temporal win and attributed it to distillation, but I never ablated it. I never ran RE-call with a distillation layer to isolate the cause. It's an inference from a mechanism story, not a measurement. If your build gives you that counterfactual from the other side, I'd rather have your data than my inference.
The "field value confused with the time it was asserted" problem is the one I'd bet most people overlook. In sensor data from robots, we hit this constantly: an IMU reading timestamped at ingestion (when the logger picked it up) vs. at observation (when the sensor fired). On a Raspberry Pi with DMA-backed SPI, that gap runs 40-80ms. Enough to throw off a dead-reckoning estimate by half a meter.
We handle this in moteDB by storing two timestamps per observation,
observed_atandingested_at, and the temporal query layer reasons overobserved_at. That works because we control the write path. RE-call inherits raw turns where that distinction is usually already lost, so the wrong-instance problem becomes structurally unfixable without re-ingestion.The $7.29 per-benchmark-memory number doesn't get enough attention. Scale it to 200 devices writing 500 memories/day each and you're at $729K/year on LLM ingestion alone. The offline argument isn't just privacy. It's unit economics that break at edge scale.
One thing I'd push on: reranking lifted hit@5 from 0.671 to 0.777 on LOCOMO but went unmeasured on BEAM. Have you considered that reranking might hurt temporal reasoning specifically? If the reranker leans on semantic similarity, it could surface the "updated deadline" over the "original deadline" because newer mentions tend to be more semantically rich. That would make the wrong-instance problem worse.
Interesting perspective. At IT Path Solutions, we've found that treating memory as an architectural concern rather than an LLM feature often results in more predictable and maintainable AI systems. Keeping deterministic operations outside the model helps reduce complexity while allowing the LLM to focus on higher-value reasoning tasks.