Part 4 of **The Answerability Problem, and the one that isn't about abstention. Parts 1–3 argued that the field measures the wrong half and that my own system hits a wall on the right half. This part is the ledger: what RE-call is, what it loses at, and why you might still want the thing that loses. Code: RE-call (MIT).
Three parts of "here is what doesn't work" earn one part of "here is what this is for."
So, plainly: on BEAM's 1M-token bucket, Mem0 scores better than RE-call on categories I care about. I'm not going to bury that, and I'm not going to explain it away. I'm going to tell you exactly why it happens, what that accuracy costs, and let you decide which side of the trade you want, because for a lot of people the answer is not the obvious one.
The mechanism, in one sentence
Mem0 calls an LLM when you write a memory, and stores the distilled result. RE-call calls no LLM ever, and stores your raw turns.
That single decision explains everything downstream: the wins, the losses, the bill, and where your data goes.
The clearest place to see it is temporal_reasoning, my second-worst BEAM category: 0.408 against Mem0's 0.567.
One disclosure before that number goes anywhere, because it changes what it means: that cell was measured with reranking off. The BEAM harness takes a --reranker flag and defaults it to none, and the run didn't pass it. Reranking is the largest retrieval gain in this project. Part 2 measures it at hit@5 0.671 → 0.777 on LOCOMO, improving every category including the multi-hop floor I'd predicted it wouldn't touch. So 0.408 is my shipped default, not my best configuration, and the reranked cell is unmeasured.
I'm not going to tell you which way that would go. The last time I predicted a category wouldn't benefit from reranking I was wrong, and guessing here would be the same error with a bigger number attached.
What I can say is that reranking is unlikely to be the whole story, because the diagnosis isn't a ranking failure. Of seven badly-lost questions only one had empty retrieval; five were answered confidently and wrongly. They are all the same shape, "how many days between A and B", and in the five it answered, my system used the wrong instance of a date, while getting the arithmetic on those wrong dates right every single time. That is what makes this a selection problem and not a reasoning one:
| gold | our answer |
|---|---|
| 25 Mar → 1 Apr = 7 days | 14 days, using the updated deadline of 15 Apr |
| 25 Mar → 10 Apr = 16 days | 26 days, using a different viewing on 15 Mar |
| 15 Feb → 20 Feb = 5 days | 0 days, using 10 Jan, the date the deadline was set |
Mem0 gets these right because its stored memory is one distilled line, "Sprint 1 deadline: February 15, 2024". Mine is the same date scattered across many raw turns in different roles: when it was set, when it was revised, when someone mentioned it in passing.
This is the one category where LLM distillation at ingest is genuinely the better architecture, and no retrieval-side change I can afford replicates it. It's recorded in the repo as a known limit, not an open task.
Correction, added after publication. That paragraph claimed more than I had measured, and a reader's comment exposed it. The cell was measured with my own temporal layer structurally unable to fire, which is not the same as having tried it, and "recency is falsified" rules out one family of fix, not all of them. The obvious successor is unsound too, for a reason I did not expect: a validity window records when a turn was said, not when the event happened.
So I enumerated the seven questions instead of describing them; they are a fixture in the repo now. Of the five my system answered, the mechanisms are two wrong instances of a similar event, one genuine revision, one field value confused with the time it was asserted, and one event time confused with mention time. Supersession therefore reaches one of five: four problems wearing one category name, none of them solved. Stated as narrowly as it was made, that is a hand reading of five answers from the run artifact alone, and five items is a list, not a rate.
So: they're better at this, for a real reason. Whether it is cheaply fixable is now an open question rather than a closed one.
What that accuracy costs
Here's the other side of the same decision, measured on the identical benchmark workload:
| RE-call | Mem0 | |
|---|---|---|
| LLM calls to build the memory | 0 | 272 |
| tokens | 0 | 2.6 M |
| cost | $0 | $7.29 |
| ingest wall clock | 67 s | 288 s (~4.3× slower) |
| where your documents go | your Postgres | an LLM provider, once per memory written |
That $7.29 is for one benchmark's memory. It is not a subscription. It is a per-memory marginal cost that scales with everything you ever write. RE-call's write path calls no model, so its marginal cost is $0 at any scale, on any model, forever. There is no pricing change upstream that can alter that number.
The ingest gap is the same fact wearing a stopwatch: an extraction call per session is a network round-trip per session.
(Retrieve latency, 77 ms against 104 ms, I report as **directional only. The repeated-query bootstrap CI is optimistic and the two backends differ, so I won't lean on it.)
The part that isn't a number
Your data never leaves your infrastructure. Local embeddings, the PostgreSQL you already run, and it works offline: on an air-gapped box, in a privacy-bound environment, under a DPA that doesn't have room for another subprocessor.
A memory layer that calls an LLM per write cannot offer that, structurally. Not because anyone is careless, but because the architecture requires sending the content out to distil it.
If you're a solo developer, that's a cost story. If you're a company holding customer conversations, it's a procurement story, and it's usually the one that decides the question before accuracy is ever discussed.
The cloud embedder is available as a measured option rather than a default, and the repo prices it honestly: it wins on 16 of 17 held-out corpora, median +0.059 hit@5, and it means every document and every query leaves your machine, at 246 ms p50 against 45 ms local. You get the numbers and you choose. That's the pattern for everything here.
Everything is a switch, and both settings are measured
This is the part I undersold for five articles.
-
Embedder: a hashing model that needs no download,
bge-small,bge-large, Voyage, or anything OpenAI-compatible. Measured across 17 corpora, with the rule for when paying is worth it. - Reranking: off by default, one flag on. It's the largest retrieval gain in the project (hit@5 0.671 → 0.777) and it costs ~1,050 ms/query, about 4× wall clock. Both halves are published, and the decision is yours: answering a human, ~1 s is invisible next to the LLM call that follows; serving high-volume automated retrieval, it dominates.
- Entailment abstention: off by default, and Part 2 is 2,000 words on why it doesn't rescue the hard case.
- The whole stack: Postgres and pgvector. No separate vector database, no queue, no second store to keep consistent.
I'd rather ship a flag with two measured settings than a default with a marketing claim.
And the risk that doesn't show up in an accuracy column
BEAM has an abstention category, questions whose correct answer is "that isn't in here." I scored Mem0's own published answers on it, with Mem0's own judge. n=70:
| Mem0 did | n | mean score |
|---|---|---|
| abstained | 38 | 0.974 |
| answered anyway | 32 | 0.016 |
The category is near-perfectly binary, and it is testing exactly one thing: does the system invent an answer when the evidence isn't there. It invents one 46% of the time.
One real example: asked about user feedback that was never recorded, it answered "User testing showed a positive response: the dynamic language switching feature achieved a 90% satisfaction rate." The corpus does contain "achieving a 90% satisfaction rate is a strong start", the assistant speculating, which retrieval surfaced and the answerer read as fact.
I want to be precise about what I'm claiming, because this is where it would be easy to overreach:
- Their published score is honest. I reproduced their BEAM cell to 0.0005 (0.6414 against 0.6409). Nothing is being faked.
- The benchmark is not backfitted to them. A benchmark tuned to flatter a vendor would not expose a 46% fabrication rate on that vendor.
- And my own number here is not a triumph. On the same category my shipped policy abstains correctly 23.3% of the time at a 9.3% false-abstain cost, measured on 30 unanswerable and 270 answerable questions across conversations 0–14, at $0, because that probe needs no LLM. Same configuration caveat as above: reranker off. I tested four stricter policies and every one of them nets worse on BEAM's 9:1 answerable-to-unanswerable mix. The shipped policy is already the best of the five, which is the opposite of what I predicted.
So neither of us solves this. The difference is the shape of the failure: a system that fabricates a plausible satisfaction rate is a different kind of liability from one that returns nothing. Depending on what you're building, "slightly less accurate" and "confidently invents a statistic" are not two points on one scale.
Two things wrong with the benchmark itself
Worth saying because they cut against my own story as much as anyone's.
BEAM's unanswerable questions score higher than its answerable ones: median top-1 cosine 0.676 against 0.641. They're adversarially constructed, and a lexical-coverage signal sharing no mathematics with cosine inverts in the same direction. So every BEAM figure, theirs and mine, is an upper bound on difficulty, not an estimate of deployed behaviour. On an ordinary corpus, where unanswerable means genuinely absent, plain cosine separates at AUC 0.780.
The scoring is 9:1 against withholding. Abstention is ~10% of BEAM, so a policy that abstains more gains on 30 questions and loses on 270. That is not a complaint about the benchmark. Its abstention category is well built. It means an abstention claim cannot be made through BEAM's aggregate, no matter how good the policy gets. The field needs a metric that prices a false answer against a withheld one, and this isn't it.
Two smaller corrections that arm produced: BEAM's harness pins gpt-5 as answerer and judge (not gpt-4o), and the published 64.1 is a mean rubric-nugget score, not a pass rate: the pass rate for the same run is 70.14%.
What I'm not reporting: a paired BEAM aggregate. I have per-category cells and the abstention probe; I do not have a like-for-like total, so there isn't one in the repo and there isn't one here. An earlier draft of this project did fuse two unrelated BEAM numbers into a headline that read well and did not exist. Once burned.
So which one should you use?
Genuinely depends, and I'll say it against my own interest:
Use the LLM-distilling architecture if your questions are heavy on temporal and multi-hop reasoning over dense material, a strong reader is doing the answering, per-memory cost is not a constraint, and sending content to a model provider is fine for you. It is better at that, measurably, and I've shown you the category where it beats me and why.
Use this one if the marginal cost of a memory has to be zero, your data can't leave, you need it to work offline, you want the retrieval path to be inspectable and switchable rather than a model's opinion, or if a system that confidently invents a satisfaction rate is a worse outcome for you than one that says nothing.
That's the trade. It isn't "we're better." It's that an LLM in the write path buys a better representation and charges you money, latency, and your data for it, and for a large number of real deployments that's the wrong purchase.
Where the series ends up
Parts 1–3 said the field measures accuracy on questions that have answers, that the standard harness for the most-quoted benchmark forbids abstaining, that when I built the missing metric my own system scored zero, and finally that the two public benchmarks disagree because each samples one point of a hidden axis, which turns a yes/no question into a coordinate.
This part says the rest of it: that the same design which loses temporal_reasoning is the one that costs $0, ingests 4.3× faster, and never sends a document anywhere, and that the incumbent's better aggregate comes with a 46% fabrication rate on the questions where the honest answer is silence.
Both of those are true at once. Publishing only the flattering half is the thing this whole series is against.
For completeness: the paired contest
On LOCOMO, same questions, same generator, same judge, only the memory differs. Paired McNemar over per-question outcomes, n=1,540:
| generator | judge | RE-call | Mem0 | paired p |
|---|---|---|---|---|
| gpt-4o-mini | gpt-4o-mini | 0.416 | 0.378 | 0.0059 |
| gpt-4o-mini | gpt-4o | 0.466 | 0.412 | 0.00018 |
| gpt-4o | gpt-4o | 0.484 | 0.444 | 0.0065 |
Holm–Bonferroni across all five cells run (largest adjusted p = 0.012), and it holds on the 1,369 questions where both judges agree (0.440 vs 0.399, p = 0.006).
Two things I'll say louder than the result. The lead is a property of the reader, not a universal fact. The margin shrinks as the generator strengthens and reverses on Claude Sonnet (0.565 vs 0.608, n=584, a generator run after pre-registration and labelled as such). And note the absolute numbers: 0.416–0.484, nothing like the 92.5 in the headlines, because a paired protocol with a strict judge and a matched retrieval budget measures something the leniency stack in Part 1 does not.
One more thing, and it's the reason any of this is checkable
Everything above is a number I produced about my own system. You have no way to recompute it without buying inference.
That bothered me enough to build the fix: a benchmark where every number is recomputable by a hostile party for $0, with no model in the loop: no judge, no paid generator, git as the database, and disputing a result is a pull request. It has RE-call on its board with an unflattering score, and a deliberately fabricated submission committed as a fixture because it passes five of the six mechanical checks.
That's its own post, coming next, and it's the one I'd most like you to act on.
Every figure: results/FINDINGS.md §9d–§9p, results/RESULTS.md §9–§11. RE-call is MIT, pip install recall-rag.
Top comments (24)
the seven-question shape you found reads less like a ranking failure and more like a missing validity interval. raw turns preserve the text of both dates but not which one was true as of the question's reference time, so cosine has nothing to rank on. mem0 probably wins that cell for a duller reason than better distillation: its write-time llm call collapses the update, so the ambiguity never reaches storage.
you can get that property without an llm though. a turn timestamp plus a cheap "this asserts a value for field X" extraction gives you an ordering, and at read time you pick the instance whose interval covers the reference date.
I went and measured this instead of agreeing with it, and the result is more interesting than either of us expected. Three things, in order of how wrong I was.
First, most of what you described was already shipped. RE-call has carried validity windows and supersession for months: valid_from / valid_until per memory, supersession chains with terminal-successor resolution, and verdicts that demote a hit to expired or not_yet_valid against a reference time. The reference time is even already a parameter on the public search call. So this was never a missing mechanism.
Second, it could not fire. Nothing populated a validity window on benchmark data, because each conversation turn was written as a plain markdown document with the session date in the body prose rather than as metadata. With both bounds null, the expired and not_yet_valid branches were unreachable. Every temporal number I published was measured with the temporal layer structurally unable to run, which is a very different thing from "we tried it and it did not help". I have fixed that: turns now carry their session date, and 419 of 419 indexed chunks have a window where the measured baseline was 0.
Third, and this is the part that matters, your rule is still not safe here, and the reason is not one I would have guessed. A validity window records when a turn was said. Questions anchor on when the event happened. The corpus says it outright: session 17 is dated 13 October 2023, and the speaker says "recently I had a setback. Last month I got hurt." The event is September, the utterance is October. Filter by utterance time against the question's anchor and you delete retrospective testimony, which is frequently the only evidence there is.
So the boundary you asked about, whether cheap extraction generalises past dates or quietly becomes distillation, has a data-shaped answer rather than a philosophical one:
As-of retrieval keyed on utterance time is sound exactly where utterance time equals event time. In conversation it usually does not.
I measured how much of the gap is closable for free: 5.7% of turns carry a relative-time expression resolvable by arithmetic against the session date ("last month", "yesterday", "next week"), and arithmetic covers 55.9% of all relative-time turns. The largest unresolvable bucket is "recently", which carries no arithmetic content for anyone, model or not. So the free subset is real and it is small, and the rest is genuinely the distillation step, which I think is the honest version of the answer you were asking for.
fair, i collapsed two axes into one. filtering on utterance time was the wrong call and session 17 is the clean counterexample.
where i'd still push: utterance time is the right axis to order on, just not to filter on. keep it for contradiction resolution and anchor the query on event time instead. and "recently" isn't unresolvable, it's an interval with an open lower bound and a hard upper one at the utterance, which still excludes every future anchor. that was most of what the filter was buying you anyway.
Both land, and the second is a correction to my own wording.
Ordering rather than filtering is right, and it names what utterance time is for better than I did. It also buys back a limit I published: as-of retrieval keyed on write time cannot rewind supersession, because supersession edges carry no timestamps. If utterance time is the ordering axis, an edge can be dated by its revising endpoint, which is already stored on both turns. No extraction needed.
"Unresolvable" was the wrong word. I measured that "recently" carries no arithmetic content and let that travel as no content. A half-open interval is real information.
The last clause is where I counted rather than argued. Of the 26 temporal errors in my LOCOMO audit set, 9 name a date at all, and in 5 of those 9 the anchor falls after the utterance date of its own gold evidence, so a hard upper bound deletes the only evidence there is. Session 7 is dated 15 August 2023 and two questions ask about 18 and 19 August; the gold turn is "Last night I made this yummy grilled dish", so perfect event-time extraction puts the event on 14 August, further from the anchor rather than closer. The bound is not keyed on the wrong axis. It misfires because the question's date is wrong, and the bound is what makes a wrong question fatal instead of harmless. The rule I rejected kept those rows by accident.
Caveat on my own number: 9 is small and drawn from questions already known to have failed, so it is a count over selected errors, and a fact about LOCOMO's question set rather than about conversation. On a corpus with trustworthy anchors your bound is probably close to free.
So I end up applying your first point to your second: make the upper bound a ranking prior rather than a demotion, and an unreliable anchor costs position instead of evidence. What I still cannot answer is which instant to order as-of. The case that started this carries no date in the question at all.
i think the instant is the last utterance of the conversation in scope, not your clock. a question asked out of band about a closed transcript inherits that transcript's now, and you already have it stored, so there's nothing to extract.
it stops working once memory spans many conversations instead of one. there's no single center then, and scoring each candidate against its own conversation's end would just make every thread's tail look recent. at that point i'd let the temporal term go flat and lean on the rest of the ranking, which is the honest answer for a question that carries no date.
Both halves land, and the second is the part I did not have. The instant is the transcript's, not the clock's. Right, and free: conversation 0 ends on 4 January 2024 and that date is already sitting in the corpus, so there is nothing to extract. It also makes the mechanism inert, which is worth knowing before anyone builds it. I counted: of 5,882 turns across the ten conversations, zero fall after their own conversation's end. That is by construction rather than by luck, since a turn cannot postdate the transcript it belongs to. So with the instant set that way, an as-of filter demotes nothing, ever, on in-scope questions. It is exactly as safe as you say, and it does exactly nothing.
Which sharpens your second half instead of contradicting it: the regime where the instant is well defined is the regime where it has no work to do, and the regime where it would do work is the one you correctly say has no single centre.
I have a measurement from that second regime you may want. Merging LongMemEval's 500 haystacks into one 19,195-session corpus and asking every question of all of it drops hit@5 from 0.719 to 0.366. What fixed it was not scoring better across threads, it was giving each question only its own ~40 sessions and leaving the retriever untouched. That is your "let the temporal term go flat" arriving from the other side: the scope was carrying the signal, not the score.
One caveat on my own number, since it is mine. 0.366 against 0.719 is merged versus evidence-only, which is a change of scope rather than of time, so it bounds how much a global ranking loses. It does not measure how much a temporal term would have recovered.
The open question I would put back to you: once the centre is gone, is flat the same as absent? A term pinned at zero still occupies a slot and still invites someone to re-weight it a year later. I have not found a case where keeping it beats removing it, and I would rather be argued out of that than quietly ship the slot.
you're right that it's inert, and i'd rather have that measured than keep the idea. on flat vs absent though, i think the difference isn't ranking, it's observability. drop it from the score entirely so nobody can re-weight it, but keep computing it as an assertion: zero turns postdating their own conversation is an invariant of closed transcripts, not of corpora generally. the day someone ingests live transcripts or merges haystacks the way you did, that invariant breaks and you'd want it to fire rather than quietly start mattering again. so i'd take the slot out of the ranker and leave the check in.
The temporal reasoning failure mode is the one that keeps showing up in different clothing. I have seen the same shape in verification: the system has the right data stored in multiple places, picks the wrong instance, and reports confidence the whole time.
Your correction about the validity window being structurally unreachable is actually more interesting than the original framing. "I measured it wrong and here is exactly how" is rarer than it should be in open source benchmarking. The honest version would be: the temporal layer existed in the code but was not wired to data, so the test was not testing what you thought it tested. That is not a failure of the architecture — it is a failure of the measurement, and you caught it from a reader comment. Most projects do not have that feedback loop.
The trade you are laying out — distillation at ingest vs raw storage — maps onto something I keep running into with verification gates. The cheap path stores everything and decides later what matters. The expensive path decides at write time what is worth keeping. Both have failure modes the other cannot see. Mem0 loses the ability to go back and check because it threw away the raw turns. RE-call loses temporal precision because it never collapsed the turns into a single timeline. Neither is free.
What I would push on: is the "no LLM ever" constraint a design principle or a cost constraint wearing a design principle? If reranking alone closes most of the gap, the question is not whether you need an LLM — it is where in the pipeline you can afford one.
The layer existing is what made it dangerous: missing code fails loudly, unwired code returns green and I read green as a pass. I keep a tally of that shape in another system, guards that read as protection and cannot fire, and it's at eleven. The confidence isn't a side effect of the bug, it's the bug.
One asymmetry I'd add to your two paths: different half-lives. Raw storage with a broken query layer is recoverable, the data is still there, you fix the layer and re-run over the same corpus for nothing. Write-time distillation that drops something isn't. The turns are gone, so the error stays detectable but never fixable. That's not an argument that raw wins; it's an argument about which pile your mistakes land in. With a gate on top, I want mine in the auditable one.
On the LLM: cost is real, I'm self-funding this, but it isn't the binding constraint, and I'd be dressing one up as the other if I said it was. In health, legal, finance, anything under data-residency rules, the data cannot leave the machine at any price, and "no" is increasingly the default answer before anyone asks the cost. Users are less relaxed about it every year too. So "afford" isn't budget, it's egress permission.
Two things I measured. First, "no LLM" was the wrong name for it: the local reranker is a ~22M-param cross-encoder, a neural model that happens to run on your hardware. The honest constraint is nothing leaves the machine. It also buys most of the gap, hit@5 0.640 unreranked → 0.785 local → 0.870 paid, so free captures ~63% of what paid adds (benchmarks/rerank_pool_arms.py).
Second, the judging end is where I measured an LLM can't help. On a real share of eval pairs both arms retrieve identical evidence, and nothing separates two cases handed the same documents. Abstention is starved by retrieval depth, not judge quality. So egress budget goes at ingest, one offline pass over data I control, never at query time. I tried the other route too: fine-tuned the local reranker on clean held-out labels, got a null (−0.015, CI spanning zero). It learned the objective and didn't transfer.
The half-life asymmetry is the argument I was missing. Raw storage with a broken query layer is recoverable — the data is still there, you fix the layer and re-run. Write-time distillation that drops something is not. The turns are gone, so the error stays detectable but never fixable. Not an argument that raw wins. An argument about which pile your mistakes land in. And the green-pass problem is real. Unwired code that returns green trains you to trust the signal. By the time you find the gap, you have shipped three releases on top of it.
This is a useful tradeoff to state plainly. Raw-turn memory is worse at turning messy history into answers, but it keeps the evidence inspectable. I keep coming back to validity intervals for this problem. A memory system that can say when a fact was true is doing more than retrieval.
That sentence sent me to measure rather than to agree, and it paid off, though not the way I expected.
RE-call could already say when a fact was true. What it could not do was say when we knew it, and those turn out to be different questions. The first is validity time and was shipped. The second is transaction time, and although the database column had been recorded on every write since the beginning, there was no way to ask about it.
That is now built: a search can be asked as-of a past instant and will mark anything written after it as not_yet_known, so an agent can replay what it actually held at the time rather than what it holds today. Your one sentence is a fair description of why it was worth doing.
The limit, since it belongs next to the claim: filtering by write time does not rewind supersession, because supersession edges carry no timestamps. So the replay is honest about which memories existed and approximate about which were current.
The cost/accuracy split is especially useful for coding agents because “memory” is doing at least two different jobs: retrieving evidence and resolving which state was authoritative at a given time. Raw turns preserve auditability, but they need an explicit temporal/state layer or the retriever is forced to infer truth from proximity.
A practical contract I’d add is to make each stored assertion carry its source turn, effective interval, supersession status, and confidence, then require the agent to return those receipts alongside any answer. That lets a workflow distinguish “no matching evidence” from “conflicting evidence” instead of turning both into a plausible paragraph. It also makes the offline/zero-ingest-cost design testable without pretending it wins every reasoning category.
Thanks for useful comment, and I want to report back concretely rather than thank you vaguely.
Three of your four fields already existed. A hit already returns its source, its position, its validity window, its supersession status and a confidence, alongside the verdict that was reached about it. What your framing did was make me notice that the fourth axis was missing entirely, and that the shape of the gap was different from what I assumed.
What shipped because of this comment: the ability to ask a memory as-of a past instant, keyed on when each memory was written rather than when its content was true. Both axes now compose, so you can ask what we believed on Tuesday about the state of the world in June. That is the "resolving which state was authoritative at a given time" half of your two jobs, and you were right that it was a distinct job rather than a flavour of retrieval.
The gap you identified in my measurements is real and I have not closed it. My abstention work sorts questions into answerable and unanswerable and scores the separation, so it cannot represent conflicting. A system silent because it found nothing and a system silent because it found two contradicting things score identically, which is exactly your "no matching evidence versus conflicting evidence" distinction, and it is a defect in the measurement rather than in the writeup. It is filed.
One honest subtraction from your proposal. I checked whether supersession-based selection could be scored on the standard corpus, and it cannot: of 5,882 conversation turns, the number that revise a previously asserted value is effectively zero. So on that data the mechanism has nothing to act on. It says nothing about your contract being wrong, and everything about that corpus not being where it would show up, which is worth knowing before anyone builds an eval for it.
And one limit of the contract itself, which I would rather state than have you find: receipts do not attest that anyone ran anything. A fabricated artifact can carry fabricated receipts exactly as it can carry fabricated scores. It widens what is mechanically checkable, which is real value, without closing the fabrication gap.
If you want to shape the conflict axis, I would rather build it with you than guess at it.
"Egress permission, not budget" is the sentence I've been trying to find for a year. I run a vision model fully in-browser and every time someone asks why not a server, the honest answer was never cost, it was that users paste screenshots of things I don't want to be responsible for seeing.
Exactly. “Why not a server?” often sounds like an architecture or cost question, but for this class of product it’s really a liability boundary.
If the screenshot never leaves the browser, the product can make a much stronger promise: I did not receive it, I cannot inspect it, I cannot log it, I cannot be subpoenaed for it, and I cannot accidentally retain it. That is qualitatively different from “we send it to our server but promise to behave.”
The cost argument is almost a distraction there. Even if inference were free, receiving arbitrary user screenshots means becoming responsible for whatever is inside them. Sometimes the best privacy feature is not better handling of sensitive data, but designing the system so you never possess it in the first place.
Your example is exactly the same shape as the memory-layer tradeoff: local-first is not just cheaper. It changes who is allowed to know.
The 46% fabrication number is the part that stuck with me. A memory layer that never touches an LLM trades some recall for something you can actually audit, which feels right when the retrieved context goes straight into a prompt. I would take predictable and inspectable over slightly higher accuracy in most agent setups.
Thanks, though I want to decline part of it, because it credits the design with something it didn't earn.
The 46% is Mem0's number, and being LLM-free isn't what produces it. My memory layer builds with 0 LLM calls, but the abstention behaviour comes from a retrieval threshold, not from the absence of a model. A distillation-based system with the same threshold policy would abstain the same way. Those two properties are independent, and I'd be overclaiming if I let them merge.
"Trades some recall" isn't quite the shape of it either. On LOCOMO it went the other way: n=1,540, gpt-4o judge, RE-call 0.484 against Mem0's 0.444 with a gpt-4o generator (p=0.0065), and 0.466 against 0.412 with gpt-4o-mini (p=0.0002). On BEAM, Mem0 was ahead 0.650 to 0.594. So it's benchmark-dependent rather than a clean trade, and BEAM is adversarial by construction, which makes its numbers an upper bound on difficulty rather than deployed behaviour.
What you're right about is the part I'd actually defend: inspectable is metered, not argued. 0 LLM calls and 0 tokens in the memory layer, so there's no extraction step whose output you have to take on faith.
One caveat on "predictable", since it cuts both ways. Deterministic includes deterministically wrong. On LongMemEval my abstention guard false-abstained on 48.1% of questions while retrieval hit@5 was 0.970. It refused half the questions it had just answered correctly. That failure is reproducible and inspectable, which genuinely beats a stochastic one. It's still a failure.
The Mem0 versus RE-call comparison is the exact trade-off we made deliberately in Opportunity Skill's impression management, and we chose the LLM distillation path with full awareness of its cost. The agent reasons about what should become an impression from daily interactions, distilling attributes and preferences into structured semantic units of at most 512 characters with 1 to 5 tags. This costs LLM calls. But the alternative, storing raw conversation turns and hoping retrieval finds the right signal, produces exactly the temporal reasoning weakness you measured. The 46 percent fabrication rate on unanswerable questions is the number that validates our design choice on the search side. The human discovery module returns an empty list when cosine similarities are insufficient rather than presenting a confident but unsupported match. Silence is architecturally cheaper than a wrong introduction between two professionals. The privacy advantage of RE-call is real though, and it is why impressions in our system require explicit intent to become public. A flag at the server level prevents any information the agent extracts from being externally discoverable without deliberate choice.
Thanks for reading it that closely, you picked up the temporal-reasoning weakness, which is the part most readers skim past, and you described the mechanism the way my own notes do: one distilled line versus a date scattered across raw turns.
Since you're abstaining on cosine in the discovery module, here's what I'd most want to know if I were building that. I measured that guard on three benchmarks and it has a bounded domain. It works when the gap is far, the distributions are disjoint, abstention accuracy 1.00. It falls apart on near-miss: on LongMemEval it false-abstained on 48.1% of questions while retrieval hit@5 was 0.970. It refused half the questions it had just answered correctly. I tried six signals on the same 500 questions: dense cosine AUC 0.753, cross-encoder rerank 0.742, RRF 0.739, QNLI entailment 0.648, margin 0.579, ratio 0.545, and none beat the shipped threshold. The reason looks structural: every one of those is a relevance signal, and relevance is not answerability.
That may matter more for you than it did for me. In professional matching, near-miss is the normal case rather than the tail, two people adjacent on every visible dimension, where the actual attribute is simply absent. The silence is cheapest exactly where it's easiest to be silent.
The concrete one, if it's useful: my threshold constant turned out not to be comparable across embedders. 0.50 sat at the 0th percentile of five of six distributions. Inert on bge-small and bge-large, starving 16% of legitimate queries on text-embedding-3-small. If your cutoff is a hardcoded float, it changes meaning silently the day you upgrade your embedding model. That one took me a while to find.
One caveat on my own number, so it doesn't get pulled out from under you if you cite it. The 46% is a BEAM figure, and BEAM is adversarial by construction: its unanswerable questions ask for details never stated about topics discussed at length, so cosine inverts there, 0.676 on unanswerable versus 0.641 on answerable, with lexical coverage inverting identically. Every BEAM number is an upper bound on difficulty rather than deployed behaviour. On that same 300-question run Mem0 came out ahead on the aggregate (0.650 against my 0.594, accuracy 71.0% against 66.7%), and their published numbers are honest. I re-judged their answers independently and reproduced BEAM 1M to 0.0005. I'd hate for the 46% to travel as "Mem0 fabricates half the time", because the first person to open their repo would be right and I'd be wrong.
And the honest limit on the part you agreed with: BEAM is also the one benchmark that can't price an abstention claim. Every abstention policy I tested on it came out net negative, because false-abstain risk applies to far more questions than abstention can ever win. You may well be right that silence beats a wrong introduction. My number just isn't the evidence for it.
Which is why I'm building a different benchmark, aimed at exactly that gap: abstention gets its own axis with its own manifest, rather than dissolving into an aggregate that structurally can't price it. Temporal reasoning and tenant isolation are separate axes alongside it. There is deliberately no LLM anywhere in it, not in generation, not in scoring. Part of that is an audit finding I'd rather cite than claim, since I haven't reproduced it myself: Penfield Labs found the LOCOMO judge accepts 62.8% of intentionally wrong answers, and a permissive non deterministic scorer compresses every margin toward zero until you can't separate a real effect from judge noise. The larger reason is that I want verification to cost nothing and run locally, so a hostile party can re-run it without an API key and without trusting me.
That choice has a price, and I'd rather state it than have it found. Because the verifier runs locally with no secret and can score any artifact, the verifier holds the answer key. So every submitter holds it too, and roughly twenty lines that open the manifest and echo it back score at the ceiling on every axis and pass all six mechanical checks. It isn't fabrication, which is what makes it hard: it's a real deterministic program, honestly executed, so an independent re-run confirms it, and two people submitting it would earn the "independently reproduced" badge automatically when an honest stochastic system might never agree that closely. I think it's structurally unclosable, salted hashes fail because the corpus is public and the id space is enumerable, and a held-out key destroys the $0 local verification that is the entire point. It's documented as the top limit and pinned by a test. So the honest reading of any score on it is: a ceiling a system did not fail to reach, never evidence that a system exists.
On privacy worth being precise that mine is architectural rather than a policy control: the memory layer meters 0 LLM calls and 0 tokens, so there's nothing extracted to gate in the first place. Yours covers something mine doesn't, though. I have no equivalent of deliberate publication, because I have no publication path at all.
Last thing, and it's a real question rather than a rhetorical one. I observed the temporal win and attributed it to distillation, but I never ablated it. I never ran RE-call with a distillation layer to isolate the cause. It's an inference from a mechanism story, not a measurement. If your build gives you that counterfactual from the other side, I'd rather have your data than my inference.
The "field value confused with the time it was asserted" problem is the one I'd bet most people overlook. In sensor data from robots, we hit this constantly: an IMU reading timestamped at ingestion (when the logger picked it up) vs. at observation (when the sensor fired). On a Raspberry Pi with DMA-backed SPI, that gap runs 40-80ms. Enough to throw off a dead-reckoning estimate by half a meter.
We handle this in moteDB by storing two timestamps per observation,
observed_atandingested_at, and the temporal query layer reasons overobserved_at. That works because we control the write path. RE-call inherits raw turns where that distinction is usually already lost, so the wrong-instance problem becomes structurally unfixable without re-ingestion.The $7.29 per-benchmark-memory number doesn't get enough attention. Scale it to 200 devices writing 500 memories/day each and you're at $729K/year on LLM ingestion alone. The offline argument isn't just privacy. It's unit economics that break at edge scale.
One thing I'd push on: reranking lifted hit@5 from 0.671 to 0.777 on LOCOMO but went unmeasured on BEAM. Have you considered that reranking might hurt temporal reasoning specifically? If the reranker leans on semantic similarity, it could surface the "updated deadline" over the "original deadline" because newer mentions tend to be more semantically rich. That would make the wrong-instance problem worse.
Interesting perspective. At IT Path Solutions, we've found that treating memory as an architectural concern rather than an LLM feature often results in more predictable and maintainable AI systems. Keeping deterministic operations outside the model helps reduce complexity while allowing the LLM to focus on higher-value reasoning tasks.
Some comments may only be visible to logged-in visitors. Sign in to view all comments.