Should our LLM wiki use vector search? We benchmarked it before deciding

Every "add vector search" proposal comes with the same promise: users phrase questions their own way, keyword search misses, and embeddings fix it. A feature request on Synthadoc made that case for us (issue #411):

Synthadoc uses BM25 exclusively. For wikis under ~200 pages, BM25 is fast and precise; at 500+ pages, paraphrase and concept queries increasingly miss without a semantic layer.

It also asked us to turn embeddings on by default. That's a big change for a pip-installed tool (more on the cost below), so before writing any retrieval code we built a benchmark to answer three questions:

1. Does BM25 really get worse at paraphrased questions as a wiki grows?
2. Does the vector search we already ship (off by default) fix it?
3. If not, what design would and by how much?

• Yes, keyword search degrades with wiki size — but only for paraphrases. On a 2,000-page wiki, BM25 put the right page in the top 5 for 48% of paraphrased questions, down from 71% at 200 pages. Questions that use the page's own words or an exact name stayed at ~99–100% at every size.
• Our existing "vector re-rank" mode would have made search worse overall. It lifted paraphrases (48% → 60%) but wrecked exact-name lookups (99% → 63%), for a net drop from 84% to 74%.
• Fusing the two rankings fixed paraphrases (48% → 69%) and lifted recall overall, but exact-name lookups still slipped to 94%.
• One more rule closed that gap: when BM25 has a clear winner (its best score at least 5% ahead of the next), keep BM25's top hits on top. Final result on 2,000 pages: paraphrases 48% → 68%, exact names 99% → 98%, page's-own-words 100% → 100%, overall 84% → 90% — and the right page reaches the top 20 for 88% of paraphrases instead of 65%.
• Bonus: the benchmark exposed that BM25 rebuilt its index on every query. Caching it made keyword search 18× faster (P95 362 ms → 20 ms) for every user, embeddings or not.
• It held up on data we never tuned on. On two English corpora with human-written queries and human relevance judgements (science claims, medical questions), semantic search was better on every measure. On Chinese it was worse - our embedding model is English-only - so Chinese, Japanese and Korean pages and questions now stay on keyword search, automatically.
• So it's now on by default for new wikis (vector = "auto"): semantic search whenever the add-on is installed, which the Docker image is.
• And faster: running the query embedding alongside BM25 instead of after it cut semantic search from 166 ms to 62 ms (P95, 2,000 pages).
• The same scale-up broke our knowledge graph. At 5,000 pages the graph view took 34 seconds to appear and then froze the browser. It now opens on a cluster overview in about 6 seconds and runs at a smooth 60 fps.

Background: how Synthadoc finds pages today

Synthadoc is an open-source tool that compiles your sources — PDFs, web pages, notes — into a Markdown wiki (it works nicely as an Obsidian vault), then answers questions from that wiki with citations. Answer quality depends almost entirely on one step: finding the right pages to hand to the LLM.

Today that step is BM25, the classic keyword-ranking function behind most search engines before neural search. It scores pages by how many of the query's words they contain, weighting rare words more. It's fast, needs no model, and is excellent when the user uses the same words as the page.

We also ship an optional [search] vector = true mode. It takes BM25's top 20 results and re-sorts them by embedding similarity - how close the query and the page are in meaning, according to a small embedding model (BAAI/bge-small-en-v1.5, run locally via fastembed).

Keyword vs. vector, in one example

Each approach fails where the other is strong. That's the whole story of this post in one table - but we wanted numbers, not intuition.

Why measure first

Turning embeddings on by default isn't free for a pip-installed tool:

• +~175 MB of dependencies (onnxruntime alone is 80 MB) and a 65 MB model download on first start — awkward for offline machines and slim Docker images.
• An English-only model, in a project that also serves Chinese, Japanese and Korean users.
• A retrieval change touches every answer the product gives.

If the gain is small, or only shows up on wikis far bigger than our users run, that cost isn't worth paying by default.

How we built the benchmark

We wanted something realistic, reproducible, cheap to run, and small enough to keep in the main repo.

A wiki-like corpus. We pinned 1,985 English Wikipedia articles from 24 overlapping computing categories - history of computing, AI, cryptography, databases, networking, and so on - each at a fixed revision ID, so the text never changes under us. Overlap matters: paraphrase misses happen when many similar pages compete. Each article is converted to a Synthadoc wiki page (lead plus sections, capped at 1,500 words, roughly the size of a compiled wiki page).

Three kinds of question per page. For 500 randomly sampled pages, an LLM (Claude Sonnet 5.5) wrote three questions that page answers:

• lexical - uses the page's own key terms;
• paraphrase - the same need, worded by someone who doesn't know the terminology: no words from the title, and at most two of the page's distinctive terms (checked mechanically, not trusted to the LLM);
• identifier - a short lookup of an exact name, acronym, number or date.

That gave 1,430 questions (some paraphrases failed the check twice and were dropped rather than mislabelled).

A human-written cross-check. LLM-written paraphrases might suit embedding models unfairly well, so we also ran 300 real questions from HotpotQA over its ~3,000 Wikipedia paragraphs.

Wiki sizes. We scored the same questions on nested 200, 500, 1,000 and 1,985-page wikis to test the "it only matters past ~200 pages" claim.

Retrieval only, no LLM in the loop. Each setup is the real Synthadoc search code, called the way the query agent calls it. Scoring a run needs no API key and gives the same numbers every time.

Small in git. Only titles + revision IDs, the frozen questions, HotpotQA question IDs and the results are committed — about 0.5 MB. A script downloads and caches the text (both sources are CC BY-SA).

Metrics. Recall@5 - how often the right page is in the top 5 results we hand to the LLM. We also tracked Recall@10/20, MRR and nDCG (how high it ranks) and P95 latency.

Two bugs the benchmark caught in itself

Worth sharing, because both would have silently produced a wrong conclusion:

1. Our "distinctive term" check thought "the" was distinctive. The first generation run threw away 74% of paraphrases. The checker reused our stopword list, which only covers question words ("what", "how") - so "the", "and" and "for" counted as giveaway terms. We switched to corpus-wide rarity: a word only counts if it appears on ≤10% of pages. Accepted paraphrases went from 26% to 87%.
2. Two random samples with the same seed picked disjoint pages. Python's random.sample and random.shuffle walk the same random stream, so our "which pages get questions" sample and our "which pages form the smaller wiki" shuffle, both seeded with 411, never overlapped. Every smaller wiki scored zero questions. Now they use different seeds, with a test that checks the overlap is proportional.

What we found

1. BM25 is near-perfect on its own words, and steadily worse on paraphrases

Recall@5, BM25 only:

The issue was right, and now we know by how much: every doubling of the wiki costs roughly 7–8 points of paraphrase recall. Even Recall@20 for paraphrases falls from 0.80 to 0.65 - a third of those questions never get the right page into the candidate list at all.

2. Our existing vector mode helps paraphrases and breaks exact lookups

Recall@5 on the full 1,985-page wiki:

Three reasons, all visible in the code once the benchmark pointed at them:

• It throws away BM25's order. Re-ranking sorts purely by embedding similarity, so a page that matched "Moore School 1946" exactly can drop below pages that are merely about early computers.
• It only embeds each page's first 512 characters. A name or number further down the page is invisible to the embedding.
• It can only reorder what BM25 already found. Paraphrase Recall@20 is 0.646 with or without it - re-ranking can't rescue a page BM25 missed.

On HotpotQA's human-written questions the re-rank looked great (Recall@5 0.717 → 0.842), but HotpotQA's distractor paragraphs were chosen for word overlap with the question - it's built to be hard for keyword search. On our own kind of content, the picture is very different. This is exactly why we benchmark on our own use case.

It's also the slowest setup: every query reloads every stored embedding from SQLite.

3. Fusion: let each method vote

If BM25 and embeddings each fail where the other is strong, don't make one the judge of the other. Our prototype runs both over the whole wiki, takes the top 20 from each, and merges them with reciprocal rank fusion (RRF): every page scores 1 / (60 + rank) in each list it appears in, and the scores are added. No score calibration - a page near the top of either list surfaces, and a page high in both wins.

That fixed paraphrases (Recall@5 0.479 → 0.694) but still cost exact-name lookups (0.994 → 0.936): an exact BM25 hit now shared the vote with pages that were merely similar in meaning.

4. Knowing when to trust the keywords

Rather than guess at weights, we asked the data: when is BM25's top result right? One signal stood out — how far BM25's best score is ahead of its second. When the top score is at least 1.2× the next:

A dominant keyword match almost always means an exact name or term hit, and it rarely happens for paraphrases. So the final rule is simple: when BM25's best score is at least 1.05× its second, keep BM25's top 3 on top; otherwise let RRF decide. (We also tried simply weighting BM25's vote higher. It backfired: BM25's entries then always outrank the embedding ones, and fusion collapses back into keyword search.)

Recall@5 on the full 1,985-page wiki:

It's the only setup that beats BM25 on every overall measure, including how high the right page ranks (MRR). And the part re-ranking could never do - paraphrase recall within the top 20, i.e. whether the right page reaches the LLM's candidate list at all:

The gain grows with the wiki — paraphrase Recall@5 improves by 9 points at 200 pages, 11 at 500, 14 at 1,000 and 20 at 1,985 — which is exactly the shape the original issue predicted. On HotpotQA's human-written questions, fusion scores 0.815 at top 5 (BM25: 0.717) and 0.970 at top 20 (BM25: 0.920).

The handful of exact-name questions it still misses are mostly ambiguous labels: "A Mathematical Theory of Communication 1948" is cited on several pages about information theory, but only one is labelled correct.

5. The bug we found in plain sight: BM25 rebuilt its index on every query

BM25 latency grew from 77 ms at 200 pages to 362 ms at 1,985. Profiling one query on the full wiki:

Synthadoc cached the tokenised pages but rebuilt the index on every search. Caching the built index (per search scope, cleared whenever a page changes) took BM25's P95 from 362 ms to 20 ms at 1,985 pages with identical results - a speed-up every user gets, whether or not they turn on embeddings.

Fusion's P95 was 122–166 ms, and profiling showed why: the search ran BM25, then embedded the query, so the two costs added up. Embedding the query is about 20 ms on a laptop CPU; comparing it with every page embedding in memory takes under 2 ms. Both steps are independent, and the embedding model releases Python's GIL while it runs, so we now start the query embedding in a worker thread before BM25 runs. Same results, and P95 at 1,985 pages dropped from 166 ms to 62 ms (BM25 alone: 15 ms). We also keep the last 256 query embeddings - an embedding depends only on the text and the model, so a cache entry can never go stale. Either way, that is well under 1% of an answer that spends seconds in the LLM.

6. The "better" embeddings that made things worse

Our prototype embedded only each page's title, tags and first 512 characters, so the obvious next step was section-level embeddings: one per heading, so content deep in a long page is searchable. On its own, the embedding search got clearly better - paraphrase Recall@5 0.616 → 0.669, and exact-name lookups 0.305 → 0.659.

Fused with BM25, it got worse:

The more precise section embeddings mostly found the pages BM25 already found. Fusion gains from the two methods disagreeing - each catching what the other misses — and the coarser page embeddings disagreed with BM25 in more useful places. Twenty times the compute for a worse result was an easy call: we ship one embedding per page.

7. Checking it on data we didn't tune on

Every threshold above - RRF's k, the 1.05× pin, the top 3 - was chosen on our Wikipedia corpus. A fair test needs data the design has never seen, with questions people wrote and relevance people judged. We added three public retrieval sets:

• SciFact - 5,183 scientific abstracts, 300 claims written by experts;
• NFCorpus - 3,633 medical documents, 323 human queries (~38 relevant documents each);
• DuRetrieval - 3,000 Chinese web passages, 300 human queries. Not because we expected it to work, but because "on by default" would include every Chinese, Japanese and Korean user of our Docker image.

English held up everywhere. Chinese didn't: recall@5 fell from 0.751 to 0.645. That is what an English-only model does with Chinese text - it turns both the question and the pages into embeddings that mostly reflect noise, and fusion dutifully mixes that noise into the results.

The fix didn't need a new model, just two rules applied automatically:

1. Per page: a page whose text is mostly Chinese, Japanese or Korean (CJK characters at least 30% of its letters) is not embedded. It stays fully searchable by keyword - BM25 handles Chinese well.
2. Per question: a question containing CJK characters is answered by BM25 alone.

A mixed wiki gets semantic search for its English pages and keyword search for the rest. With the rules in place, DuRetrieval scores exactly what BM25 scores, and every English result is unchanged:

A multilingual model could turn that Chinese tie into a win; that's a separate experiment, with its own benchmark run.

Then a hand-built test wiki caught one more problem. We made a small mixed wiki - six pages in Chinese (Turing, the transistor, the von Neumann architecture, the origins of the internet, Chinese character input, China's first computers) and three in English (Grace Hopper, Unix, the Linux kernel) - and asked it questions through the real query path. Startup logged exactly what we wanted:

Vector search: embedding 3 page(s)… (6 mostly Chinese/Japanese/Korean page(s) left to keyword search)

But every Chinese question came back with the English pages. The query agent translates CJK questions into English before searching - a feature built for English wikis asked in Chinese - and it searched only the translation. "Who invented the transistor" can't match a page written in Chinese, so the right page never reached the LLM. The benchmark missed this because it calls search directly, without the agent's translation step.

The fix: search the question as written and its translation, and merge the results. Now each language finds its own pages:

The last row is the honest limit of an English-only model, and the best case for trying a multilingual one.

Vector vs. our current approach - what changes

The benefit for users is simple: ask in your own words and still get the page that answers you, without losing the precision you have today when you search for a name, a product code or an acronym.

What we shipped (in v1.5.0 release)

Everything above now runs in Synthadoc's real search path, and the benchmark measures the production code, not a prototype - it reproduces the prototype's numbers exactly:

1. Fusion with the keyword pin for queries, MCP search and cross-wiki retrieval. Ingest's "which page does this belong to?" step stays on BM25 for now: changing it changes what ingest writes, which this benchmark doesn't measure.
2. Embeddings held in memory as one matrix, not reloaded from SQLite per query.
3. Fresh embeddings - each page's embedding carries a hash of its text; a background sync re-embeds pages that changed and drops pages that were archived, at startup, after ingest, and after any edit. Until the first sync finishes, search simply stays on BM25.
4. Honest gap detection - "the wiki doesn't cover this" is still judged on keyword evidence only, so a page the embeddings merely think is related can't hide a real gap.
5. Cross-wiki ranking by rank, not raw score - results from several wikis are interleaved by each wiki's own ranking, because BM25 scores were never comparable between wikis.
6. BM25 18× faster for everyone, embeddings or not.
7. A Docker image with the model built in, so containers start without a download and work offline.
8. Chinese, Japanese and Korean left to keyword search - per page and per question, as above - and a translated CJK question is searched as written too, so pages in either language are found. Nothing to configure.
9. Query embedding alongside BM25, plus a small cache of recent query embeddings: 166 ms → 62 ms P95 at 2,000 pages, identical results.
10. On by default for new wikis - see below.

Meanwhile, in the knowledge graph

Search wasn't the only thing that had only ever met small wikis. Synthadoc's web UI has a knowledge graph view: every page is a dot, links between pages are lines, and pages are coloured by topic cluster (found with the Louvain community-detection
algorithm). With the benchmark wiki at hand we opened it on bigger wikis, then timed it in a headless browser on generated wikis of up to 5,000 pages.

Before: all 5,000 pages and 100,801 links drawn at once. Once the layout settles, after about half a minute of a frozen browser, the clusters show as colour clumps. But none of them has a name, the legend just says "Cluster 0" to "Cluster 23", and the individual pages are too small to pick out.

At 5,000 pages the browser had to animate about 115,000 SVG elements, roughly 100,000 of them links. The clumps are pretty, but they don't tell you what the wiki is about. The server was slow too: building the graph response read and parsed every page's YAML front matter with the pure-Python parser.

The fix is the same idea as fusion: show the reader what they can actually use.

• Cluster overview first. Past 300 pages, the graph opens on one bubble per topic cluster. Each bubble is sized by page count and named after its most-connected pages, joined by the strongest links between clusters. Click a bubble to see its pages, with each page's strongest few links; one click takes you back. Smaller wikis look exactly as before.
• A faster server. Front matter is parsed with libyaml's C loader when it's available (about 7× faster, with identical output); links between pages that share a source are found through an index instead of comparing every pair; and whole-wiki scans run off the event loop, so they don't stall other requests.
• Stable colours. Louvain now uses a fixed seed, so clusters keep their colours from one lint run to the next.

Opening the largest cluster (289 pages) on the 5,000-page wiki takes about 2 seconds, also at 60 fps.

After: the same 5,000 pages as 24 topic clusters. A bubble's size and the number after its name give the cluster's page count. Thicker lines mean more links between two clusters. The list below the graph names every cluster, largest first, and clicking a name opens that cluster. The overview draws about 100 shapes instead of 115,000.

Inside one cluster: 268 pages, drawn with each page's strongest 605 links rather than every link. Clicking a page opens the usual side panel with its type, its total connections (41 here, links to other clusters included) and suggested questions to ask the wiki. "← All clusters" goes back.

The faster YAML parsing helps more than the graph: every feature that reads pages gets it, search included.

On by default — for new wikis

Before starting we set a bar for making semantic search the default:
paraphrase Recall@5 up by at least 15 points, lexical and identifier down by no more than 1, latency within our existing targets. The shipped build clears it on the full wiki: +19.7 for paraphrases, 0 for lexical, −1.0 for identifiers, 62 ms P95. Identifiers sat exactly on the line, which is why we waited for data the design had never seen. The independent check settled it: better on both untuned English corpora, and — once CJK text is left to BM25 - never worse on Chinese.

So new wikis are now created with vector = "auto":

Existing wikis keep the vector = false already in their config; we never rewrite a user's settings. Changing it to "auto" or true turns semantic search on.

Caveats

• Questions are generated, and each has one labelled page; another page might also answer it. That lowers every setup's scores equally.
• The fusion thresholds were chosen on the Wikipedia corpus; SciFact, NFCorpus and DuRetrieval are the independent check. The DuRetrieval corpus is a 3,000-passage slice, so each query's distractors are other queries' answers.
• We skip the query agent's LLM step that splits complex questions into sub-questions, so absolute numbers are a lower bound; comparisons between setups still hold.
• One embedding model, English only. Only Chinese was measured among CJK languages; Japanese and Korean are handled the same way, unmeasured. A multilingual model is a follow-up.
• The graph timings use generated wikis: the benchmark's articles, repeated past 1,985 pages, with random links inside each category. Real wikis have more structure, so their clusters usually look cleaner than ours.

Try it yourself

The benchmark lives in the repo under
benchmarks/retrieval/:

git clone https://github.com/axoviq-ai/synthadoc.git && cd synthadoc
pip install -e ".[vectors]"
python -m benchmarks.retrieval fetch       # once: download + build the wikis (~30 min)
python -m benchmarks.retrieval evaluate --configs bm25,hybrid
python -m benchmarks.retrieval evaluate --corpora scifact,nfcorpus,duretrieval --configs bm25,hybrid
python -m benchmarks.retrieval charts      # redraw the charts in this post

No API key needed: scoring runs only the search code, and gives the same numbers every time.

If you run a search or RAG feature, our one piece of advice: benchmark
retrieval on your own kind of content, split by question type. An average would have told us vector re-ranking was a small loss. The per-type split told us why — and what to build instead.

Acknowledgments

A special thank you to [Chaire Zhang] (@clairezhang-228) for her invaluable contributions to this post. Her insights on rigorous work on the benchmark validations made this piece possible.