RAG on Tables: Why BM25 Still Beats Vector Search

On documents that mix prose and tables — invoices, order confirmations, annual accounts, service reports — retrieval fails far more often than teams assume, and 73% of those failures come from table structure, not from vocabulary. A 2026 benchmark across 23,088 queries found plain BM25 keyword search beating a state-of-the-art embedding model on almost every metric. The fix is cheaper than the stack most teams are buying.

Why do embeddings miss the row you asked for?

An embedding model compresses a passage into a single vector that represents its overall meaning. That works when a passage is about one thing. A table is not about one thing. A 40-row price list, an invoice line table, a general ledger extract — each row is a separate fact, and averaging them produces a vector that sits nowhere useful in the embedding space.

The second problem is identifiers. Ask “what was net income in 2019?” of a document where “net income” is a row label and “2019” is a column header, and the two tokens that define the answer never appear adjacent in the serialised text. Ask about invoice 2026-04412 and you are searching for a near-random string. Semantic similarity has nothing to work with; lexical matching, by contrast, treats a rare token as a strong discriminator. That is exactly what inverse document frequency is for.

This is not a new intuition. What changed in 2026 is that somebody measured it properly.

What did the 2026 benchmark actually measure?

A team from Technische Hochschule Ingolstadt, Uludağ University and Radiate benchmarked ten retrieval strategies over T²-RAGBench: 23,088 queries across 7,318 documents containing mixed text and tabular content. Documents average 920 tokens and were indexed whole, without chunking, deliberately isolating the retrieval method from chunking decisions. Every adjacent pair of results is statistically significant at p<0.001 under a paired bootstrap test with 10,000 resamples and Bonferroni correction.

Recall@5, the share of queries where the correct document appears in the top five:

• BM25 (sparse, lexical): 0.644
• Dense retrieval, text-embedding-3-large: 0.587
• HyDE (LLM-generated hypothetical document): 0.544
• Multi-query expansion with RRF: 0.640
• Hybrid BM25 + dense, reciprocal rank fusion: 0.695
• Contextual hybrid (LLM summary prepended at index time): 0.717
• Hybrid + cross-encoder reranking: 0.816

BM25 beat one of the strongest commercial embedding models on every metric except Recall@20, where the two effectively tie at 0.797 against 0.798. Read that back slowly: on this document class, the free component that ships inside your database outperformed the paid one you call over the network.

The authors are candid about scope, and so should anyone quoting them be. It is one corpus, financial in nature, with numeric answers, one embedding model, and whole-document retrieval. What generalises is the mechanism, not the decimal places.

Where does retrieval actually break?

The most useful table in the paper is the error analysis. Of the 7,188 queries (31.1%) where hybrid retrieval missed the correct document in its top five, the authors sampled 100 and categorised the failures:

• Table structure mismatch — 73%
• Numerical reasoning required — 20%
• Vocabulary mismatch — 5%
• Ambiguous query — 1%
• Document too long — 1%

In 71% of failures the correct document appeared in neither the dense nor the BM25 top five, so these are genuinely hard cases rather than artefacts of how the two rankings were merged.

That distribution should reorder your backlog. Swapping to a better embedding model attacks the 5%. Changing how tables enter the index attacks the 73%. Most teams we meet have spent their last three sprints on the 5%.

Why did query rewriting make things worse?

HyDE — having an LLM write a hypothetical answer and embedding that instead of the question — scored 0.544, below plain dense retrieval at 0.587. The reason is specific and worth internalising: when the question demands a precise figure, the model invents a plausible one, and the hypothetical document drags the query vector towards a number that does not exist anywhere in the corpus. Query expansion helps when the problem is phrasing. It hurts when the problem is precision.

Multi-query expansion landed at 0.640, practically indistinguishable from doing nothing beyond BM25's 0.644. Corrective RAG reached 0.658 — better than either single retriever, worse than simple fusion at 0.695 — and notably, 63% of queries (14,569 of 23,088) triggered its correction pathway. A retriever that needs correcting on nearly two of every three queries is telling you something about the retriever, not about the queries.

The general lesson is one we keep relearning in production agent work: an agentic loop wrapped around a weak retriever spends more money to be wrong more elaborately.

What should you fix first?

An order of operations, cheapest and highest-yield first:

1. Parse tables into retrievable units. Row-level or record-level chunks, with the column headers and the document header carried into every row so a single chunk is self-describing. This is unglamorous engineering and it addresses the dominant failure mode.
2. Add a lexical index beside the vector index and fuse the rankings. Reciprocal rank fusion with k=60 is the common default, but the paper's ablation put convex combination at α=0.5 on 0.726 and RRF with k=10 on 0.716, against 0.695 for the default. That is a configuration change worth an afternoon.
3. Rerank — over a deep enough candidate pool. This is where the biggest single gain sits, and where most implementations quietly waste it. Reranking 20 candidates scored 0.458. Fifty candidates scored 0.826. A hundred scored 0.888. (The main table reports 0.816 for what is effectively the same two-stage pipeline the depth ablation puts at 0.826; treat that gap as measurement noise rather than a tuning gain.) If you rerank your top ten, you are paying for a cross-encoder to reorder documents that mostly do not contain the answer.
4. Enrich at index time. Prepending a short LLM-written context summary to each document — company, period, key metrics — added 2.8 points to dense and 2.2 to hybrid retrieval, at a one-off indexing cost.
5. Only then reach for a bigger model.

Is a better model cheaper than better retrieval?

The benchmark answers this directly, and the answer is uncomfortable for anyone whose improvement plan is a model upgrade.

Holding retrieval constant at hybrid fusion and swapping GPT-4.1-mini for GPT-5.4 moved end-to-end answer accuracy from 0.282 to 0.346, a gain of 6.4 points. Holding the small model constant and replacing hybrid retrieval with oracle retrieval — the correct document handed over every time — moved it from 0.282 to 0.350, a gain of 6.8 points. Fixing retrieval was worth slightly more than a frontier-model upgrade, and the two compound: the large model on oracle context reached 0.403.

Retrieval quality and answer quality correlated at r = 0.98 across the retrieval configurations they ran end to end. There is no prompt that recovers a document the model never received, and no context window large enough to substitute for knowing which document to put in it.

One honest caveat: oracle retrieval is an upper bound, not something you can purchase. The best configuration the study actually achieved — hybrid fusion plus reranking — reached 0.816 Recall@5, not 1.0. The comparison tells you where the marginal euro goes, not that perfect retrieval is on the price list.

Does this hold for Dutch business documents?

The mechanism travels even though the benchmark does not. The corpora Dutch SMEs actually want to query are unusually identifier-dense: factuurnummers, artikelnummers, ordernummers, BTW-nummers, KvK-nummers, postcodes, and the structured field names that arrive with UBL and Peppol e-invoices. Exports from Exact Online, AFAS and e-Boekhouden are tables first and prose second. A werkbon is a header block plus a line table. Ask “hoeveel uur is er in juni op project 4412 geschreven?” and every token that identifies the answer is a rare string — the best possible input for a lexical index and close to the worst for a dense one.

Two practical consequences for a 20-50 person company already sitting on Postgres. First, a lexical index is a configuration value rather than a project: Postgres ships stemming configurations for 28 languages including Dutch, and pgvector can live in the same table as a full-text index. If you want real BM25 ranking rather than ts_rank, pg_textsearch and pg_search both provide it as an extension, and the fusion query is a single statement. No second datastore, no sync job, no consistency problem.

Second, a mixed Dutch/English corpus — which is most Dutch technical documentation — punishes a single embedding model harder than it punishes BM25, because lexical matching never has to place two languages in one geometry.

And the structural caveat: if the answer lives in a system of record rather than a document — hours in the ERP, stock in the WMS, balances in the bookkeeping package — then retrieval is the wrong mechanism entirely and a tool call against the database is the right one. We wrote about that distinction in training AI on your own data, and it remains the most common reason a document-search project underdelivers. Related: how the retrieval layer fits the wider stack in RAG vs GraphRAG, and what the same problem looks like in invoice processing.

How do you test this on your own corpus?

You can answer the question for your own documents in an afternoon, and the answer is worth more than any benchmark someone else ran.

1. Build a gold set of 50 real questions people have actually asked, each annotated with the document or row that contains the answer. Fifty is enough to see a ten-point difference; it is not enough to publish.
2. Measure Recall@5 for your current retriever. That number is the hard ceiling on your answer quality — the correlation above is not a coincidence.
3. Add a lexical index over the same chunks and fuse the two rankings with RRF. Re-measure. If the number moves materially, your embedding budget was aimed at the wrong layer.
4. Add reranking over at least 50 candidates and re-measure. Cohere's Rerank 4, released in December 2025, prices the Rerank 4 Pro model at $0.0025 per search unit, where a unit is one query against up to 100 documents. At 200 queries a day over 50 candidates that is one unit per query: on the order of $15 a month, on top of a first-stage retrieval you already run. Note what moves that bill — the candidate count, not the query count. Go past 100 candidates and every query costs two units.
5. Categorise the misses that remain. If they cluster on table structure, go back to step one — no amount of ranking rescues a chunk that never contained the answer.

That loop is also the honest way to scope the work. It tells you whether you have a retrieval problem, a parsing problem, or a data problem, before anyone commits to a budget. If you want the wider cost picture first, we keep the numbers on what an AI project costs; if the retrieval layer is already the bottleneck in an agent you are building, that is the work we do.

Last updated 21 August 2026.

Frequently asked questions