Hybrid search for RAG — BM25 + vector beats pure vector

The textbook RAG pipeline uses vector search alone — embed the question, find the closest chunks by cosine distance. On semantic queries where the user phrases a question differently from the source text, vector search wins. On queries where the user types the exact string they want to find — a product SKU, an error message, a function name — vector search is slower, more expensive, and often less precise than a 30-year-old keyword algorithm called BM25.

Real production traffic is both kinds mixed together. Pure vector loses roughly 20-30% of exact-match queries. Pure BM25 loses 40-60% of paraphrase queries. Hybrid search — running both and combining the results — fixes most of each failure class. This article covers when hybrid wins, how to combine the two scores, and how each major vector DB implements it in late 2026.

Vector search ranks chunks by cosine similarity between the question’s embedding and each chunk’s embedding. Two kinds of queries trip it up:

Exact-match queries. User types E_INVALID_SIGNATURE_v2.3. The embedding of that string is some generic “error-code-shaped vector” that may be closer to a chunk discussing a different error code than to the chunk discussing the exact one asked about. Vector search returns neighbours in semantic space, not string space.

Rare technical terms. The embedding model never saw langgraph.StateGraph during pretraining, so its vector is approximate. A chunk mentioning the exact term may not be the nearest neighbour to the query’s embedding.

Keyword search (BM25) has the opposite blind spots. It cannot bridge paraphrase: “how do I reset my password” and “credential recovery procedure” share zero overlapping tokens and BM25 scores them as unrelated. For a corpus where users ask in their own words and documents are written in the organisation’s words, BM25 alone is poor.

The two methods are complementary. Where one fails, the other usually works.

What hybrid search actually is

Hybrid search runs both retrievers in parallel and merges the results. Three steps:

BM25 pass. Score every chunk against the query using the BM25 algorithm. Return the top-K BM25 chunks with scores.
Vector pass. Embed the query, compute cosine similarity against the chunk embeddings. Return the top-K vector chunks with scores.
Fusion. Merge the two ranked lists into a single ranking.

The fusion step is where most of the practical choice lives. Three standard approaches.

Fusion strategy 1 — Reciprocal Rank Fusion (RRF)

Score each chunk based on its RANK in each list, not its raw score. The score for a chunk that appears at rank r in a retriever is 1 / (k + r) where k is a small constant (typically 60). A chunk appearing in both lists gets summed scores from each.

def reciprocal_rank_fusion(
    rankings: list[list[str]],
    k: int = 60,
) -> list[tuple[str, float]]:
    """
    rankings — a list of ranked chunk ID lists, one per retriever.
    Returns a single fused ranking with scores.
    """
    scores: dict[str, float] = {}
    for ranking in rankings:
        for rank, chunk_id in enumerate(ranking):
            # rank starts at 0 in enumerate, so use rank+1
            scores[chunk_id] = scores.get(chunk_id, 0) + 1.0 / (k + rank + 1)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)


# Example usage
bm25_results = ["chunk_42", "chunk_17", "chunk_8", "chunk_3"]
vector_results = ["chunk_8", "chunk_42", "chunk_19", "chunk_55"]
fused = reciprocal_rank_fusion([bm25_results, vector_results])
# chunk_42 and chunk_8 appear in both — they rank highest

Why RRF wins by default: it does not need the two retrievers to produce comparable score scales. BM25 scores and cosine similarities are on wildly different ranges, and normalising them per-query is fragile. RRF only cares about ranks.

When it loses: when one retriever is systematically better than the other on this corpus and the ranks of the weaker one still contribute equally. The next two strategies address this.

Fusion strategy 2 — Weighted score fusion

Normalise each retriever’s raw score to [0, 1] per query, then combine with weights. Requires either min-max normalisation (sensitive to outliers) or z-score normalisation.

def weighted_fusion(
    bm25_scores: dict[str, float],
    vector_scores: dict[str, float],
    bm25_weight: float = 0.3,
    vector_weight: float = 0.7,
) -> list[tuple[str, float]]:
    def _normalise(scores: dict[str, float]) -> dict[str, float]:
        if not scores:
            return {}
        values = list(scores.values())
        lo, hi = min(values), max(values)
        span = hi - lo if hi > lo else 1.0
        return {k: (v - lo) / span for k, v in scores.items()}

    bm25_n = _normalise(bm25_scores)
    vector_n = _normalise(vector_scores)
    all_chunks = set(bm25_n) | set(vector_n)
    combined = {
        c: bm25_weight * bm25_n.get(c, 0) + vector_weight * vector_n.get(c, 0)
        for c in all_chunks
    }
    return sorted(combined.items(), key=lambda x: x[1], reverse=True)

When this wins: when one retriever is clearly stronger (e.g. 0.7 weight on vector for mostly-paraphrase corpora, 0.3 for mostly-exact). Measurable on an eval set.

When it loses: when the weight needs to vary per query. A product-code query wants high BM25 weight; a conversational query wants high vector weight. Fixed weights settle for an average that is suboptimal for both.

Fusion strategy 3 — Cross-encoder reranker on union

Instead of fusing scores, concatenate the top-K from each retriever (deduplicated), then rerank the union using a cross-encoder model (a model that scores the question and chunk together, rather than embedding them separately).

# Pseudocode — depends on reranker library (cohere.rerank, bge-reranker, cross-encoder)
candidate_chunks = set(bm25_top_20 + vector_top_20)
reranked = rerank_model.score(question, candidate_chunks)
final_top_k = sorted(reranked, key=lambda x: x.score, reverse=True)[:5]

When this wins: quality-critical workloads where the extra 100-300ms is acceptable. Cross-encoders see the question and chunk together, so they catch subtle relevance mismatches that neither BM25 nor vector-alone score correctly.

When it loses: latency budgets below 500ms end-to-end. Each cross-encoder call is one additional model invocation per candidate.

Default pick in 2026: RRF for everything first. Upgrade to reranker on union once the eval set shows RRF is leaving quality on the table.

When hybrid is overkill

Three workload shapes where going straight to pure vector is fine:

Conversational Q&A over a help-content corpus where users nearly always phrase questions in their own words. BM25 contributes <5% uplift.
Small corpora under ~10K chunks where retrieval precision is high regardless of method. Hybrid adds complexity for marginal gain.
Highly structured corpora where metadata filters do the heavy lifting (filter by product, by date, by region) and the vector search is only breaking ties within an already-narrow candidate set.

Three workload shapes where going straight to pure BM25 is fine:

Code search where users search by exact function or class names. Vector adds noise.
Error-message search where the user has pasted a stack trace token. BM25 is exact and fast.
Product search over SKU-heavy catalogues. Exact match dominates.

Hybrid is the right default when the corpus is text-heavy AND users mix natural-language and exact-string queries. Which is most real workloads.

How each vector DB implements hybrid in late 2026

Qdrant — Hybrid is native via sparse-dense vectors in the same collection. Index a sparse vector (BM25-like, using fastembed or similar sparse encoders) alongside the dense embedding. Query with both, Qdrant does the fusion internally. Supports RRF out of the box.

Weaviate — Native hybrid operator in the GraphQL/REST API. Specify an alpha parameter from 0.0 (pure BM25) to 1.0 (pure vector); Weaviate handles normalisation and fusion. Simplest API among the four for hybrid.

Pinecone — Hybrid via sparse-dense indexes using pinecone-text for sparse encoding. Query sends both sparse and dense vectors; Pinecone fuses with its own weighted-score strategy. Not RRF.

pgvector — Not native. Compose manually: a tsvector column with Postgres full-text search for BM25, a vector column for dense. Query both in SQL with a UNION or a CTE, apply RRF or weighted scoring in SQL. More code but zero extra infrastructure if you’re already on Postgres.

-- pgvector hybrid search sketch (RRF in SQL)
WITH bm25_results AS (
    SELECT id, ROW_NUMBER() OVER (ORDER BY ts_rank_cd(ts, query) DESC) AS rank
    FROM docs, plainto_tsquery('english', 'user question here') AS query
    WHERE ts @@ query
    LIMIT 20
),
vector_results AS (
    SELECT id, ROW_NUMBER() OVER (ORDER BY embedding <=> $1::vector) AS rank
    FROM docs
    ORDER BY embedding <=> $1::vector
    LIMIT 20
),
fused AS (
    SELECT id, 1.0 / (60 + rank) AS score FROM bm25_results
    UNION ALL
    SELECT id, 1.0 / (60 + rank) AS score FROM vector_results
)
SELECT id, SUM(score) AS rrf_score
FROM fused
GROUP BY id
ORDER BY rrf_score DESC
LIMIT 10;

Six lines of SQL for a hybrid retriever. Not pretty but correct.

The eval question nobody asks before enabling hybrid

Before turning hybrid on in production, measure both pure methods and hybrid on the same eval set. Three outcomes:

Hybrid > both pure methods by >10%. Keep hybrid. The complexity is paying off.
Hybrid > one but equals the other. Use the pure winning method. Hybrid adds work without gain.
Hybrid < either pure method. Something is wrong — fusion weights off, one retriever broken, score normalisation leaking. Investigate before shipping.

The common mistake: enabling hybrid in a vector DB that supports it natively, assuming it must be an improvement, and never measuring. On corpora where BM25 contributes noise (very conversational Q&A), hybrid can regress quality by 5-10% versus pure vector.

Evaluating RAG — retrieval + generation metrics that matter covers the eval harness that catches these regressions.

Bottom line

Hybrid search is the right default for RAG when the corpus contains a mix of natural-language and exact-string information needs. For most B2B document search, developer documentation, legal and policy corpora, hybrid outperforms pure vector by 10-25% on recall at the same precision — numbers verified against the organisation’s own eval set, not against vendor benchmarks.

The implementation cost is low on Qdrant and Weaviate (native), moderate on Pinecone (sparse-dense indexes), and SQL-level on pgvector (manual composition). RRF as the fusion strategy, cross-encoder reranker on the union when quality matters more than latency. Measure both the pure baselines and hybrid before committing.