BM25

Overview

BM25 is a classic lexical retrieval ranking function used in search systems to score how well a document matches a query based on term frequency, document length, and related weighting factors. In AI product workflows, BM25 remains highly useful because it is fast, interpretable, and effective for retrieving exact or near-exact matches from structured corpora such as tool definitions, internal documentation, specs, and knowledge bases.

For AI Product Managers, BM25 matters because retrieval quality directly affects downstream model behavior. When used to select only the most relevant tools, prompts, or documents, BM25 can improve tool-selection accuracy, reduce irrelevant context sent to a model, and lower context-window bloat. It also plays an important role in hybrid search stacks, where lexical retrieval is combined with embeddings or probabilistic signals to produce more reliable retrieval pipelines.

Key Developments

• 2026-03-07 — Doug Turnbull’s note Can BM25 be a probability? explored whether BM25 scores can be interpreted as odds or probabilities, introducing a Bayesian framing and discussing implications for calibrating hybrid search systems.
• 2026-03-10 — Doug Turnbull shared Ugly hack to force BM25 to 0-1, a practical normalization technique to map BM25 scores into a 0–1 range so they are easier to compare with other retrieval signals.
• 2026-04-20 — In Claude Design’s export-to-Canva workflow, the Tool Search Tool used BM25 to load only relevant tool definitions, reducing context-window usage and improving tool-selection accuracy.

Relevance to AI PMs

• Improve retrieval precision in agentic systems: BM25 is useful when an AI product must choose from many tools, documents, or actions. PMs can use it to ensure the model sees only the most relevant candidates instead of an entire registry.
• Reduce cost and context-window bloat: By filtering large sets of tool definitions or knowledge snippets before they are passed into a model, BM25 can cut token usage and improve latency without requiring a fully embedding-first architecture.
• Design better hybrid search experiences: PMs working on search, RAG, or tool routing can use BM25 alongside semantic retrieval. Understanding score calibration and normalization helps when combining lexical and vector-based relevance signals.

Related

• Doug Turnbull — Frequently associated with practical and theoretical explorations of BM25 scoring, calibration, and search relevance.
• Hybrid search — BM25 is often one half of a hybrid retrieval system, paired with semantic or embedding-based ranking.
• Tool Search Tool — An example application where BM25 is used to retrieve only the most relevant tool definitions for model use.
• Claude Design — Product context where BM25-powered tool retrieval was highlighted as a way to improve accuracy and reduce context usage.