BM25

Overview

BM25 is a classic lexical retrieval ranking function used to score how relevant a document is to a query based on term matching, frequency, and document length normalization. In AI product systems, BM25 is often used as a fast, reliable first-pass retrieval method for finding the most relevant items from a large corpus—such as tool definitions, documentation, prompts, tickets, or knowledge-base articles.

For AI Product Managers, BM25 matters because it improves retrieval precision without requiring embeddings or heavy model inference for every lookup. In practical PM tooling, it can reduce context-window bloat by selecting only the most relevant tool or knowledge definitions before passing them to an LLM. It also plays an important role in hybrid search stacks, where lexical matching is combined with semantic retrieval and reranking to improve both relevance and controllability.

Key Developments

• 2026-03-07 — Doug Turnbull’s note Can BM25 be a probability? explored whether BM25 scores can be interpreted as odds or probabilities, introducing a Bayesian framing and highlighting implications for calibrating hybrid search systems.
• 2026-03-10 — Doug Turnbull shared Ugly hack to force BM25 to 0-1, a pragmatic normalization technique intended to make BM25 scores easier to compare and operationalize in retrieval pipelines.
• 2026-04-20 — In coverage of Claude Design’s export-to-Canva workflow, the Tool Search Tool was described as using BM25 to load only relevant tool definitions, reducing context-window usage and improving tool-selection accuracy.

Relevance to AI PMs

• Improve tool and knowledge retrieval quality: BM25 is useful when users or agents search across tool specs, API docs, SOPs, or internal knowledge. It performs especially well when exact wording, product names, feature labels, or domain-specific terms matter.
• Reduce token costs and context overload: PMs designing agentic workflows can use BM25 to pre-filter large tool or document sets so only the most relevant candidates are passed into the model, lowering latency, cost, and prompt clutter.
• Build better hybrid search systems: When combined with embeddings, rerankers, or confidence models, BM25 provides a strong lexical signal that helps catch exact-match queries and improves robustness for enterprise and operational search use cases.

Related

• Doug Turnbull — Frequently associated with technical discussion of BM25 scoring, calibration, and practical search relevance techniques.
• Hybrid search — BM25 is a common lexical component in hybrid retrieval systems that combine keyword and semantic signals.
• Tool Search Tool — An example of BM25 applied in AI tooling to retrieve only relevant tool definitions before model invocation.
• Claude Design — Referenced in a workflow where BM25-powered tool retrieval helped support export-to-Canva functionality with less context-window overhead.