Skip to content


BM25 Transformer#

BM25 is a sublinear term weighting scheme that takes term frequency (TF), document frequency (DF), and document length into account. It is similar to TF-IDF but with variable sublinearity and the addition of document length normalization.

Note: BM25 Transformer assumes that its inputs are token frequency vectors such as those created by Word Count Vectorizer.

Interfaces: Transformer, Stateful, Elastic

Data Type Compatibility: Continuous only


# Param Default Type Description
1 dampening 1.2 float The term frequency (TF) dampening factor i.e. the K1 parameter in the formula. Lower values will cause the TF to saturate quicker.
2 normalization 0.75 float The importance of document length in normalizing the term frequency i.e. the b parameter in the formula.


use Rubix\ML\Transformers\BM25Transformer;

$transformer = new BM25Transformer(1.2, 0.75);

Additional Methods#

Return the document frequencies calculated during fitting:

public dfs() : ?array

Return the average number of tokens per document:

public averageDocumentLength() : ?float


  • S. Robertson et al. (2009). The Probabilistic Relevance Framework: BM25 and Beyond.
  • K. Sparck Jones et al. (2000). A probabilistic model of information retrieval: development and comparative experiments.