Skip to content


Token Hashing Vectorizer#

Token Hashing Vectorizer builds token count vectors on the fly by employing a hashing trick. It is a stateless transformer that uses a hashing algorithm to assign token occurrences to a bucket in a vector of user-specified dimensionality. The advantage of hashing over storing a fixed vocabulary is that there is no memory footprint however there is a chance that certain tokens will collide with other tokens especially in lower-dimensional vector spaces.


The default hashing function is CRC32 and is a good mix between speed and output space utilization. MurmurHash has even greater utilization but at the cost of some speed and it is only available on PHP 8.1 and above. FNV1 is comparable to CRC32 but with slightly more overhead.

Interfaces: Transformer

Data Type Compatibility: Categorical


# Param Default Type Description
1 dimensions int The dimensionality of the vector space.
2 tokenizer Word Tokenizer The tokenizer used to extract tokens from blobs of text.
3 hashFn callable 'crc32' The hash function that accepts a string token and returns an integer.


use Rubix\ML\Transformers\TokenHashingVectorizer;
use Rubix\ML\Tokenizers\Word();

$transformer = new TokenHashingVectorizer(10000, new Word(), TokenHashingVectorizer::MURMUR3);

Additional Constants#

The CRC32 callback function.

public const CRC32 callable(string):int

The MurmurHash3 callback function.

public const MURMUR3 callable(string):int

The FNV1 callback function.

public const FNV1 callable(string):int

Additional Methods#

The MurmurHash3 hashing function:

public static murmur3(string $input) : int


MurmurHash3 is only available on PHP 8.1 or above.

The FNV1a 32-bit hashing function:

public static fnv1a32(string $input) : int