Whitespace#
Tokens are delimited by a user-specified whitespace character.
Parameters#
# | Name | Default | Type | Description |
---|---|---|---|---|
1 | delimiter | ' ' | string | The whitespace character that delimits each token. |
Example#
use Rubix\ML\Tokenizers\Whitespace;
$tokenizer = new Whitespace(',');
Last update:
2021-03-27