Whitespace#
Tokens are delimited by a user-specified whitespace character.
Parameters#
| # | Name | Default | Type | Description |
|---|---|---|---|---|
| 1 | delimiter | ' ' | string | The whitespace character that delimits each token. |
Example#
use Rubix\ML\Tokenizers\Whitespace;
$tokenizer = new Whitespace(',');