Dataset Objects#

Data are passed in specialized in-memory containers called Dataset objects. Dataset objects are table-like data structures that have operations for data manipulation. They can hold a heterogeneous mix of data types and they make it easy to transport data in a canonical way. Datasets consist of a matrix of samples in which each row constitutes a sample and each column represents the value of the feature represented by that column. They have the additional constraint that each feature column must contain values of the same high-level data type. Some datasets can contain labels for training or cross validation. In the example below, we instantiate a new Labeled dataset object by passing the samples and their labels as arguments to the constructor.

use Rubix\ML\Datasets\Labeled;

$samples = [
    [0.1, 20, 'furry'],
    [2.0, -5, 'rough'],
];

$labels = ['not monster', 'monster'];

$dataset = new Labeled($samples, $labels);

Missing Values#

By convention, missing continuous values are denoted by the NAN constant and missing categorical values are denoted by a special placeholder category (ex. the ? category). Dataset objects do not allow missing values of resource or other data types.

$samples = [
    [0.01, -500, 'furry'], // Complete sample
    [0.001, NAN, 'rough'], // Missing a continuous value
    [0.25, -1000, '?'], // Missing a categorical value
];

Factory Methods#

Build a dataset with the rows from a 2-dimensional iterable data table:

public static fromIterator(Traversable $iterator) : self

Note: When building a Labeled dataset, the label values should be in the last column of the data table.

Example

use Rubix\ML\Datasets\Labeled;
use Rubix\ML\Datasets\Extractors\CSV;

$dataset = Labeled::fromIterator(new CSV('example.csv'));

Selecting#

Return all the samples in the dataset in a 2-dimensional array:

public samples() : array

Select a single row containing the sample at a given offset (offsets begin at 0):

public sample(int $offset) : array

Select the values of a feature column at a given offset (offsets begin at 0):

public column(int $offset) : array

Return the columns of the sample matrix:

public columns() : array

Return the columns of the sample matrix of a particular type:

public columnsByType(DataType $type) : array

Example

use Rubix\ML\DataType;

$columns = $dataset->columnsByType(DataType::continuous());

Properties#

Return the number of rows in the dataset:

public numRows() : int

Return the number of columns in the samples matrix:

public numColumns() : int

Example

$m = $dataset->numRows();

$n = $dataset->numColumns();

Return a 2-tuple with the shape of the samples matrix:

public shape() : array

Example

[$m, $n] = $dataset->shape();

var_dump($m, $n);

int(1000)
int(30)

Return the data types for each feature column:

public columnTypes() : array

Return the data type for a given column offset:

public columnType(int $offset) : DataType

Example

echo $dataset->columnType(15);

categorical

Applying Transformations#

You can apply a Transformer directly to the samples in a Dataset object by passing it as an arguent to the apply() method on the dataset object.

public apply(Transformer $transformer) : self

Example

use Rubix\ML\Transformers\OneHotEncoder;

$dataset->apply(new OneHotEncoder());

You can also transform a single feature column using a callback function with the transformColumn() method.

public transformColumn(int $column, callable $callback) : self

Example

$dataset->transformColumn(0, 'log1p');

$dataset->transformColumn(5, function ($value) {
    return $value === 0 ? NAN : $value;
});

$dataset->transformColumn(6, function ($value) {
    return min($value, 1000);
});

Stacking Datasets#

Stack any number of dataset objects on top of each other to form a single dataset:

public static stack(array $datasets) : self

Note: Datasets must have the same number of feature columns i.e. dimensionality.

Example

use Rubix\ML\Datasets\Labeled;

$dataset = Labeled::stack([
    $dataset1,
    $dataset2,
    $dataset3,
    // ...
]);

Merging Datasets#

To merge the rows of this dataset with another dataset:

public merge(Dataset $dataset) : self

Note: Datasets must have the same number of columns.

Example

$dataset = $dataset1->merge($dataset2);

To merge the columns of this dataset with another dataset:

public augment(Dataset $dataset) : self

Note: Datasets must have the same number of rows.

Example

$dataset = $dataset1->augment($dataset2);

Head and Tail#

Return the first n rows of data in a new dataset object:

public head(int $n = 10) : self

Return the last n rows of data in a new dataset object:

public tail(int $n = 10) : self

Example

$subset = $dataset->head(10);

$subset = $dataset->tail(30);

Taking and Leaving#

Remove n rows from the dataset and return them in a new dataset:

public take(int $n = 1) : self

Leave n samples on the dataset and return the rest in a new dataset:

public leave(int $n = 1) : self

Splitting#

Split the dataset into left and right subsets:

public split(float $ratio = 0.5) : array

Example

[$training, $testing] = $dataset->split(0.8);

Folding#

Fold the dataset to form k equal size datasets:

public fold(int $k = 10) : array

Note: If there are not enough samples to completely fill the last fold of the dataset then it will contain slightly fewer samples than the rest of the folds.

Example

$folds = $dataset->fold(8);

Slicing and Splicing#

Return an n size portion of the dataset in a new dataset:

public slice(int $offset, int $n) : self

Remove a size n chunk of the dataset starting at offset and return it in a new dataset:

public splice(int $offset, int $n) : self

Batching#

Batch the dataset into subsets containing a maximum of n rows per batch:

public batch(int $n = 50) : array

Example

$batches = $dataset->batch(250);

Randomization#

Randomize the order of the dataset and return it for method chaining:

public randomize() : self

Example

$dataset->randomize();

Generate a random subset of the dataset without replacement of size n:

public randomSubset(int $n) : self

Example

$subset = $dataset->randomSubset(50);

Generate a random subset with replacement:

public randomSubsetWithReplacement($n) : self

Example

$subset = $dataset->randomSubsetWithReplacement(500);

Generate a random weighted subset with replacement of size n:

public randomWeightedSubsetWithReplacement($n, array $weights) : self

Example

$subset = $dataset->randomWeightedSubsetWithReplacement(200, $weights);

Filtering#

Filter the rows of the dataset using the values of a feature column at the given offset as the arguments to a filter callback. The callback should return false for rows that should be filtered.

public filterByColumn(int $offset, callable $fn) : self

Example

$tallPeople = $dataset->filterByColumn(3, function ($value) {
    return $value > 178.5;
});

Sorting#

To sort a dataset in place by a specific feature column:

public sortByColumn(int $offset, bool $descending = false) : self

Example

$dataset->sortByColumn(5);

Dropping Rows and Columns#

Drop the row at the given offset:

public dropRow(int $offset) : self

Drop the rows at the given offsets:

public dropRows(array $indices) : self

Drop the column at the given offset:

public dropColumn(int $offset) : self

Drop the columns at the given indices:

public dropColumns(array $indices) : self

Descriptive Statistics#

Return an array of statistics such as the central tendency, dispersion and shape of each continuous feature column and the joint probabilities of each category for every categorical feature column:

public describe() : array

Example

print_r($dataset->describe());

Array
(
    [2] => Array
        (
            [type] => categorical
            [num_categories] => 2
            [probabilities] => Array
                (
                    [friendly] => 0.66666666666667
                    [loner] => 0.33333333333333
                )

        )

    [3] => Array
        (
            [type] => continuous
            [mean] => 0.33333333333333
            [variance] => 9.7922222222222
            [std_dev] => 3.1292526619342
            [skewness] => -0.44810308436906
            [kurtosis] => -1.1330702741786
            [min] => -5
            [25%] => -1.375
            [median] => 0.8
            [75%] => 2.825
            [max] => 4
        )
)

De-duplication#

Remove duplicate rows from the dataset:

public deduplicate() : self

Output Formats#

Return the dataset object as a data table array:

public toArray() : array

Example

$table = $dataset->toArray();

Return a JSON representation of the dataset:

public toJSON(bool $pretty = false) : string

Return a newline delimited JSON representation of the dataset:

public toNDJSON() : string

Example

file_put_contents('dataset.ndjson', $dataset->toNDJSON());

Return the dataset as comma-separated values (CSV) string:

public toCSV(string $delimiter = ',', string $enclosure = '"') : string

Example

file_put_contents('dataset.csv', $dataset->toCSV());

Previewing in the Console#

You can echo the dataset object to preview the first few rows and columns in the console.

echo $dataset;

| Column 0    | Column 1    | Column 2    | Column 3    | Label       |
-----------------------------------------------------------------------
| nice        | furry       | friendly    | 4           | not monster |
-----------------------------------------------------------------------
| mean        | furry       | loner       | -1.5        | monster     |
-----------------------------------------------------------------------
| nice        | rough       | friendly    | 2.6         | not monster |
-----------------------------------------------------------------------
| mean        | rough       | friendly    | -1          | monster     |
-----------------------------------------------------------------------
| nice        | rough       | friendly    | 2.9         | not monster |
-----------------------------------------------------------------------
| nice        | furry       | loner       | -5          | not monster |
-----------------------------------------------------------------------