Sparse Datasets

Learn how to ingest and perform basic ML operations on the MovieLens 100k sparse dataset.

A sparse dataset is one in which a significant portion of the values are zero.

• In a sparse dataset, most elements in the data matrix or array contain zero values, resulting in a high level of sparsity.
• Sparse datasets are common in various domains, such as natural language processing (where many words may not appear in a document), recommendation systems (where users may interact with only a small subset of items), and certain scientific applications (where measurements are missing or irrelevant for many observations).
• Sparse datasets can be represented more efficiently using sparse matrix or sparse tensor formats, which store only the non-zero values along with their indices. This representation saves memory and computational resources compared to dense representations.

Sparse ingestion

For the sparse case, this tutorial will use the MovieLens 100k dataset.

Note

The MovieLens dataset is a well-known and widely used dataset in the field of recommendation systems and collaborative filtering research. It comprises various collections of movie ratings provided by users of the MovieLens website. These ratings are typically on a scale of 1 to 5, with 5 indicating the highest rating and 1 indicating the lowest. The MovieLens dataset comes in several versions, with the most commonly used ones being MovieLens 100k, MovieLens 1M, MovieLens 10M, MovieLens 20M, and MovieLens 25M, which denote the approximate number of ratings in each dataset. The sparsity of the MovieLens dataset depends on the specific version and how it is represented. In terms of ratings, the MovieLens dataset can be considered somewhat sparse, especially as the dataset size increases. This sparsity arises because not all users rate all movies, and the dataset typically contains many missing ratings.

PyTorch

Import libraries

Start by importing the libraries used in this tutorial.

import os
import urllib.request
import numpy as np
import tiledb
import pandas as pd


from tiledb.ml.readers.pytorch import PyTorchTileDBDataLoader
from tiledb.ml.readers.types import ArrayParams

Download the dataset

original_path = os.path.join(os.path.pardir, "data")
filename = os.path.join(original_path, "movielens-ml-100k-u.data")
url = "https://files.grouplens.org/datasets/movielens/ml-100k/u.data"
urllib.request.urlretrieve(url, filename)

Transform to sparse dataset

data = pd.read_csv(
    filename, sep="\t", usecols=[0, 1, 2], names=["user_id", "item_id", "rating"]
)
data_one_hot = pd.get_dummies(data, columns=["user_id", "item_id"])
user_movie = data_one_hot[data_one_hot.columns.difference(["rating"])]
ratings = data["rating"]

Ingest in TileDB

def get_schema(data: np.array, batch_size: int, sparse: bool) -> tiledb.ArraySchema:
    dims = [
        tiledb.Dim(
            name="dim_" + str(dim),
            domain=(0, data.shape[dim] - 1),
            tile=data.shape[dim] if dim > 0 else batch_size,
            dtype=np.int32,
        )
        for dim in range(data.ndim)
    ]
    # TileDB schema
    schema = tiledb.ArraySchema(
        domain=tiledb.Domain(*dims),
        sparse=sparse,
        attrs=[tiledb.Attr(name="features", dtype=np.float32)],
    )
    return schema


# Let's define an ingestion function
def ingest_in_tiledb(data: np.array, batch_size: int, uri: str, sparse: bool):
    schema = get_schema(data, batch_size, sparse)
    # Create the (empty) array on disk.
    tiledb.Array.create(uri, schema)
    # Ingest
    with tiledb.open(uri, "w") as tiledb_array:
        idx = np.nonzero(data) if sparse else slice(None)
        tiledb_array[idx] = {"features": data[idx]}

data_dir = os.path.join(original_path, "readers", "sparse")
os.makedirs(data_dir, exist_ok=True)

# Ingest images
training_images = os.path.join(data_dir, "training_images")
ingest_in_tiledb(
    data=user_movie.to_numpy(), batch_size=64, uri=training_images, sparse=True
)

# Ingest labels
training_labels = os.path.join(data_dir, "training_labels")
ingest_in_tiledb(
    data=ratings.to_numpy(), batch_size=64, uri=training_labels, sparse=False
)

TileDB dataset

user_movie_array = tiledb.open(training_images)
ratings_array = tiledb.open(training_labels)

Arrays schemas

user_movie_array.schema

Domain
Name Domain Tile Data Type Is Var-length Filters dim_0 (0, 99999) 64 int32 False - dim_1 (0, 2624) 2625 int32 False -
Attributes
Name Data Type Is Var-Len Is Nullable Filters features float32 False False -
Cell Order
row-major
Tile Order
row-major
Capacity
10000
Sparse
True
Allows DuplicatesK/th>
False

ratings_array.schema

Domain

Name Domain Tile Data Type Is Var-length Filters dim_0 (0, 99999) 64 int32 False -

Attributes

Name Data Type Is Var-Len Is Nullable Filters features float32 False False -

Cell Order

row-major

Tile Order

row-major

Sparse

False

TensorFlow

Import libraries

Start by importing the libraries used in this tutorial.

import os
import urllib.request
import tiledb
import numpy as np
import pandas as pd


from tiledb.ml.readers.tensorflow import TensorflowTileDBDataset, ArrayParams

Download the dataset

original_path = os.path.join(os.path.pardir, "data")
filename = os.path.join(original_path, "movielens-ml-100k-u.data")
url = "https://files.grouplens.org/datasets/movielens/ml-100k/u.data"
urllib.request.urlretrieve(url, filename)

Transform to sparse dataset

data = pd.read_csv(
    filename, sep="\t", usecols=[0, 1, 2], names=["user_id", "item_id", "rating"]
)
data_one_hot = pd.get_dummies(data, columns=["user_id", "item_id"])
user_movie = data_one_hot[data_one_hot.columns.difference(["rating"])]
ratings = data["rating"]

Ingest in TileDB

def get_schema(data: np.array, batch_size: int, sparse: bool) -> tiledb.ArraySchema:
    dims = [
        tiledb.Dim(
            name="dim_" + str(dim),
            domain=(0, data.shape[dim] - 1),
            tile=data.shape[dim] if dim > 0 else batch_size,
            dtype=np.int32,
        )
        for dim in range(data.ndim)
    ]
    # TileDB schema
    schema = tiledb.ArraySchema(
        domain=tiledb.Domain(*dims),
        sparse=sparse,
        attrs=[tiledb.Attr(name="features", dtype=np.float32)],
    )
    return schema


# Let's define an ingestion function
def ingest_in_tiledb(data: np.array, batch_size: int, uri: str, sparse: bool):
    schema = get_schema(data, batch_size, sparse)
    # Create the (empty) array on disk.
    tiledb.Array.create(uri, schema)
    # Ingest
    with tiledb.open(uri, "w") as tiledb_array:
        idx = np.nonzero(data) if sparse else slice(None)
        tiledb_array[idx] = {"features": data[idx]}

data_dir = os.path.join(original_path, "readers", "sparse")
os.makedirs(data_dir, exist_ok=True)

# Ingest images
training_images = os.path.join(data_dir, "training_images")
ingest_in_tiledb(
    data=user_movie.to_numpy(), batch_size=64, uri=training_images, sparse=True
)

# Ingest labels
training_labels = os.path.join(data_dir, "training_labels")
ingest_in_tiledb(
    data=ratings.to_numpy(), batch_size=64, uri=training_labels, sparse=False
)

TileDB dataset

user_movie_array = tiledb.open(training_images)
ratings_array = tiledb.open(training_labels)

Arrays schemas

user_movie_array.schema

Domain
Name Domain Tile Data Type Is Var-length Filters dim_0 (0, 99999) 64 int32 False - dim_1 (0, 2624) 2625 int32 False -
Attributes
Name Data Type Is Var-Len Is Nullable Filters features float32 False False -
Cell Order
row-major
Tile Order
row-major
Capacity
10000
Sparse
True
Allows DuplicatesK/th>
False

ratings_array.schema

Domain

Name Domain Tile Data Type Is Var-length Filters dim_0 (0, 99999) 64 int32 False -

Attributes

Name Data Type Is Var-Len Is Nullable Filters features float32 False False -

Cell Order

row-major

Tile Order

row-major

Sparse

False

Dataloaders

TileDB offers an API with native dataloaders for all the ML frameworks with which TileDB integrates. After you store your data, you can use the API to create dataloaders in each framework that will be later used as input to the model’s training stage. The API takes two TileDB arrays as inputs: x, which refers to the sample data; and y, which holds the label data corresponding to each sample in x. The dataloader collates these two arrays into a single data object that can later be used as input for training a model.

PyTorch

with tiledb.open(training_images) as x, tiledb.open(training_labels) as y:
    train_loader = PyTorchTileDBDataLoader(
        ArrayParams(x),
        ArrayParams(y),
        batch_size=128,
        num_workers=0,
        shuffle_buffer_size=256,
    )
    batch_imgs, batch_labels = next(iter(train_loader))
    print(f"Input Shape: {batch_imgs.shape}")
    print(f"Label Shape: {batch_labels.shape}")

Input Shape: torch.Size([128, 2625])
Label Shape: torch.Size([128])

TensorFlow

with tiledb.open(training_images) as x, tiledb.open(training_labels) as y:
    tiledb_dataset = TensorflowTileDBDataset(
        ArrayParams(array=x),
        ArrayParams(array=y),
    )
    batched_dataset = tiledb_dataset.batch(128)
    batch_imgs, batch_labels = next(batched_dataset.as_numpy_iterator())
    print(f"Input Shape: {batch_imgs.shape}")
    print(f"Label Shape: {batch_labels.shape}")

Input Shape: (128, 2625)
Label Shape: (128,)