Basic Sparse Array

Learn about basic functionality of sparse arrays within TileDB, from ingestion to querying and beyond.

How to run this tutorial

You can run this tutorial in two ways:

1. Locally on your machine.
2. On TileDB Cloud.

However, since TileDB Cloud has a free tier, we strongly recommend that you sign up and run everything there, as that requires no installations or deployment.

This tutorial explains the most basic functionality of sparse arrays, by creating a small 2×2 sparse array and reading it.

First, import the necessary libraries, set the array URI (that is, its path, which in this tutorial will be on local storage), and delete any previously created arrays with the same name.

Python

# Import necessary libraries
import os.path
import shutil

import numpy as np
import tiledb

# Set array URI
array_uri = os.path.expanduser("~/basic_sparse")

# Delete array if it already exists
if os.path.exists(array_uri):
    shutil.rmtree(array_uri)

R

# Import necessary libraries
library(tiledb)

# Set array URI
sparse_array <- path.expand("~/sparse_array_r")

# Delete array if it already exists
if (file.exists(sparse_array)) {
  unlink(sparse_array, recursive = TRUE)
}

Next, create the array by specifying its schema.

Python

# Create the two dimensions
d1 = tiledb.Dim(name="d1", domain=(0, 3), tile=2, dtype=np.int32)
d2 = tiledb.Dim(name="d2", domain=(0, 3), tile=2, dtype=np.int32)

# Create a domain using the two dimensions
dom = tiledb.Domain(d1, d2)
# Order of the dimensions matters when slicing subarrays.
# Remember to give priority to more selective dimensions to
# maximize the pruning power during slicing.

# Create an attribute
a = tiledb.Attr(name="a", dtype=np.int32)

# Create the array schema with `sparse=True`.
# Set `cell_order` to 'row-major' (default) or 'C', 'col-major' or 'F', or 'hilbert'.
# Set `tile_order` to 'row-major' (default) or 'C', 'col-major' or 'F'.
sch = tiledb.ArraySchema(domain=dom, sparse=True, attrs=[a])

# Create the array on disk (it will initially be empty)
tiledb.Array.create(array_uri, sch)

R

# Create the two dimensions
d1 <- tiledb_dim("d1", c(0L, 3L), 2L, "INT32")
d2 <- tiledb_dim("d2", c(0L, 3L), 2L, "INT32")

# Create a domain using the two dimensions
dom <- tiledb_domain(dims = c(d1, d2))

# Create an attribute
a <- tiledb_attr("a", type = "INT32")

# Create the array schema with `sparse = TRUE`
sch <- tiledb_array_schema(dom, a, sparse = TRUE)

# Create the array on disk (it will initially be empty)
arr <- tiledb_array_create(sparse_array, sch)

Note that you can specify tiling for each dimension, whereas you can set the cell and tile orders in the array schema. Tiling, tile order, and cell order collectively define the data layout on storage, which plays an important role in performance. For more information, visit the following sections:

• Key Concepts: Tiles
• Key Concepts: Data Layout

Create a set of 1D input arrays: one for the coordinates of each dimension, and one for the attribute values. TileDB sparse arrays expect the coordinate (COO) format.

Python

# Prepare some data in numpy arrays
d1_data = np.array([2, 0, 3, 2, 0, 1], dtype=np.int32)
d2_data = np.array([0, 1, 1, 2, 3, 3], dtype=np.int32)
a_data = np.array([4, 1, 6, 5, 2, 3], dtype=np.int32)

R

d1_data <- c(2L, 0L, 3L, 2L, 0L, 1L)
d2_data <- c(0L, 1L, 1L, 2L, 3L, 3L)
a_data <- c(4L, 1L, 6L, 5L, 2L, 3L)

You can optionally visualize what the array will look like before you write data to the array:

Python

# Only for visualization purposes
from scipy.sparse import coo_matrix

coo_matrix((a_data, (d1_data, d2_data)), shape=(4, 4)).toarray()

R

# Only for visualization purposes
library(Matrix)

# Create the sparse matrix in COO format
sparse_matrix <- sparseMatrix(i = d1_data + 1, j = d2_data + 1, x = a_data, dims = c(4, 4))

# Convert to a dense matrix for visualization
matrix_array <- as.matrix(sparse_matrix)

# Print the matrix
print(matrix_array)

Now write the data to the array.

Python

# Open the array in write mode and write the data in COO format
with tiledb.open(array_uri, "w") as A:
    A[d1_data, d2_data] = a_data

R

# Prepare some data in an array

# Open the array for writing and write data to the array
arr <- tiledb_array(
  uri = sparse_array,
  query_type = "WRITE",
  return_as = "data.frame"
)
arr[d1_data, d2_data] <- a_data

# Close the array
invisible(tiledb_array_close(arr))

The array is a folder in the path specified in array_uri. You can learn about the different contents of the array folder in other sections of the Academy.

Python

/Users/stavrospapadopoulos/basic_sparse
├── __commits
│   └── __1715724313475_1715724313475_51a706c1260678960baa4aae2f98d83c_21.wrt
├── __fragment_meta
├── __fragments
│   └── __1715724313475_1715724313475_51a706c1260678960baa4aae2f98d83c_21
│       ├── __fragment_metadata.tdb
│       ├── a0.tdb
│       ├── d0.tdb
│       └── d1.tdb
├── __labels
├── __meta
└── __schema
    ├── __1715724313471_1715724313471_00000002d79b6955cd3795ecfd9d8ec9
    └── __enumerations

9 directories, 6 files

R

/Users/stavrospapadopoulos/basic_sparse
├── __commits
│   └── __1715724313475_1715724313475_51a706c1260678960baa4aae2f98d83c_21.wrt
├── __fragment_meta
├── __fragments
│   └── __1715724313475_1715724313475_51a706c1260678960baa4aae2f98d83c_21
│       ├── __fragment_metadata.tdb
│       ├── a0.tdb
│       ├── d0.tdb
│       └── d1.tdb
├── __labels
├── __meta
└── __schema
    ├── __1715724313471_1715724313471_00000002d79b6955cd3795ecfd9d8ec9
    └── __enumerations

9 directories, 6 files

Read the data by using the slicing methods supported in TileDB.

Python

# Open the array in read mode
A = tiledb.open(array_uri, "r")

# Return the non-empty domain of the array
print("Non-empty domain: ")
print(A.nonempty_domain())
print("\n")

# Show the entire array
print("Entire array: ")
print(A[:])
print("\n")

# Show the 'a' attribute of the array
print("Attribute 'a': ")
print(A[:]["a"])
print("\n")

# Slice a portion of the array, which is useful
# when the arrays are too big to fit in main memory
print("Slice [0:2), [0:3): ")
print(A[0:2, 0:3])
print("\n")

# Slice a multi-range subarray
# note multi_index uses closed ranges
print("Multi-range, rows 0,1 and 3, and columns 0-2: ")
print(A.multi_index[[slice(0, 1), 3], slice(0, 2)])

# Remember to close the array
A.close()

Non-empty domain: 
((0, 3), (0, 3))


Entire array: 
OrderedDict({'a': array([1, 2, 3, 4, 6, 5], dtype=int32), 'd1': array([0, 0, 1, 2, 3, 2], dtype=int32), 'd2': array([1, 3, 3, 0, 1, 2], dtype=int32)})


Attribute 'a': 
[1 2 3 4 6 5]


Slice [0:2), [0:3): 
OrderedDict({'a': array([1], dtype=int32), 'd1': array([0], dtype=int32), 'd2': array([1], dtype=int32)})


Multi-range, rows 0,1 and 3, and columns 0-2: 
OrderedDict({'d1': array([0, 3], dtype=int32), 'd2': array([1, 1], dtype=int32), 'a': array([1, 6], dtype=int32)})

R

# Open the array in read mode
invisible(tiledb_array_open(arr, type = "READ"))

# Show the entire array
cat("Entire array:\n")
print(arr[])

# Show the 'a' attribute of the array
cat("Attribute 'a':\n")
print(arr[]["a"])

# Slice a portion of the array, which is useful
# when the arrays are too big to fit in main memory
cat("Slice [0:2], [0:3]:\n")
print(arr[0:2, 0:3]["a"])

# Close the array
invisible(tiledb_array_close(arr))

You can also do multi-range slicing in R:

# Define selected ranges
selected_ranges <- list(
  d1 = matrix(c(
    0, 1,
    3, 3
  ), 2, 2, byrow = TRUE),
  d2 = cbind(0, 2)
)

# Open the array using the selected ranges
arr <- tiledb_array(
  sparse_array,
  selected_ranges = selected_ranges,
  return_as = "data.frame"
)
print(arr[])

Clean up in the end by deleting the array.

Python

# Delete the array
if os.path.exists(array_uri):
    shutil.rmtree(array_uri)

R

if (file.exists(sparse_array)) {
  unlink(sparse_array, recursive = TRUE)
}