Nullable Attributes

Learn how to work with arrays that contain nullable attributes.

Two main types of nullable attributes exist in TileDB:

Fixed-length, nullable attributes
Variable-length, nullable attributes

Fixed-length, nullable attributes

Both the Python and R APIs support fixed-length, nullable attributes. The following array you’ll use is a sparse array, but these concepts also apply to dense arrays.

To get started, import the necessary libraries, set the array URI (that is, its path, which in this tutorial will be on local storage), and delete any previously created arrays with the same name.

Python
R

# Import necessary libraries
import os.path
import shutil

import numpy as np
import tiledb

# Set array URI
array_uri = os.path.expanduser("~/fixed_length_nullable_python")

# Delete array if it already exists
if os.path.exists(array_uri):
    shutil.rmtree(array_uri)

# Import necessary libraries
library(tiledb)

# Set array URI
sparse_array <- path.expand("~/fixed_length_nullable_r")

# Delete array if it already exists
if (file.exists(sparse_array)) {
  unlink(sparse_array, recursive = TRUE)
}

Next, create the array by specifying its schema.

Python
R

# Create the two dimensions
d1 = tiledb.Dim(name="d1", domain=(1, 4), tile=2, dtype=np.int32)
d2 = tiledb.Dim(name="d2", domain=(1, 4), tile=2, dtype=np.int32)

# Create a domain using the two dimensions
dom = tiledb.Domain(d1, d2)
# Order of the dimensions matters when slicing subarrays.
# Remember to give priority to more selective dimensions to
# maximize the pruning power during slicing.

# Create an attribute
a = tiledb.Attr(name="a", dtype=np.float64, nullable=True)

# Create the array schema with `sparse=True`.
# Set `cell_order` to 'row-major' (default) or 'C', 'col-major' or 'F', or 'hilbert'.
# Set `tile_order` to 'row-major' (default) or 'C', 'col-major' or 'F'.
sch = tiledb.ArraySchema(domain=dom, sparse=True, attrs=[a])

# Create the array on disk (it will initially be empty)
tiledb.Array.create(array_uri, sch)

# Create the two dimensions
d1 <- tiledb_dim("d1", c(1L, 4L), 2L, "INT32")
d2 <- tiledb_dim("d2", c(1L, 4L), 2L, "INT32")

# Create a domain using the two dimensions
dom <- tiledb_domain(dims = c(d1, d2))

# Create an attribute
a <- tiledb_attr("a", type = "FLOAT64", nullable = TRUE)

# Create the array schema with `sparse = TRUE`
sch <- tiledb_array_schema(dom, a, sparse = TRUE)

# Create the array on disk (it will initially be empty)
arr <- tiledb_array_create(sparse_array, sch)

Populate the TileDB array with a set of 1D input arrays: one for the coordinates of each dimension, and one for the attribute values. TileDB sparse arrays expect the coordinate (COO) format.

Python
R

# Prepare some data in numpy arrays
d1_data = np.array([1, 2, 3, 4], dtype=np.int32)
d2_data = np.array([2, 1, 3, 4], dtype=np.int32)
a_data = np.array(
    [1.1, 2.2, None, 4.4],
    dtype="O",
)

# Open the array in write mode and write the data in COO format
with tiledb.open(array_uri, "w") as A:
    A[d1_data, d2_data] = {"a": a_data}

# Prepare some data in an array
d1_data <- c(1, 2, 3, 4)
d2_data <- c(2, 1, 3, 4)
a_data <- c(1.1, 2.2, NA, 4.4)

# Open the array for writing and write data to the array
arr <- tiledb_array(
  uri = sparse_array,
  query_type = "WRITE",
  return_as = "data.frame"
)
arr[d1_data, d2_data] <- a_data

# Close the array
invisible(tiledb_array_close(arr))

The array is a folder in the path specified in array_uri. You can learn about the different contents of the array folder in other sections of the Academy.

Python
R

/Users/nickv/basic_sparse
├── __commits
│   └── __1739368327679_1739368327679_51fb1c9a2caa9db13e987a223f066c77_22.wrt
├── __fragment_meta
├── __fragments
│   └── __1739368327679_1739368327679_51fb1c9a2caa9db13e987a223f066c77_22
│       ├── __fragment_metadata.tdb
│       ├── a0.tdb
│       ├── a0_validity.tdb
│       ├── d0.tdb
│       └── d1.tdb
├── __labels
├── __meta
└── __schema
    ├── __1739368327666_1739368327666_000000028e0fbc74cc1ea0cf0e43ae11
    └── __enumerations

9 directories, 7 files

/Users/nickv/basic_sparse
├── __commits
│   └── __1739368327679_1739368327679_51fb1c9a2caa9db13e987a223f066c77_22.wrt
├── __fragment_meta
├── __fragments
│   └── __1739368327679_1739368327679_51fb1c9a2caa9db13e987a223f066c77_22
│       ├── __fragment_metadata.tdb
│       ├── a0.tdb
│       ├── a0_validity.tdb
│       ├── d0.tdb
│       └── d1.tdb
├── __labels
├── __meta
└── __schema
    ├── __1739368327666_1739368327666_000000028e0fbc74cc1ea0cf0e43ae11
    └── __enumerations

9 directories, 7 files

Read the data by using the slicing methods supported in TileDB.

Python
R

# Open the array in read mode
A = tiledb.open(array_uri, "r")

# Show the entire array
print("Entire array: ")
print(A[:])
print("\n")

print("Entire array as a data frame:")
print(A.df[:])
print("\n")

# Remember to close the array
A.close()

Entire array: 
OrderedDict({'a': masked_array(data=[1.1, 2.2, --, 4.4],
             mask=[False, False,  True, False],
       fill_value=1e+20), 'd1': array([1, 2, 3, 4], dtype=int32), 'd2': array([2, 1, 3, 4], dtype=int32)})


Entire array as a data frame:
   d1  d2    a
0   1   2  1.1
1   2   1  2.2
2   3   3  NaN
3   4   4  4.4

# Open the array in read mode
invisible(tiledb_array_open(arr, type = "READ"))

# Show the entire array
cat("Entire array:\n")
print(arr[])

# Close the array
invisible(tiledb_array_close(arr))

Entire array:
  d1 d2   a
1  2  1 2.2
2  1  2 1.1
3  3  3  NA
4  4  4 4.4

Variable-length, nullable attributes

As highlighted in Variable-Length Attributes, variable-length attributes exist in two forms:

Attributes that accept variable-length lists of basic datatypes.
Attributes that accept variable-length string values.

The same concept applies to variable-length, nullable attributes.

Nullable, variable-length list attributes

Writing variable-length attribute values to an array involves passing three buffers to TileDB: one for the variable-length cell values, one for the starting offset of each value in the first buffer, and one for the cell validity values. The following code block illustrates this with a sparse write, but it is also applicable for dense writes.

Python
R

# TODO

## Variable-length nullable (numerical) attributes are not yet supported

The code snippet above produces the following sparse fragment:

Nullable, variable-length string attributes

First, import the necessary libraries, set the array URI (that is, its path, which in this tutorial will be on local storage), and delete any previously created arrays with the same name.

Python
R

# Import necessary libraries
import os.path
import shutil

import numpy as np
import tiledb

# Set array URI
array_uri = os.path.expanduser("~/var_length_null_string_py")

# Delete array if it already exists
if os.path.exists(array_uri):
    shutil.rmtree(array_uri)

library(tiledb)

# Set array URI
array_uri <- path.expand("~/var_length_attributes_string_r")

# Delete array if it already exists
if (file.exists(array_uri)) {
  unlink(array_uri, recursive = TRUE)
}

Next, create a 2D sparse array by specifying its schema (this applies to dense arrays as well). Notice how to specify a variable-length attribute that accepts variable-length strings.

Python
R

# Create the two dimensions
d1 = tiledb.Dim(name="d1", domain=(0, 3), tile=2, dtype=np.int32)
d2 = tiledb.Dim(name="d2", domain=(0, 3), tile=2, dtype=np.int32)

# Create a domain using the two dimensions
dom = tiledb.Domain(d1, d2)

# Create a string attribute by setting dtype=np.bytes_.
# This attribute will accept variable-length strings.
a = tiledb.Attr(name="a", dtype=np.bytes_, nullable=True)

# Create the array schema with `sparse=True`
sch = tiledb.ArraySchema(domain=dom, sparse=True, attrs=[a])

# Create the array on disk (it will initially be empty)
tiledb.Array.create(array_uri, sch)

# Create the two dimensions
d1_str <- tiledb_dim("d1", c(0L, 3L), 2L, "INT32")
d2_str <- tiledb_dim("d2", c(0L, 3L), 2L, "INT32")

# Create a domain using the two dimensions
dom <- tiledb_domain(dims = c(d1_str, d2_str))

# Create string attribute a
a <- tiledb_attr("a", type = "ASCII", ncells = NA, nullable = TRUE)

# Create the array schema, setting `sparse = TRUE`
sch <- tiledb_array_schema(dom, a, sparse = TRUE)

# Create the array on disk (it will initially be empty)
arr <- tiledb_array_create(array_uri, sch)

Populate the array with 1D NumPy arrays in COO format.

Python
R

import numpy as np
import tiledb

# Set the coordinates
d1_data = np.array([1, 2, 3, 3, 0])
d2_data = np.array([2, 1, 3, 2, 0])

# Set the string attribute values
a_data = np.array(["aa", "", "Ccc", "d", None], dtype="O")

# Write the data to the array
with tiledb.open(array_uri, "w") as A:
    A[d1_data, d2_data] = a_data

# Set the coordinates
d1_data <- c(1L, 2L, 3L, 3L, 0L)
d2_data <- c(2L, 1L, 3L, 2L, 0L)

# Set the string attribute values
a_data <- c("aa", "", "Ccc", "d", NA)

# Write the data to the array
arr <- tiledb_array(uri = array_uri, query_type = "WRITE", return_as = "data.frame")
arr[d1_data, d2_data] <- a_data

Read the entire array and observe the returned variable-length strings.

Python
R

# Variable-length arrays may be sliced as usual in Python.
# The API handles unpacking and type conversion, and returns
# a NumPy object array-of-arrays.

# Read all array data
with tiledb.open(array_uri) as A:
    print(A[:]["a"])
    print(A.df[:])

[-- b'aa' b'' b'd' b'Ccc']
   d1  d2       a
0   0   0    None
1   1   2   b'aa'
2   2   1     b''
3   3   2    b'd'
4   3   3  b'Ccc'

# Variable-length arrays may be sliced as usual in R
# The API handles unpacking and type conversion, and returns
# an array of arrays.

# Read all array data
print(arr[])

  d1 d2    a
1  0  0 <NA>
2  2  1     
3  1  2   aa
4  3  2    d
5  3  3  Ccc

Clean up in the end by deleting the array.

Python
R

# Delete the array
if os.path.exists(array_uri):
    shutil.rmtree(array_uri)

if (dir.exists(array_uri)) {
  unlink(array_uri, recursive = TRUE)
}