TileDB supports attributes of variable length, including lists of basic datatypes and string attributes.
How to run this tutorial
You can run this tutorial in two ways:
Locally on your machine.
On TileDB Cloud.
However, since TileDB Cloud has a free tier, we strongly recommend that you sign up and run everything there, as that requires no installations or deployment.
This tutorial explains how to create arrays with variable-length attributes. It shows two similar cases:
Attributes that accept variable-length lists of basic datatypes.
Attributes that accept variable-length string values.
First, import the necessary libraries, set the array URI (that is, its path, which in this tutorial will be on local storage), and delete any previously created arrays with the same name.
# Import necessary librariesimport os.pathimport shutilimport numpy as npimport tiledb# Set array URIarray_uri = os.path.expanduser("~/var_length_attributes_list")# Delete array if it already existsif os.path.exists(array_uri): shutil.rmtree(array_uri)
# Import necessary librarieslibrary(tiledb)# Set array URIarray_uri <-path.expand("~/var_length_attributes_list_r")# Delete array if it already existsif (file.exists(array_uri)) {unlink(array_uri, recursive =TRUE)}
Next, create a 2D sparse array by specifying its schema (this applies to dense arrays as well). Notice how to specify a integer attribute that accepts variable-length lists of integers.
# Create the two dimensionsd1 = tiledb.Dim(name="d1", domain=(0, 3), tile=2, dtype=np.int32)d2 = tiledb.Dim(name="d2", domain=(0, 3), tile=2, dtype=np.int32)# Create a domain using the two dimensionsdom = tiledb.Domain(d1, d2)# Create a variable-length attribute by setting var=True.# This attribute will accept variable-length lists of integers.a = tiledb.Attr(name="a", dtype=np.int32, var=True)# Create the array schema with `sparse=True`sch = tiledb.ArraySchema(domain=dom, sparse=True, attrs=[a])# Create the array on disk (it will initially be empty)tiledb.Array.create(array_uri, sch)
# Create the two dimensionsd1 <-tiledb_dim("d1", c(1L, 4L), 2L, "INT32")d2 <-tiledb_dim("d2", c(1L, 4L), 2L, "INT32")# Create a domain using the two dimensionsdom <-tiledb_domain(dims =c(d1, d2))# Create an attributea <-tiledb_attr("a", type ="INT32")tiledb:::libtiledb_attribute_set_cell_val_num(a@ptr, NA)# Create the array schema with `sparse = TRUE`sch <-tiledb_array_schema(dom, a, sparse =TRUE)# Create the array on disk (it will initially be empty)arr <-tiledb_array_create(array_uri, sch)
Populate the TileDB array with 1D input arrays in coordinate (COO) format.
# Variable-length arrays may be sliced as usual in Python.# The API handles unpacking and type conversion, and returns# a NumPy object array-of-arrays.# Read all array datawith tiledb.open(array_uri) as A:print(A[:]["a"])
# Delete the arrayif os.path.exists(array_uri): shutil.rmtree(array_uri)
if (file.exists(array_uri)) {unlink(array_uri, recursive =TRUE)}
Variable-length strings
The case of variable-length string attributes is similar.
First, import the necessary libraries, set the array URI (that is, its path, which in this tutorial will be on local storage), and delete any previously created arrays with the same name.
# Import necessary librariesimport os.pathimport shutilimport numpy as npimport tiledb# Set array URIarray_uri = os.path.expanduser("~/var_length_attributes_string")# Delete array if it already existsif os.path.exists(array_uri): shutil.rmtree(array_uri)
library(tiledb)# Set array URIarray_uri <-path.expand("~/var_length_attributes_string_r")# Delete array if it already existsif (file.exists(array_uri)) {unlink(array_uri, recursive =TRUE)}
Next, create a 2D sparse array by specifying its schema (this applies to dense arrays as well). Notice how to specify a variable-length attribute that accepts variable-length strings.
# Create the two dimensionsd1 = tiledb.Dim(name="d1", domain=(0, 3), tile=2, dtype=np.int32)d2 = tiledb.Dim(name="d2", domain=(0, 3), tile=2, dtype=np.int32)# Create a domain using the two dimensionsdom = tiledb.Domain(d1, d2)# Create a string attribute by setting dtype=np.bytes_.# This attribute will accept variable-length strings.a = tiledb.Attr(name="a", dtype=np.bytes_)# Create the array schema with `sparse=True`sch = tiledb.ArraySchema(domain=dom, sparse=True, attrs=[a])# Create the array on disk (it will initially be empty)tiledb.Array.create(array_uri, sch)
# Create the two dimensionsd1_str <-tiledb_dim("d1", c(0L, 3L), 2L, "INT32")d2_str <-tiledb_dim("d2", c(0L, 3L), 2L, "INT32")# Create a domain using the two dimensionsdom <-tiledb_domain(dims =c(d1_str, d2_str))# Create string attribute aa <-tiledb_attr("a", type ="ASCII", ncells =NA)# Create the array schema, setting `sparse = TRUE`sch <-tiledb_array_schema(dom, a, sparse =TRUE)# Create the array on disk (it will initially be empty)arr <-tiledb_array_create(array_uri, sch)
Populate the array with 1D NumPy arrays in COO format.
import numpy as npimport tiledb# Set the coordinatesd1_data = np.array([1, 2, 3, 3])d2_data = np.array([2, 1, 3, 2])# Set the string attribute valuesa_data = np.array(["aa", "", "Ccc", "d"], dtype="O")# Write the data to the arraywith tiledb.open(array_uri, "w") as A: A[d1_data, d2_data] = a_data
# Set the coordinatesd1_data <-c(1L, 2L, 3L, 3L)d2_data <-c(2L, 1L, 3L, 2L)# Set the string attribute valuesa_data <-c("aa", "", "Ccc", "d")# Write the data to the arrayarr <-tiledb_array(uri = array_uri, query_type ="WRITE", return_as ="data.frame")arr[d1_data, d2_data] <- a_data
Read the entire array and observe the returned variable-length strings.
# Variable-length arrays may be sliced as usual in Python.# The API handles unpacking and type conversion, and returns# a NumPy object array-of-arrays.# Read all array datawith tiledb.open(array_uri) as A:print(A[:]["a"])
[b'aa' b'' b'd' b'Ccc']
# Variable-length arrays may be sliced as usual in R# The API handles unpacking and type conversion, and returns# an array of arrays.# Read all array dataprint(arr[])