Access Arrays on TileDB Cloud

arrays

tutorials

python

remote

Learn how to interact with arrays on TileDB Cloud programmatically.

How to run this tutorial

You can run this tutorial only on TileDB Cloud. However, TileDB Cloud has a free tier. We strongly recommend that you sign up and run everything there, as that requires no installations or deployment.

This tutorial shows basic usage of arrays on TileDB Cloud. It assumes you have already completed the Get Started section.

Start by setting some environment variables. It can look something like this:

Python

import os

os.environ["S3_BUCKET"] = "s3://<your_bucket_name>"
os.environ["TILEDB_ACCOUNT"] = "<your_tiledb_account_username>"

Next, fetch the environment variables you created and define the global context:

Python

import os

import tiledb

# TileDB automatically sets the TILEDB_REST_TOKEN variable in your
# running server.
tiledb_token = os.environ["TILEDB_REST_TOKEN"]

# You should set the appropriate environment variables with your keys.
# Get the keys from the environment variables.
tiledb_account = os.environ["TILEDB_ACCOUNT"]

# Get the bucket from environment variables
s3_bucket = os.environ["S3_BUCKET"]

# Define a configuration object and set the default context.
# This context initialization can be performed only once.
cfg = tiledb.Config(
    {
        "rest.token": tiledb_token,
    }
)
tiledb.default_ctx(cfg)

The rest of the tutorial is almost the same as the Tutorials: Basic Dense section, whereas you can create, write, and read any array in the same manner after setting up your TileDB credentials as shown earlier.

Import the necessary libraries, set the array URI (that is, its path, which in this tutorial will be on local storage), and delete any previously created arrays with the same name.

Python

# Import necessary libraries
import numpy as np

# Set array URI
array_name = "basic_tiledb_cloud"
array_uri = "tiledb://" + tiledb_account + "/" + array_name

# Delete array if it already exists
if tiledb.array_exists(array_uri):
    tiledb.Array.delete_array(array_uri)

Next, create the array by specifying its schema. The only difference between TileDB Cloud and TileDB Open-source when creating and registering arrays is that the TileDB Cloud URI should be of the form: tiledb://<account>/s3://<bucket>/<array_name>. TileDB Cloud understands that you are trying to create an array in S3 URI s3://<bucket>/<array_name> and register it under <account>. After you create and register the array, you can access the array as tiledb://<account>/<array_name>.

Python

# Create the two dimensions
d1 = tiledb.Dim(name="d1", domain=(1, 4), tile=2, dtype=np.int32)
d2 = tiledb.Dim(name="d2", domain=(1, 4), tile=2, dtype=np.int32)

# Create a domain using the two dimensions
dom = tiledb.Domain(d1, d2)

# Create an attribute
a = tiledb.Attr(name="a", dtype=np.int32)

# Create the array schema, setting `sparse=False` to indicate a dense array.
sch = tiledb.ArraySchema(domain=dom, sparse=False, attrs=[a])

# NOTE: This is the only special thing about TileDB Cloud when
# creating and registering arrays: the URI should be of the form:
# tiledb://<account>/s3://<bucket>/<array_name>
# TileDB Cloud understands that you are trying to create an array in
# s3://<bucket>/<array_name> and register it under <account>.
# After the array is created and registered, it will be accessible
# as just tiledb://<account>/<array_name>
array_uri_reg = "tiledb://" + tiledb_account + "/" + s3_bucket + "/" + array_name

# Create the array on disk (it will initially be empty)
tiledb.Array.create(array_uri_reg, sch)

Populate the array with a 2-dimensional array. Observe that now you use the array URI in the form tiledb://<account>/<array_name>.

Python

# Prepare some data in a NumPy array
data = np.array(
    [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12], [13, 14, 15, 16]], dtype=np.int32
)

# Write data to the array
with tiledb.open(array_uri, "w") as A:
    A[:] = data

Read the data by using the slicing methods supported in TileDB. Again, you access the array with URI tiledb://<account>/<array_name>.

Python

# Open the array in read mode
A = tiledb.open(array_uri, "r")

# Show the entire array
print("Entire array: ")
print(A[:])
print("\n")

# Slice a portion of the array, which is useful
# when the arrays are too big to fit in main memory
print("Slice [1:3), [1:2): ")
print(A[1:3, 1:2]["a"])

# Remember to close the array
A.close()

Entire array: 
OrderedDict([('a', array([[ 1,  2,  3,  4],
       [ 5,  6,  7,  8],
       [ 9, 10, 11, 12],
       [13, 14, 15, 16]], dtype=int32))])


Slice [1:3), [1:2): 
[[1]
 [5]]

Clean up in the end by deleting the array. Observet that the standard TileDB object management functions work directly with tiledb:// URIs (that is, TileDB Cloud arrays).

Python

# Delete the array
if tiledb.array_exists(array_uri):
    tiledb.Array.delete_array(array_uri)