Basic S3 Example

Learn how to interact with TileDB arrays hosted in S3 and other object stores.
How to run this tutorial

You can run this tutorial in two ways:

  1. Locally on your machine.
  2. On TileDB Cloud.

However, since TileDB Cloud has a free tier, we strongly recommend that you sign up and run everything there, as that requires no installations or deployment.

This tutorial shows the basic usage of TileDB on Amazon S3. It assumes you have already created an account, a bucket, and the credentials required to access the bucket. For more details on the TileDB S3 usage, as well as information about how to use TileDB with other object stores, visit the Advanced Backends section.

In order for TileDB to be able to access S3 buckets, it needs to know the S3 region and your secret keys. You need to pass this information into the configuration of a TileDB context object. It’s good practice not to share private information in notebooks. One way to do this more securely is to set your keys into environment variables and then have your code read those variables, which is what this tutorial does in the following code snippet.

import tiledb
import os

# You should set the appropriate environment variables with your keys.
# Get the keys from the environment variables.
aws_access_key_id = os.environ["AWS_ACCESS_KEY_ID"]
aws_secret_access_key = os.environ["AWS_SECRET_ACCESS_KEY"]

# Get the bucket and region from environment variables
s3_bucket = os.environ["S3_BUCKET"]
s3_region = os.environ["S3_REGION"]

# Set the AWS keys and region to the config of the default context
# This context initialization can be performed only once.
cfg = tiledb.Config(
    {
        "vfs.s3.aws_access_key_id": aws_access_key_id,
        "vfs.s3.aws_secret_access_key": aws_secret_access_key,
        "vfs.s3.region": s3_region,
    }
)
tiledb.default_ctx(cfg)

The rest of the tutorial is almost the same as the Tutorials: Basic Dense section, whereas you can create, write, and read any array in the same manner after setting up your AWS keys as shown earlier.

First, import the necessary libraries, set the array URI (that is, its path, which in this tutorial will be on local storage), and delete any previously created arrays with the same name.

# Import necessary libraries
import numpy as np

# Set array URI
array_name = "basic_object_stores"
array_uri = s3_bucket + "/" + array_name

# Delete array if it already exists
if tiledb.array_exists(array_uri):
    tiledb.Array.delete_array(array_uri)

Next, create the array by specifying its schema.

# NOTE: No other change is required for this array to work on Amazon S3.

# Create the two dimensions
d1 = tiledb.Dim(name="d1", domain=(1, 4), tile=2, dtype=np.int32)
d2 = tiledb.Dim(name="d2", domain=(1, 4), tile=2, dtype=np.int32)

# Create a domain using the two dimensions
dom = tiledb.Domain(d1, d2)

# Create an attribute
a = tiledb.Attr(name="a", dtype=np.int32)

# Create the array schema, setting `sparse=False` to indicate a dense array.
sch = tiledb.ArraySchema(domain=dom, sparse=False, attrs=[a])

# Create the array on disk (it will initially be empty)
tiledb.Array.create(array_uri, sch)

Populate the array with a 2D NumPy array.

# Prepare some data in a NumPy array
data = np.array(
    [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12], [13, 14, 15, 16]], dtype=np.int32
)

# Write data to the array
with tiledb.open(array_uri, "w") as A:
    A[:] = data

The array is now a prefix in the path specified in array_uri, which is similar to a subfolder on your local storage.

Read the data by using the slicing methods supported in TileDB.

# Open the array in read mode
A = tiledb.open(array_uri, "r")

# Show the entire array
print("Entire array: ")
print(A[:])
print("\n")

# Slice a portion of the array, which is useful
# when the arrays are too big to fit in main memory
print("Slice [1:3), [1:2): ")
print(A[1:3, 1:2]["a"])

# Remember to close the array
A.close()
Entire array: 
OrderedDict([('a', array([[ 1,  2,  3,  4],
       [ 5,  6,  7,  8],
       [ 9, 10, 11, 12],
       [13, 14, 15, 16]], dtype=int32))])


Slice [1:3), [1:2): 
[[1]
 [5]]

Clean up in the end by deleting the array.

# Delete the array
if tiledb.array_exists(array_uri):
    tiledb.Array.delete_array(array_uri)