Array Assets

How to manage your arrays in TileDB.

Arrays are first-class citizens in TileDB

TileDB architects its entire data engine around the multi-dimensional array, a powerful data structure that shape-shifts to adapt and structure any modality, no matter how complex. You can learn about its mechanics, theoretical background, and detailed tutorials in section Structure: Arrays.

Add array

With TileDB, you can physically create and store all arrays in TileDB’s cloud-optimized storage format in a cloud object store like Amazon S3, which is completely under your control. You can then register the array with TileDB, so that it is added to its catalog. In other words, a TileDB array can physically exist, even if it is not part of the TileDB catalog (which offers flexibility, but it is not recommended as you may lose track of your arrays).

Given the above, you can add an array to the TileDB catalog in two ways:

• Create and register in one step: TileDB offers you programmatic APIs to physically create and register your arrays in a single step.
• Create first, register after: Alternatively, you may first physically create an array in your object store, without necessitating cataloging it with TileDB as well. At any point in time, you can register any existing arrays with TileDB using the UI or an API command.

Create and register in one step

Creating and registering an array in a single step is covered in detail in section Array Tutorials: Basic TileDB Cloud. It boils down to the following command of the TileDB core library, where array_uri_reg has format tiledb://<account>/s3://<bucket>/<array_name> and schema is the array schema object you selected:

Python

tiledb.Array.create("tiledb://<account>/s3://<bucket>/<array_name>", schema)

In the above example, the array will be physically stored in path s3://<bucket>/<array_name>, but it will appear and be referenced by tiledb://<account>/<array_name> in the TileDB catalog after creation.

Create first, register later

Suppose you have already created a TileDB array on Amazon S3, which is covered in detail in section Array Tutorials: Basic S3. You can register this array in two ways, either from the TileDB UI console, or programmatically.

From the UI, follow these steps:

1. Select Assets from the left navigation menu.
2. Select the Add Asset button.
3. Choose Data on the emerging window, and then Array.
4. On the emerging modal, provide the following:
   1. Cloud credentials - (required) The credentials used to access the source file on cloud storage. You must have already set these up in your account settings.
   2. Register from - (required) This is the full URI path of the physical array on Amazon S3.
   3. Array name - (required) The name you will give to the array that will be listed in your catalog. This name can be different from the physical array name.
   4. License - (optional) The license under which this array will be available. It is strongly recommended to use a license when you make an asset publicly available.
   5. Tags - (optional) You may want to apply tags on the array, which are useful for catalog search later.
5. Select Register to complete the process.

Programmatically, you can register an existing array with the following command:

Python

tiledb.cloud.array.register_array(
    uri="s3://<bucket>/<array_name>",
    namespace="<account>",  # Optional, you may register it under your username, or one of your organizations
    array_name="<array_name>",  # Could be different from the physical array name
    description=None,  # Optional, this will appear under Overview
    access_credentials_name="<credentials>",  # Required, which AWS credentials from your account can access the array.
    # You must have already added your AWS credentials in your account settings
)

A registered array will appear under Assets -> Data -> Arrays in the TileDB catalog.

Overview

In this screen, you can find basic information about the array:

• Array name - This appears at the very top of the screen, and consists of the account name and the name you provided to the array when registering it.

• Description - If you provided a description to the array (programmatically or in Settings), it is visible here. The description is indexed and searchable in the catalog. Therefore, it’s recommended to add a meaningful description for all your assets.

• License - The type of license for the array, especially if you are making this publicly available.

• Tags - These can be used for efficient search in the catalog.

• UUID - The unique identifier for the array.

• Original URI - The location on cloud storage where the array is physically stored.

• TileDB URI - The unique resource identifier for TileDB, based on which you can call the array when coding. It comprises the namespace identifier and the UUID of the array.

• Size - The size of the array.

• Type - The type of the array. It can be Dense or Sparse.

• Permissions - What rights the current user has on this array. Possible values are READ and ADMIN.

Referring to the array programmatically

It is important to understand how to refer to your array programmatically. You can do it in two ways:

1. Using the TileDB URI format tiledb://<account>/<array_name>. This is the most user-friendly way, but TileDB allows duplicated array names, and if you have an array with a non-unique name, this will throw an error.
2. Using the TileDB URI from the asset’s Overview tab (that is, the URI with format tiledb://<account>/<UUID>). TileDB URIs referencing the asset’s UUID are unique. Thus, this method will always work.

You can programmatically get overview information about the array with the following command:

Python

# The following will return a JSON file with various info about the array.
tiledb.cloud.asset.info("tiledb://<account>/<array_name>")

Schema

You can see the detailed array schema in this screen. For more information about the array schema, read section Array Tutorials: Schema.

Metadata

Each array can be associated with metadata, which is in the form of key-value pairs. For more information on how to attach metadata to arrays programmatically, read the Array Tutorials: Metadata section. Note that the array metadata is indexed by the TileDB catalog and, therefore, is fully searchable.

Sharing & activity

The Sharing screen allows you to securely share your array with other TileDB users, whereas the Activity screen shows you the various accesses performed on the array by you or any other user you have shared your array with. They are both covered in detail in the Collaborate section.

Settings

In the array settings, you can:

• Add a description - Note that this is indexed and, thus, searchable in the TileDB catalog.
• Make array read-only - This is useful if you want to prevent any array changes by you or someone with whom you shared the array.
• Tags - These can be used for efficient search in the catalog.
• License - The type of license for the array, especially if you are making this publicly available.
• Make public - If you wish to share the array with all the TileDB users. This will appear in the Marketplace tab in the left navigation menu. If you make an array public, you can easily change it back to private in the same manner.
• Change cloud credentials - Credentials should be provided so that TileDB can securely access the array on the cloud store where it is physically stored.
• Rename array - Read the Rename array subsection below.
• Delete array - Read the Delete array subsection below.

You can programmatically update some array settings with the following command:

Python

tiledb.cloud.asset.update_info(
    uri="tiledb://<account>/<array_name>",
    description=None,  # Optional - A new description
    name=None,  # Optional - A new name for the array
    tags=None,  # Optional - Array tags that will be searchable in the catalog
    access_credentials_name=None,  # Optional - The cloud credentials that access the array (should already exist in your account settings)
)

To make an array public programmatically, run the following:

Python

tiledb.cloud.asset.share(
    "tiledb://<account>/<array_name>", namespace="public", permissions="read"
)

Rename array

A useful property of the TileDB catalog and the way it registers arrays is that you can easily rename an array, without physically moving it, thus avoiding the very expensive copying operations entailed in object stores when physically renaming/moving file objects. You can rename an array from the Settings tab.

You can programmatically rename an array as follows:

Python

tiledb.cloud.asset.update_info(
    "`tiledb://<account>/<previous_name>`", name="<new_name>"
)

Warning

Take caution when renaming arrays, as any URIs including the previous array name will no longer work.

Delete array

When deleting an array, you have two options:

• Unregister: This operation removes the array from the TileDB catalog, but it does not physically remove it from the object store. Since the array will persist on storage, you can register it again in the TileDB catalog in the future.
• Delete: This operation both unregisters and physically removes the array from storage. Note that this operation cannot be undone.

You can delete the array from the Settings tab, which will prompt you to choose among the two operations above.

You can also programmatically delete or unregister the array as follows:

Python

# Unregister an array
tiledb.cloud.array.deregister_array(uri="tiledb://<account>/<array_name>")

# Delete an array
tiledb.cloud.array.delete_array(uri="tiledb://<account>/<array_name>")