Vector Search

Learn how TileDB catalogs indexes for vector search.

TileDB structures and catalogs vector search indexes with its arrays

TileDB is an ideal solution for managing vector search indexes. A vector is a 1D array, and multi-dimensional arrays are first-class citizens in TileDB. Read the Structure: Vector Search section for more details on vector search, TileDB’s programmatic features, and tool integrations. This section focuses mostly on TileDB’s catalog features on the UI console.

Add vector index

Currently, you can create and register vector indexes only programmatically. Similar to arrays, you can add vectors search indexes by creating and registering them in programmatically in one step, or create them first, and register them later in a separate step. Read section Structure: Vector Search to learn how to create and register vector indexes in one step.

If you have created a vector index in your physical storage separately, and you wish to register it to the TileDB catalog, you can do so programmatically as follows:

Python

tiledb.cloud.asset.register(
    storage_uri="s3://<bucket>/<vector_index_name>",
    namespace="<account>",  # Optional, you may register it under your username, or one of your organizations
    name="<vector_name>",  # Could be different from the physical vector index name
    description=None,  # Optional, this will appear under Overview
    credentials_name="<credentials>",  # Required, which AWS credentials from your account can access the vector index.
    # You must have already added your AWS credentials in your account settings
)

A registered vector index will appear under Assets -> Data -> Vector Search in the TileDB catalog.

Overview

In this screen, you can find basic information about the vector index:

• Vector index name - This appears at the very top of the screen, and consists of the account name and the name you provided to the vector index when registering it.
• Description - If you provided a description to the vector index (programmatically or in Settings), it is visible here. The description is indexed and searchable in the catalog. Therefore, it’s recommended to add a meaningful description for all your assets.
• License - The type of license for the vector index, especially if you are making this publicly available.
• Tags - These can be used for efficient search in the catalog.
• UUID - The unique identifier for the vector index.
• Original URI - The location on cloud storage where the vector index is physically stored.
• TileDB URI - The unique resource identifier for TileDB, based on which you can call the vector index when coding. It comprises the namespace identifier and the UUID of the vector index.
• Author - Which account registered the vector index.
• Permissions - What rights the current user has on this vector index. Possible values are READ and ADMIN.
• Number of assets - TileDB models the vector index as a TileDB group, which may contain other TileDB groups and TileDB arrays.

Referring to the vector index programmatically

It is important to understand how to refer to your vector index programmatically. You can do it in two ways:

1. Using the TileDB URI format tiledb://<account>/<vector_index_name>. This is the most user-friendly way, but TileDB allows duplicated vector index names, and if you have a vector index with a non-unique name, this will throw an error.
2. Using the TileDB URI from the asset’s Overview tab (that is, the URI with format tiledb://<account>/<UUID>). TileDB URIs referencing the asset’s UUID are unique. Thus, this method will always work.

You can programmatically get overview information about the vector index with the following command:

Python

# The following will return a JSON file with various info about the vector index.
tiledb.cloud.asset.info("tiledb://<account>/<vector_index_name>")

Contents

TileDB models vector indexes as TileDB groups, which may contain TileDB arrays and other TileDB groups. The contents of a vector index are visible in the Contents tab, where you can browse them and see their details.

Metadata

Vector indexes may be associated with metadata in the form of key-value pairs, which is visible in the Metadata tab.

Sharing & activity

The Sharing screen allows you to securely share your vector index with other TileDB users, whereas the Activity screen shows you the various accesses performed on the vector index by you or any other user you have shared your vector index with. They are both covered in detail in the Collaborate section.

Settings

In the vector index settings, you can modify the following:

• Description - Note that this is indexed and, thus, searchable in the TileDB catalog.
• Tags - These can be used for efficient search in the catalog.
• License - The type of license for the vector index, especially if you are making this publicly available.
• Make public - If you wish to share the vector index with all the TileDB users. This will appear in the Marketplace tab in the left navigation menu. If you make a vector index public, you can easily change it back to private in the same manner.
• Change cloud credentials - Credentials should be provided so that TileDB can securely access the vector index on the cloud store where it is physically stored.
• Rename vector index - Read the Rename vector index subsection below.
• Delete vector index - Read the Delete vector index subsection below.

You can programmatically update some vector index settings with the following command:

Python

tiledb.cloud.asset.update_info(
    uri="tiledb://<account>/<vector_index_name>",
    description=None,  # Optional - A new description
    name=None,  # Optional - A new name for the vector index
    tags=None,  # Optional - Vector index tags that will be searchable in the catalog
    access_credentials_name=None,  # Optional - The cloud credentials that access the vector index (should already exist in your account settings)
)

To make a vector index public programmatically, run the following:

Python

tiledb.cloud.asset.share(
    "tiledb://<account>/<vector_index_name>", namespace="public", permissions="read"
)

Rename vector index

A useful property of the TileDB catalog and the way it registers vector indexes is that you can easily rename a vector index, without physically moving it, thus avoiding the very expensive copying operations entailed in object stores when physically renaming/moving file objects. You can rename a vector index from the Settings tab.

You can programmatically rename a vector index as follows:

Python

tiledb.cloud.asset.update_info(
    "`tiledb://<account>/<previous_name>`", name="<new_name>"
)

Warning

Take caution when renaming vector indexes, as any URIs including the previous vector index name will no longer work.

Delete vector index

When deleting a vector index, you have two options:

• Unregister: This operation removes the vector index from the TileDB catalog, but it does not physically remove it from the object store. Since the vector index will persist on storage, you can register it again in the TileDB catalog in the future.
• Delete: This operation both unregisters and physically removes the vector index from storage. Note that this operation cannot be undone.

You can delete the vector index from the Settings tab, which will prompt you to choose among the two operations above.

You can also programmatically delete or unregister the vector index as follows:

Python

# Unregister a vector index
tiledb.cloud.asset.deregister(uri="tiledb://<account>/<vector_index_name>")

# Delete a vector index
tiledb.cloud.asset.delete(uri="tiledb://<account>/<vector_index_name>")