Ingestion & Querying

ai/ml
vector search
tutorials
ingestion
queries
Learn how to ingest basic vector data with TileDB-Vector-Search and perform similarity search.
How to run this tutorial

We recommend running this tutorial, as well as the other various tutorials in the Tutorials section, inside TileDB Cloud. This will allow you to quickly experiment avoiding all the installation, deployment, and configuration hassles. Sign up for the free tier, spin up a TileDB Cloud notebook with a Python kernel, and follow the tutorial instructions. If you wish to learn how to run tutorials locally on your machine, read the Tutorials: Running Locally tutorial.

In this tutorial, you will learn how to ingest a set vectors into an IVF_FLAT index and perform basic similarity search.

Setup

You will ingest the small (10k) SIFT dataset from the Datasets for approximate nearest neighbor search site. You will download a mirrored copy of the dataset from the TileDB-Vector-Search repo on GitHub. The original source can be found here: ftp://ftp.irisa.fr/local/texmex/corpus/sift.tar.gz.

First, import the necessary libraries, set up the URIs you will use throughout the tutorial, and clean up any previously created directories and vector indexes.

# Import necessary libraries
import os
import shutil
import tarfile
import urllib.request

import numpy as np
import tiledb
import tiledb.vector_search as vs
from tiledb.vector_search.utils import load_fvecs, load_ivecs

# The URIs for the data to download and ingest
data_uri = "https://github.com/TileDB-Inc/TileDB-Vector-Search/releases/download/0.0.1/siftsmall.tgz"
data_filename = "siftsmall.tar.gz"
data_dir = os.path.expanduser("~/sift10k/")
local_data_path = os.path.join(data_dir, data_filename)
index_uri = os.path.expanduser("~/sift10k_ivf_flat")

# Clean up previous data
if os.path.exists(data_dir):
    shutil.rmtree(data_dir)
if os.path.exists(index_uri):
    shutil.rmtree(index_uri)

Next, download and untar the source dataset.

# Create a directory to store the source dataset
os.makedirs(os.path.dirname(data_dir))

# Download the file that contains the vector dataset
urllib.request.urlretrieve(data_uri, local_data_path)

# untar the file
tarfile.open(local_data_path, "r:gz").extractall(
    os.path.dirname(local_data_path), filter="fully_trusted"
)

Ingest

Once the tarball has been extracted, ingest the data into a IVF_FLAT index as follows:

index = vs.ingest(
    index_type="IVF_FLAT",
    source_uri=os.path.join(data_dir, "siftsmall_base.fvecs"),
    index_uri=index_uri,
    source_type="FVEC",
    partitions=100,
)

You can similarly create other types of indexes, such as FLAT and VAMANA. Visit the Key Concepts: Vector Indexing Algorithms, Key Concepts: Distance Metrics, and Key Concepts: Vector Search Performance sections to learn more about how to choose the right index for your use case and fine-tune it for maximum performance.

Inspect

This ingestion creates a group with three arrays, which can be shown as shown below. In addition, you can see the array schema of the array that stores the vectors, as well as print the contents of the first vector.

# Show the physical group
group = tiledb.Group(index_uri, "r")
print("Index physical contents:\n")
print(group)

# Prepare the index for reading
index = vs.IVFFlatIndex(index_uri)

# Open the vector array to inspect it
print("Vector array URI:", index.db_uri, "\n")
A = tiledb.open(index.db_uri)

# Print the schema of the vector array
print("Vector array schema:\n")
print(A.schema)

# Print the first vector
print("Contents of first vector:\n")
print(A[:, 0]["values"])
Index physical contents:

sift10k_ivf_flat GROUP
|-- partition_centroids ARRAY
|-- partition_indexes ARRAY
|-- shuffled_vector_ids ARRAY
|-- shuffled_vectors ARRAY
|-- updates ARRAY

Vector array URI: file:///home/jovyan/sift10k_ivf_flat/shuffled_vectors 

Vector array schema:

ArraySchema(
  domain=Domain(*[
    Dim(name='rows', domain=(0, 127), tile=128, dtype='int32', filters=FilterList([ZstdFilter(level=-1), ])),
    Dim(name='cols', domain=(0, 2147483647), tile=125000, dtype='int32', filters=FilterList([ZstdFilter(level=-1), ])),
  ]),
  attrs=[
    Attr(name='values', dtype='float32', var=False, nullable=False, enum_label=None, filters=FilterList([ZstdFilter(level=-1), ])),
  ],
  cell_order='col-major',
  tile_order='col-major',
  sparse=False,
)

Contents of first vector:

[ 14.  14.  26.   1.   0.   0.  12.  12.   6.  21. 123.  12.   0.   0.
  10.   7.   0.   9. 112.  14.   1.   0.  12.   1.   0.  17.  96.   4.
   0.   0.   2.   0. 139.  49.   7.   0.   0.   1.  12.  24. 128.  99.
 139.  35.   0.   0.   0.   0.   0.  18. 139.  52.   1.   0.   0.   0.
   0.  54. 139.   3.   0.   0.   0.   0. 139.  52.   0.   0.   0.   0.
   1.  35. 139.  36.   8.   3.   0.   0.   0.  35.   2.   2.  28.  13.
  14.   6.   0.   3.  10.  25.  13.   2.   4.   3.   1.   1. 126.   0.
   0.   0.   0.   0.   0. 116.  89.   0.   0.   0.   0.   0.  12. 139.
   5.   1.   0.   0.   5.   7.  28.  21.   3.   2.   0.   0.   1.   3.
  12.   6.]

Clean up

Clean up in the end by removing the created directory and index:

# Clean up
if os.path.exists(data_dir):
    shutil.rmtree(data_dir)
if os.path.exists(index_uri):
    shutil.rmtree(index_uri)

Indexes on TileDB Cloud

Note the following when creating indexes and registering them on TileDB Cloud:

  1. When creating a new index, you need to use a URI in the form tiledb://<your_username>/<S3_path>/<index_name>, where S3_path is the location on S3 where you wish to physically store the index.
  2. When referring to the index after creating it (e.g., when submitting queries), use a URI in the form tiledb://<your_username>/<index_name> (i.e., no need to specify the S3 physical path anymore).

Set up the URIs and clean up any previously created index with the same name on TileDB Cloud.

# Get your username
username = tiledb.cloud.user_profile().username

# Get the bucket from an environment variable
s3_bucket = os.environ["S3_BUCKET"]

# Set index URI
index_name = "cloud_index"
index_uri = "tiledb://" + username + "/" + index_name
index_reg_uri = "tiledb://" + username + "/" + s3_bucket + "/" + index_name

# The TileDB Cloud context
ctx = tiledb.cloud.Ctx()

# Clean up index if it already exists
if tiledb.object_type(index_uri, ctx=ctx) == "group":
    tiledb.cloud.asset.delete(index_uri, recursive=True)

Create the index, using the tiledb://<your_username>/<S3_path>/<index_name> URI format explained above:

# Create an index, where the dimensionality of each vector is 3,
# the type of the vector values is float32, and the index will
# use 3 partitions.
index = vs.ivf_flat_index.create(
    ctx=ctx,
    uri=index_reg_uri,
    dimensions=3,
    partitions=3,
    vector_type=np.dtype(np.float32),
)

From this point onwards, you can populate and query the index just using the tiledb://<your_username>/<index_name> URI format explained above.

The following with delete the physical files of the index and deregister it from TileDB Cloud.

# Clean up index if it already exists
if tiledb.object_type(index_uri, ctx=ctx) == "group":
    tiledb.cloud.asset.delete(index_uri, recursive=True)

What’s next?

Now that you know how to perform basic ingestion and similarity search, you are ready to learn how to perform updates and deletions.