Quickstart

Learn how to install TileDB-Vector-Search, build an index, and perform basic searches with this guided tutorial.
How to run this tutorial

We recommend running this tutorial, as well as the other various tutorials in the Tutorials section, inside TileDB Cloud. This will allow you to quickly experiment avoiding all the installation, deployment, and configuration hassles. Sign up for the free tier, spin up a TileDB Cloud notebook with a Python kernel, and follow the tutorial instructions. If you wish to learn how to run tutorials locally on your machine, read the Tutorials: Running Locally tutorial.

This is a quickstart tutorial on the basic vector search capabilities of TileDB.

Setup

You will ingest the small (10k) SIFT dataset from the Datasets for approximate nearest neighbor search site. You will download a mirrored copy of the dataset from the TileDB-Vector-Search repo on GitHub. The original source can be found here: ftp://ftp.irisa.fr/local/texmex/corpus/sift.tar.gz.

First, import the necessary libraries, set up the URIs you will use throughout the tutorial, and clean up any previously created directories and vector indexes.

# Import necessary libraries
import os
import tarfile
import shutil
import urllib.request
import numpy as np
import tiledb.vector_search as vs
from tiledb.vector_search.utils import load_fvecs, load_ivecs
import tiledb

# The URIs for the data to download and ingest
data_uri = "https://github.com/TileDB-Inc/TileDB-Vector-Search/releases/download/0.0.1/siftsmall.tgz"
data_filename = "siftsmall.tar.gz"
data_dir = os.path.expanduser("~/sift10k/")
local_data_path = os.path.join(data_dir, data_filename)
index_uri = os.path.expanduser("~/sift10k_flat")

# Clean up previous data
if os.path.exists(data_dir):
    shutil.rmtree(data_dir)
if os.path.exists(index_uri):
    shutil.rmtree(index_uri)

Next, download and untar the source dataset.

# Create a directory to store the source dataset
os.makedirs(os.path.dirname(data_dir))

# Download the file that contains the vector dataset
urllib.request.urlretrieve(data_uri, local_data_path)

# untar the file
tarfile.open(local_data_path, "r:gz").extractall(
    os.path.dirname(local_data_path), filter="fully_trusted"
)

Ingest

Once the tarball has been extracted, ingest the data into a FLAT index (the simplest index in TileDB-Vector-Search) as follows:

index = vs.ingest(
    index_type="FLAT",
    index_uri=index_uri,
    source_uri=os.path.join(data_dir, "siftsmall_base.fvecs"),
    source_type="FVEC",
)

Inspect

This ingestion creates a group with three arrays, which can be shown as shown below. In addition, you can see the array schema of the array that stores the vectors, as well as print the contents of the first vector.

# Show the physical group
group = tiledb.Group(index_uri, "r")
print("Index physical contents:\n")
print(group)

# Prepare the index for reading
index = vs.FlatIndex(index_uri)

# Open the vector array to inspect it
print("Vector array URI:", index.db_uri, "\n")
A = tiledb.open(index.db_uri)

# Print the schema of the vector array
print("Vector array schema:\n")
print(A.schema)

# Print the first vector
print("Contents of first vector:\n")
print(A[:, 0]["values"])
Index physical contents:

sift10k_flat GROUP
|-- shuffled_vector_ids ARRAY
|-- shuffled_vectors ARRAY
|-- updates ARRAY

Vector array URI: file:///home/jovyan/sift10k_flat/shuffled_vectors 

Vector array schema:

ArraySchema(
  domain=Domain(*[
    Dim(name='rows', domain=(0, 127), tile=128, dtype='int32', filters=FilterList([ZstdFilter(level=-1), ])),
    Dim(name='cols', domain=(0, 2147483647), tile=250000, dtype='int32', filters=FilterList([ZstdFilter(level=-1), ])),
  ]),
  attrs=[
    Attr(name='values', dtype='float32', var=False, nullable=False, enum_label=None, filters=FilterList([ZstdFilter(level=-1), ])),
  ],
  cell_order='col-major',
  tile_order='col-major',
  sparse=False,
)

Contents of first vector:

[  0.  16.  35.   5.  32.  31.  14.  10.  11.  78.  55.  10.  45.  83.
  11.   6.  14.  57. 102.  75.  20.   8.   3.   5.  67.  17.  19.  26.
   5.   0.   1.  22.  60.  26.   7.   1.  18.  22.  84.  53.  85. 119.
 119.   4.  24.  18.   7.   7.   1.  81. 106. 102.  72.  30.   6.   0.
   9.   1.   9. 119.  72.   1.   4.  33. 119.  29.   6.   1.   0.   1.
  14.  52. 119.  30.   3.   0.   0.  55.  92. 111.   2.   5.   4.   9.
  22.  89.  96.  14.   1.   0.   1.  82.  59.  16.  20.   5.  25.  14.
  11.   4.   0.   0.   1.  26.  47.  23.   4.   0.   0.   4.  38.  83.
  30.  14.   9.   4.   9.  17.  23.  41.   0.   0.   2.   8.  19.  25.
  23.   1.]

Next steps

Now that you have a basic understanding of how to ingest data and run similarity search queries using TileDB-Vector-Search, you can continue learning about TileDB-Vector-Search:

  1. The foundation docs explain how TileDB has implemented vector search.
  2. The tutorials cover the broad functionality and use cases of vector search in TileDB.
  3. The API reference provides more information about the usage of TileDB-Vector-Search.