Delete Vector Index Data

ai/ml

vector search

tutorials

deletions

Learn how to delete data from your vectors with TileDB-Vector-Search.

How to run this tutorial

We recommend running this tutorial, as well as the other various tutorials in the Tutorials section, inside TileDB Cloud. This will allow you to quickly experiment avoiding all the installation, deployment, and configuration hassles. Sign up for the free tier, spin up a TileDB Cloud notebook with a Python kernel, and follow the tutorial instructions. If you wish to learn how to run tutorials locally on your machine, read the Tutorials: Running Locally tutorial.

This tutorial shows you how to delete from a vector index, either by deleting vectors one by one, or deleting multiple vectors in a batch.

For more background theory on how TileDB-Vector-Search implements deletions, read the Key Concepts: Vector Index Updates section.

Setup

First, import the appropriate libraries, set the index URI, and delete any previously created data.

# Import necessary libraries
import os
import shutil

import numpy as np
import tiledb.vector_search as vs

# Set the index URI for this tutorial
index_uri = os.path.expanduser("~/deletes")

# Clean up previous data
if os.path.exists(index_uri):
    shutil.rmtree(index_uri)

Next, create an empty IVF_FLAT index.

# Create an index, where the dimensionality of each vector is 3,
# the type of the vector values is float32, and the index will
# use 3 partitions.
index = vs.ivf_flat_index.create(
    uri=index_uri, dimensions=3, partitions=3, vector_type=np.dtype(np.float32)
)

Query the index and note that TileDB-Vector-Search returns no results, as you haven’t inserted any vectors yet.

# Create a query
query_vector = np.array([[2, 2, 2]], dtype=np.float32)

# Search for its 3 nearest neigbors in the index,
# looking into 3 partitions.
result_d, result_i = index.query(query_vector, k=3, nprobe=3)
if result_i[0][0] == vs.index.MAX_UINT64:
    print("No results")

No results

Now, add a set of vectors to the index:

# Apply a set of appends to the index, adding one vector at a time
index.update(vector=np.array([0, 0, 0], dtype=np.dtype(np.float32)), external_id=0)
index.update(vector=np.array([1, 1, 1], dtype=np.dtype(np.float32)), external_id=1)
index.update(vector=np.array([2, 2, 2], dtype=np.dtype(np.float32)), external_id=2)
index.update(vector=np.array([3, 3, 3], dtype=np.dtype(np.float32)), external_id=3)
index.update(vector=np.array([4, 4, 4], dtype=np.dtype(np.float32)), external_id=4)

Query the index again, and observe the three nearest neighbors (2, 3, and 1) to the original query vector:

# Perform the query again
result_d, result_i = index.query(query_vector, k=3, nprobe=3)
print("Result vector ids:\n")
print(result_i)
print("\nResult vector distances:\n")
print(result_d)

Result vector ids:

[[2 3 1]]

Result vector distances:

[[0. 3. 3.]]

Delete single vectors

You can delete vectors as follows:

# Delete two vectors
index.delete(external_id=1)
index.delete(external_id=2)

Query the index again and observe that the three nearest neighbors are now 3, 4, and 0.

# Perform the query again, and see that the result changes
result_d, result_i = index.query(query_vector, k=3, nprobe=3)
print("Result vector ids:\n")
print(result_i)
print("\nResult vector distances:\n")
print(result_d)

Result vector ids:

[[3 4 0]]

Result vector distances:

[[ 3. 12. 12.]]

Batch deletes

You can delete vectors in bulk using the .delete_batch() method. Delete the remaining three vectors from the index.

# You can also delete multiple vectors in a batch
index.delete_batch(external_ids=[0, 3, 4])

Query the index again and observe that the result is empty again, as there are no vectors in the index.

# Perform the query again, and see that the result is empty again
result_d, result_i = index.query(query_vector, k=3, nprobe=3)
if result_i[0][0] == vs.index.MAX_UINT64:
    print("No results")

No results

Clean up

Clean up in the end by removing the index:

# Clean up
if os.path.exists(index_uri):
    shutil.rmtree(index_uri)

What’s next?

You can learn how to perform updates in your vector index. If you are searching for more advanced tutorials, you should learn about: