Update Vector Index Data

You can update vector data using TileDB-Vector-Search, either single vectors or multiple vectors in a batch.

How to run this tutorial

We recommend running this tutorial, as well as the other various tutorials in the Tutorials section, inside TileDB Cloud. This will allow you to quickly experiment avoiding all the installation, deployment, and configuration hassles. Sign up for the free tier, spin up a TileDB Cloud notebook with a Python kernel, and follow the tutorial instructions. If you wish to learn how to run tutorials locally on your machine, read the Tutorials: Running Locally tutorial.

This tutorial shows you how to update a vector index, either by incrementally appending new vectors or updating existing vectors one by one, or appending/updating multiple vectors in a batch.

For more background theory on how TileDB-Vector-Search implements updates, read the Key Concepts: Vector Index Updates section.

Setup

First, import the appropriate libraries, set the index URI, and delete any previously created data.

# Import necessary libraries
import os
import shutil

import numpy as np
import tiledb.vector_search as vs

# Set the index URI for this tutorial
index_uri = os.path.expanduser("~/updates")

# Clean up previous data
if os.path.exists(index_uri):
    shutil.rmtree(index_uri)

Next, create an empty IVF_FLAT index.

# Create an index, where the dimensionality of each vector is 3,
# the type of the vector values is float32, and the index will
# use 3 partitions.
index = vs.ivf_flat_index.create(
    uri=index_uri, dimensions=3, partitions=3, vector_type=np.dtype(np.float32)
)

Query the index and note that TileDB-Vector-Search returns no results, as you have not inserted any vectors yet.

# Create a query
query_vector = np.array([[2, 2, 2]], dtype=np.float32)

# Search for its 3 nearest neigbors in the index,
# looking into 3 partitions.
result_d, result_i = index.query(query_vector, k=3, nprobe=3)
if result_i[0][0] == vs.index.MAX_UINT64:
    print("No results")

No results

Append single vectors

Now, add a set of vectors to the index:

# Apply a set of appends to the index, adding one vector at a time
index.update(vector=np.array([0, 0, 0], dtype=np.dtype(np.float32)), external_id=0)
index.update(vector=np.array([1, 1, 1], dtype=np.dtype(np.float32)), external_id=1)
index.update(vector=np.array([2, 2, 2], dtype=np.dtype(np.float32)), external_id=2)
index.update(vector=np.array([3, 3, 3], dtype=np.dtype(np.float32)), external_id=3)
index.update(vector=np.array([4, 4, 4], dtype=np.dtype(np.float32)), external_id=4)

Query the index again, and observe the three nearest neighbors (2, 3, and 1) to the original query vector:

# Perform the query again
result_d, result_i = index.query(query_vector, k=3, nprobe=3)
print("Result vector ids:\n")
print(result_i)
print("\nResult vector distances:\n")
print(result_d)

Result vector ids:

[[2 3 1]]

Result vector distances:

[[0. 3. 3.]]

Update existing vectors

You can also update an existing vector with the .update() method. Modify the values of vectors with external ids 1 and 2:

# Update existing vectors, by providing their external_id values
index.update(
    vector=np.array([10, 10, 10], dtype=np.dtype(np.float32)),
    external_id=1,
)
index.update(
    vector=np.array([11, 11, 11], dtype=np.dtype(np.float32)),
    external_id=2,
)

Query the index again and observe that the three nearest neighbors are now 3, 4, and 0.

# Perform the query again, and see that the result changes
result_d, result_i = index.query(query_vector, k=3, nprobe=3)
print("Result vector ids:\n")
print(result_i)
print("\nResult vector distances:\n")
print(result_d)

Result vector ids:

[[3 4 0]]

Result vector distances:

[[ 3. 12. 12.]]

Batch updates

You can update vectors in bulk using the .update_batch() method. Put the original vector values back for vectors with external ids 1 and 2:

# You can also update multiple vectors in a batch
update_vectors = np.empty([2], dtype=object)
update_vectors[0] = np.array([1, 1, 1], dtype=np.dtype(np.float32))
update_vectors[1] = np.array([2, 2, 2], dtype=np.dtype(np.float32))
index.update_batch(vectors=update_vectors, external_ids=np.array([1, 2]))

Query the index again and observe that the result is the same as before, namely 2, 3, and 1.

# Perform the query again, and see that the result changes again
result_d, result_i = index.query(query_vector, k=3, nprobe=3)
print("Result vector ids:\n")
print(result_i)
print("\nResult vector distances:\n")
print(result_d)

Result vector ids:

[[2 3 1]]

Result vector distances:

[[0. 3. 3.]]

Clean up

Clean up in the end by removing the index:

# Clean up
if os.path.exists(index_uri):
    shutil.rmtree(index_uri)

What’s next?

You can learn how to perform deletions in your vector index. If you are searching for more advanced tutorials, you should learn about:

• Versioning
• Time traveling
• Consolidation