Vector Search: Time Traveling

Each update and deletion creates an index version in TileDB-Vector-Search, and you can time travel over these versions.

How to run this tutorial

We recommend running this tutorial, as well as the other various tutorials in the Tutorials section, inside TileDB Cloud. This will allow you to quickly experiment avoiding all the installation, deployment, and configuration hassles. Sign up for the free tier, spin up a TileDB Cloud notebook with a Python kernel, and follow the tutorial instructions. If you wish to learn how to run tutorials locally on your machine, read the Tutorials: Running Locally tutorial.

This tutorial shows how you can perform time traveling with TileDB-Vector-Search (i.e., how to query different index “views” over the various versions that were created via vector updates or deletions). We recommend reading the following sections before proceeding with this tutorial:

• Tutorials: Versioning
• Array Key Concepts: Time traveling

Setup

First, import the appropriate libraries, set the index URI, create a query vector, and delete any previously created data.

# Import necessary libraries
import os
import shutil

import numpy as np
import tiledb.vector_search as vs

# Set the index URI for this tutorial
index_uri = os.path.expanduser("~/time_traveling")

# Set the query vector
query_vector = np.array([[2, 2, 2]], dtype=np.float32)

# Clean up previous data
if os.path.exists(index_uri):
    shutil.rmtree(index_uri)

Next, create an empty index:

# Create an index, where the dimensionality of each vector is 3,
# the type of the vector values is float32, and the index will
# use 3 partitions.
index = vs.ivf_flat_index.create(
    uri=index_uri, dimensions=3, partitions=3, vector_type=np.dtype(np.float32)
)

Populate index

Perform a series of updates, starting with adding some new vectors in bulk. Note that here you will provide timestamp = 1 as a parameter to the update_batch command. In other tutorials, this parameter was omitted, and a default timestamp was set to the current time (in milliseconds since epoch).

# Prepare some vectors to add
update_vectors = np.empty([5], dtype=object)
update_vectors[0] = np.array([0, 0, 0], dtype=np.dtype(np.float32))
update_vectors[1] = np.array([1, 1, 1], dtype=np.dtype(np.float32))
update_vectors[2] = np.array([2, 2, 2], dtype=np.dtype(np.float32))
update_vectors[3] = np.array([3, 3, 3], dtype=np.dtype(np.float32))
update_vectors[4] = np.array([4, 4, 4], dtype=np.dtype(np.float32))

# Add the vectors to the index, specifying a timestamp (1)
index.update_batch(
    vectors=update_vectors, external_ids=np.array([0, 1, 2, 3, 4]), timestamp=1
)

Next, update the values of some existing vectors one by one at timestamp = 2 and timestamp = 3, respectively. Then perform a bulk update at timestamp = 4.

# Update vectors individually
index.update(
    vector=np.array([10, 10, 10], dtype=np.dtype(np.float32)),
    external_id=1,
    timestamp=2,
)
index.update(
    vector=np.array([11, 11, 11], dtype=np.dtype(np.float32)),
    external_id=2,
    timestamp=3,
)

Delete some vectors at timestamp = 5:

# Delete the vectors with external ids 0 and 1, but at a later timestamp
index.delete_batch(external_ids=[0, 1], timestamp=5)

Query over timestamps

In order to time travel, you need to “open” the index via one of the following ways:

• At a specific timestamp (which will you give you a “view” of the array “as of” that timestamp).
• At a timestamp range (which will give you a “view” of the array considering only updates and deletions that occurred within the timestamp range).

First, query the index at timestamp=10 (i.e., a timestamp after all the performed updates and deletions), which is equivalent to not providing any timestamp (which sets by default the timestamp to the current time). The result reflects the current state of the index, with all the changes considered.

# Query the vector with `timestamp=10`.
# The result will be the same if `timestamp` is omitted,
# which implies the current time in milliseconds (>> 10).
index = vs.IVFFlatIndex(uri=index_uri, timestamp=10)
result_d, result_i = index.query(query_vector, k=3, nprobe=3)
print("Result vector ids:\n")
print(result_i)
print("\nResult vector distances:\n")
print(result_d)

Result vector ids:

[[3 4 2]]

Result vector distances:

[[  3.  12. 243.]]

Now, query the index at timestamp=0 (i.e., before any change occurred). In that case, there will be no results.

# Query the vector with `timestamp=0` -> no results
index = vs.IVFFlatIndex(uri=index_uri, timestamp=0)
result_d, result_i = index.query(query_vector, k=3, nprobe=3, timestamp=0)
if result_i[0][0] == vs.index.MAX_UINT64:
    print("No results")

No results

Next, query the index at timestamp=1 (i.e., considering only the very first bulk insertions), which will ignore the two vector value updates and deletions.

# Query the vector with `timestamp=1`.
index = vs.IVFFlatIndex(uri=index_uri, timestamp=1)
result_d, result_i = index.query(query_vector, k=3, nprobe=3)
print("Result vector ids:\n")
print(result_i)
print("\nResult vector distances:\n")
print(result_d)

Result vector ids:

[[2 3 1]]

Result vector distances:

[[0. 3. 3.]]

Finally, query with timestamp=(1,2), which considers the initial bulk vector insertion, and only the update of vector with external id equal to 1. That will ignore the update of the vector with external id equal to 2 and the two vector deletions.

# Query the vector with `timestamp=(1,2)`.
index = vs.IVFFlatIndex(uri=index_uri, timestamp=(1, 2))
result_d, result_i = index.query(query_vector, k=3, nprobe=3)
print("Result vector ids:\n")
print(result_i)
print("\nResult vector distances:\n")
print(result_d)

Result vector ids:

[[2 3 0]]

Result vector distances:

[[ 0.  3. 12.]]

Clean up

Clean up in the end by removing the index:

# Clean up
if os.path.exists(index_uri):
    shutil.rmtree(index_uri)

What’s next?

Now that you learned about how to time travel with TileDB-Vector-Search, we recommend reading the Tutorials: Consolidation section.