Learn how to time travel on array schema evolution, which is useful for when your array data model changes over time.
How to run this tutorial
You can run this tutorial in two ways:
Locally on your machine.
On TileDB Cloud.
However, since TileDB Cloud has a free tier, we strongly recommend that you sign up and run everything there, as that requires no installations or deployment.
This tutorial demonstrates the time traveling functionality on array schema evolution. For more information on time traveling and array schema evolution, visit the following sections:
First, import the necessary libraries, set the array URI (i.e., its path, which in this tutorial will be on local storage), and delete any previously created arrays with the same name.
# Import necessary librariesimport os.pathimport shutilimport numpy as npimport tiledb# Set array URIarray_uri = os.path.expanduser("~/time_traveling_schema_evolution")# Delete array if it already existsif os.path.exists(array_uri): shutil.rmtree(array_uri)
Next, create the array by specifying its schema. This example uses a dense array, but the described time traveling functionality is applicable to any array.
# Create the two dimensionsd1 = tiledb.Dim(name="d1", domain=(1, 4), tile=2, dtype=np.int32)d2 = tiledb.Dim(name="d2", domain=(1, 4), tile=2, dtype=np.int32)# Create a domain using the two dimensionsdom = tiledb.Domain(d1, d2)# Create two attributesa1 = tiledb.Attr(name="a1", dtype=np.int32)a2 = tiledb.Attr(name="a2", dtype=np.float32)# Create the array schema, setting `sparse=False` to indicate a dense array.sch = tiledb.ArraySchema(domain=dom, sparse=False, attrs=[a1, a2])# Create the array on disk (it will initially be empty)tiledb.Array.create(array_uri, sch)
Write some data to the array. Also obtain the timestamp at which the write is performed, as you will use it later for time traveling.
# Prepare some data in NumPy arraysa1_data = np.array( [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12], [13, 14, 15, 16]], dtype=np.int32)a2_data = np.array( [ [1.1, 2.2, 3.3, 4.4], [5.5, 6.6, 7.7, 8.8], [9.9, 10.10, 11.11, 12.12], [13.13, 14.14, 15.15, 16.16], ], dtype=np.float32,)# Write data to the array and retrieve the write timestamp `ts1with tiledb.open(array_uri, "w") as A: A[:] = {"a1": a1_data, "a2": a2_data} ts =list(A.last_write_info.values())[0][0]
Evolve the schema, setting for convenience the timestamp of this evolution to one millisecond after the last write. This will facilitate demonstrating time traveling below.
se = tiledb.ArraySchemaEvolution()se.timestamp(ts +1) # Perform the evolution at timestamp ts+1se.drop_attribute("a1")se.array_evolve(array_uri)
The array lives in the folder specified by array_uri. Other sections of Academy explain the contents of the array folder, but you’ll find two files in the __schema directory: one created when the array was initially created, and one created during schema evolution. TileDB prefixes these files with their creation timestamps, which you’ll use for time travel.
No timestamp provided: This means read at the current timestamp, which will return data after the initial write and the schema evolution occured. This means that only one attribute is present with its data populated.
# Read at timestamp just before schema evolutionwith tiledb.open(array_uri, "r", timestamp=ts) as A:print("Read at timestamp just before schema evolution:")print(A.schema)print(A[:])
At timestamp just after the schema evolution: This is identical to when reading the array at current timestamp (i.e., only one attribute is present with its data populated, since the evolution dropped one attribute).
# Read at timestamp after schema evolutionwith tiledb.open(array_uri, "r", timestamp=ts +1) as A:print("Read at timestamp after schema evolution:")print(A.schema)print(A[:])
At timestamp range that includes only the schema evolution: One attribute is dropped (leaving only one attribute), but the original write is outside the timestamp range. Therefore, the attribute values are not written, and the read returns empty values.
# Read at timestamp when only schema evolution occurredwith tiledb.open(array_uri, "r", timestamp=(ts +1, ts +1)) as A:print("Read at timestamp range when only schema evolution occurred:")print(A.schema)print(A[:])