Fragments

Learn how to time travel across different write operations on your arrays.

How to run this tutorial

You can run this tutorial in two ways:

Locally on your machine.
On TileDB Cloud.

However, since TileDB Cloud has a free tier, we strongly recommend that you sign up and run everything there, as that requires no installations or deployment.

This tutorial demonstrates the time traveling functionality on fragments (that is, on the various write operations performed to the array). For more information on time traveling, visit the Key Concepts: Time Traveling section.

First, import the necessary libraries, set the array URI (i.e., its path, which in this tutorial will be on local storage), and delete any previously created arrays with the same name.

Python
R

# Import necessary libraries
import tiledb
import numpy as np
import shutil
import os.path

# Set array URI
array_uri = os.path.expanduser("~/time_traveling_fragments")

# Delete array if it already exists
if os.path.exists(array_uri):
    shutil.rmtree(array_uri)

# Import necessary libraries
library(tiledb)

# Set array URI
array_uri <- path.expand("~/time_traveling_fragments_r")

# Delete array if it already exists
if (file.exists(array_uri)) {
  unlink(array_uri, recursive = TRUE)
}

Next, create the array by specifying its schema. This example uses a sparse array, but the described time traveling functionality is applicable to any array.

Python
R

# Create the two dimensions
d1 = tiledb.Dim(name="d1", domain=(0, 3), tile=2, dtype=np.int32)
d2 = tiledb.Dim(name="d2", domain=(0, 3), tile=2, dtype=np.int32)

# Create a domain using the two dimensions
dom = tiledb.Domain(d1, d2)

# Create an attribute
a = tiledb.Attr(name="a", dtype=np.int32)

# Create the array schema with `sparse=True`.
sch = tiledb.ArraySchema(domain=dom, sparse=True, attrs=[a])

# Create the array on disk (it will initially be empty)
tiledb.Array.create(array_uri, sch)

# Create the two dimensions
d1 <- tiledb_dim("d1", c(0L, 3L), 2L, "INT32")
d2 <- tiledb_dim("d2", c(0L, 3L), 2L, "INT32")

# Create a domain using the two dimensions
dom <- tiledb_domain(dims = c(d1, d2))

# Create an attribute
a <- tiledb_attr("a", type = "INT32")

# Create the array schema with `sparse = TRUE`
sch <- tiledb_array_schema(dom, a, sparse = TRUE)

# Create the array on disk (it will initially be empty)
arr <- tiledb_array_create(array_uri, sch)

Populate the array using a set of 1D arrays, one for the coordinates of each dimension, and one for the attribute values (TileDB sparse arrays expect the COO format). Observe that the write is perfomed at a specific user-defined timestamp (namely, at 1), in order to make time traveling easier later.

Python
R

# Prepare some data in numpy arrays
d1_data = np.array([2, 0, 3, 2, 0, 1], dtype=np.int32)
d2_data = np.array([0, 1, 1, 2, 3, 3], dtype=np.int32)
a_data = np.array([4, 1, 6, 5, 2, 3], dtype=np.int32)

# Open the array in write mode and write the data in COO format
with tiledb.open(array_uri, "w", timestamp=1) as A:
    A[d1_data, d2_data] = a_data

# Prepare some data in an array
d1_data <- c(2L, 0L, 3L, 2L, 0L, 1L)
d2_data <- c(0L, 1L, 1L, 2L, 3L, 3L)
a_data <- c(4L, 1L, 6L, 5L, 2L, 3L)

# Open the array for writing and write data to the array
arr <- tiledb_array(
  uri = array_uri
)
invisible(tiledb_array_close(arr))

arr <- tiledb_array_open_at(
  arr,
  type = "WRITE",
  timestamp = as.POSIXct(1)
)

arr[d1_data, d2_data] <- a_data

# Close the array
invisible(tiledb_array_close(arr))

arr <- tiledb_array_open_at(
  arr,
  type = "READ",
  timestamp = as.POSIXct(1)
)
cat("Entire array at timestamp 1:\n")
print(arr[])
cat("\n\n")
arr <- tiledb_array_close(arr)

Entire array at timestamp 1:
  d1 d2 a
1  0  1 1
2  2  0 4
3  3  1 6
4  0  3 2
5  1  3 3
6  2  2 5

Perform a second write at timestamp 2.

Python
R

# Prepare some data in numpy arrays
d1_data = np.array([1], dtype=np.int32)
d2_data = np.array([1], dtype=np.int32)
a_data = np.array([100], dtype=np.int32)

# Open the array in write mode and write the data in COO format
with tiledb.open(array_uri, "w", timestamp=2) as A:
    A[d1_data, d2_data] = a_data

# Prepare some data
d1_data <- 1L
d2_data <- 1L
a_data <- 100L

# Open the array for writing and write data to the array
arr <- tiledb_array_open_at(
  arr,
  type = "WRITE",
  timestamp = as.POSIXct(2)
)

arr[d1_data, d2_data] <- a_data

# Close the array
invisible(tiledb_array_close(arr))

The array is a folder in the path specified in array_uri. The contents are explained in other sections of the Academy, but you can observe that the above write operations created two fragment folders inside the fragments directory, one prefixed with _1_1 and one with _2_2, which are the timestamps the writes were performed at.

/Users/stavrospapadopoulos/time_traveling_fragments
├── __commits
│   ├── __1_1_5c4d161a2f605cbfc4fd9cc73b8707d0_21.wrt
│   └── __2_2_5a61590ef44f484ed16dc62d1541ceb0_21.wrt
├── __fragment_meta
├── __fragments
│   ├── __1_1_5c4d161a2f605cbfc4fd9cc73b8707d0_21
│   │   ├── __fragment_metadata.tdb
│   │   ├── a0.tdb
│   │   ├── d0.tdb
│   │   └── d1.tdb
│   └── __2_2_5a61590ef44f484ed16dc62d1541ceb0_21
│       ├── __fragment_metadata.tdb
│       ├── a0.tdb
│       ├── d0.tdb
│       └── d1.tdb
├── __labels
├── __meta
└── __schema
    ├── __1715893437411_1715893437411_00000002fe50a29790bda3a356126990
    └── __enumerations

10 directories, 11 files

Next, read the array at different timestamp combinations:

No timestamp provided: This means read at the current timestamp, which will include data from both the write operations.
At timestamp 1: This will include data from only the first write (i.e., read the array as of timestamp 1).
At timestamp 2: This will include data from both writes (i.e., read the array as of timestamp 2)
At timestamp range [2,2]: This will include data written only within the timestamp range, i.e., data only from the second write.

In a real-world scenario, the fragment timestamps will be the actual times at which the writes occured. In order to see what fragments the array contains and guide your time traveling, visit the Tutorials: Get Fragment Info section.

Python
R

# Read the entire array at current timestamp
with tiledb.open(array_uri, "r") as A:
    print("Entire array at current timestamp: ")
    print(A[:])
    print("\n")

# Read the entire array at timestamp 1
with tiledb.open(array_uri, "r", timestamp=1) as A:
    print("Entire array at timestamp 1: ")
    print(A[:])
    print("\n")

# Read the entire array at timestamp 2
with tiledb.open(array_uri, "r", timestamp=2) as A:
    print("Entire array at timestamp 2: ")
    print(A[:])
    print("\n")

# Read the entire array at timestamp range [2,2]
with tiledb.open(array_uri, "r", timestamp=(2, 2)) as A:
    print("Entire array at timestamp range [2,2]: ")
    print(A[:])

Entire array at current timestamp: 
OrderedDict({'a': array([  1, 100,   2,   3,   4,   6,   5], dtype=int32), 'd1': array([0, 1, 0, 1, 2, 3, 2], dtype=int32), 'd2': array([1, 1, 3, 3, 0, 1, 2], dtype=int32)})


Entire array at timestamp 1: 
OrderedDict({'a': array([1, 2, 3, 4, 6, 5], dtype=int32), 'd1': array([0, 0, 1, 2, 3, 2], dtype=int32), 'd2': array([1, 3, 3, 0, 1, 2], dtype=int32)})


Entire array at timestamp 2: 
OrderedDict({'a': array([  1, 100,   2,   3,   4,   6,   5], dtype=int32), 'd1': array([0, 1, 0, 1, 2, 3, 2], dtype=int32), 'd2': array([1, 1, 3, 3, 0, 1, 2], dtype=int32)})


Entire array at timestamp range [2,2]: 
OrderedDict({'a': array([100], dtype=int32), 'd1': array([1], dtype=int32), 'd2': array([1], dtype=int32)})

# Read the entire array at current timestamp
arr <- tiledb_array_open(arr, type = "READ")
cat("Entire array:\n")
print(arr[])
cat("\n\n")
arr <- tiledb_array_close(arr)

# Read the array up to and including timestamp 1
arr <- tiledb_array(
  uri = array_uri,
  query_type = "READ",
  timestamp_end = as.POSIXct(1),
  return_as = "data.frame"
)
cat("Entire array at timestamp 1:\n")
print(arr[])
cat("\n\n")
arr <- tiledb_array_close(arr)

# Read the array at timestamp 2
arr <- tiledb_array(
  uri = array_uri,
  query_type = "READ",
  timestamp_end = as.POSIXct(2),
  return_as = "data.frame"
)
cat("Entire array at timestamp 2:\n")
print(arr[])
cat("\n\n")
arr <- tiledb_array_close(arr)

# Read the array at timestamp range [2,2]
arr <- tiledb_array(
  uri = array_uri,
  query_type = "READ",
  timestamp_start = as.POSIXct(2),
  timestamp_end = as.POSIXct(2),
  return_as = "data.frame"
)
cat("Entire array at timestamp range [2,2]:\n")
print(arr[])
cat("\n\n")
arr <- tiledb_array_close(arr)

Entire array:
  d1 d2   a
1  0  1   1
2  1  1 100
3  2  0   4
4  3  1   6
5  0  3   2
6  1  3   3
7  2  2   5


Entire array at timestamp 1:
  d1 d2 a
1  0  1 1
2  2  0 4
3  3  1 6
4  0  3 2
5  1  3 3
6  2  2 5


Entire array at timestamp 2:
  d1 d2   a
1  0  1   1
2  1  1 100
3  2  0   4
4  3  1   6
5  0  3   2
6  1  3   3
7  2  2   5


Entire array at timestamp range [2,2]:
  d1 d2   a
1  1  1 100

Clean up in the end by deleting the array.

Python
R

# Delete the array
if os.path.exists(array_uri):
    shutil.rmtree(array_uri)

if (file.exists(array_uri)) {
  unlink(array_uri, recursive = TRUE)
}