Array Metadata

Learn how to consolidate array metadata with this tutorial.

How to run this tutorial

You can run this tutorial in two ways:

Locally on your machine.
On TileDB Cloud.

However, since TileDB Cloud has a free tier, we strongly recommend that you sign up and run everything there, as that requires no installations or deployment.

This tutorial demonstrates how to consolidate array metadata. Before running this tutorial, it is recommended that you read the following sections:

First, import the necessary libraries, set the array URI (i.e., its path, which in this tutorial will be on local storage), and delete any previously created arrays with the same name.

Python
R

# Import necessary libraries
import tiledb
import numpy as np
import shutil
import os.path

# Set array URI
array_uri = os.path.expanduser("~/consolidation_array_meta")

# Delete array if it already exists
if os.path.exists(array_uri):
    shutil.rmtree(array_uri)

# Import necessary libraries
library(tiledb)

# Set array URI
array_uri <- path.expand("~/consolidation_array_meta_r")

# Delete array if it already exists
if (file.exists(array_uri)) {
  unlink(array_uri, recursive = TRUE)
}

Next, create the array by specifying its schema. This example uses a sparse array, but the functionality is similar for dense arrays.

Python
R

# Create the two dimensions
d1 = tiledb.Dim(name="d1", domain=(0, 3), tile=2, dtype=np.int32)
d2 = tiledb.Dim(name="d2", domain=(0, 3), tile=2, dtype=np.int32)

# Create a domain using the two dimensions
dom = tiledb.Domain(d1, d2)

# Create an attribute
a = tiledb.Attr(name="a", dtype=np.int32)

# Create the array schema with `sparse=True`.
sch = tiledb.ArraySchema(domain=dom, sparse=True, attrs=[a])

# Create the array on disk (it will initially be empty)
tiledb.Array.create(array_uri, sch)

# Create the two dimensions
d1 <- tiledb_dim("d1", c(0L, 3L), 2L, "INT32")
d2 <- tiledb_dim("d2", c(0L, 3L), 2L, "INT32")

# Create a domain using the two dimensions
dom <- tiledb_domain(dims = c(d1, d2))

# Create an attribute
a <- tiledb_attr("a", type = "INT32")

# Create the array schema, setting `sparse = TRUE`
sch <- tiledb_array_schema(dom, a, sparse = TRUE)

# Create the array on disk (it will initially be empty)
arr <- tiledb_array_create(array_uri, sch)

Write some metadata to the array.

Python
R

# Write some metadata
with tiledb.open(array_uri, "w") as A:
    A.meta["a"] = 1

arr <- tiledb_array(
  array_uri,
  query_type = "WRITE",
  keep_open = TRUE,
  return_as = "data.frame"
)

tiledb_put_metadata(arr, "a", 1L)

Perform a second write of metadata, so that two fragments are generated.

Python
R

# Write some metadata
with tiledb.open(array_uri, "w") as A:
    A.meta["b"] = 2

tiledb_put_metadata(arr, "b", 2L)

arr <- tiledb_array_close(arr)

The array lives in the folder specified by array_uri. The directory __meta has two files, prefixed by _t1_t1 and _t2_t2, where t1 and t2 are the timestamps of the two write operations you performed.

/Users/stavrospapadopoulos/consolidation_array_meta
├── __commits
├── __fragment_meta
├── __fragments
├── __labels
├── __meta
│   ├── __1716155166708_1716155166708_07ba4794b90389abeced2eb79bcc3f18
│   └── __1716155167547_1716155167547_3e033f2bebf1067a2da8306033b7867d
└── __schema
    ├── __1716155166326_1716155166326_00000002eb7e47150fb1c1656ea9be15
    └── __enumerations

8 directories, 3 files

Read the array data. Observe that the metadata from both writes are present in the result, as expected.

Python
R

# Read all metadata
with tiledb.open(array_uri, "r") as A:
    for key, value in A.meta.items():
        print(f"{key}: {value}")

a: 1
b: 2

arr <- tiledb_array_open(arr, type = "READ")

print(tiledb_get_all_metadata(arr))

arr <- tiledb_array_close(arr)

a:  1
b:  2

Now consolidate the array, passing array_meta as the value of configuration parameter sm.consolidation.mode.

Python
R

# Consolidate
config = tiledb.Config({"sm.consolidation.mode": "array_meta"})
tiledb.consolidate(array_uri, config=config)

# Consolidate
cfg <- tiledb_config(
  config = c(
    "sm.consolidation.mode" = "array_meta"
  )
)

array_consolidate(array_uri, cfg = cfg)

Inspect the file hierarchy of the array again. Observe that TileDB created two new files inside directory __meta. Those are now prefixed by _t1_t2.

/Users/stavrospapadopoulos/consolidation_array_meta
├── __commits
├── __fragment_meta
├── __fragments
├── __labels
├── __meta
│   ├── __1716155166708_1716155166708_07ba4794b90389abeced2eb79bcc3f18
│   ├── __1716155166708_1716155167547_557b435830a8814f00f173f9bc32de3c
│   ├── __1716155166708_1716155167547_557b435830a8814f00f173f9bc32de3c.vac
│   └── __1716155167547_1716155167547_3e033f2bebf1067a2da8306033b7867d
└── __schema
    ├── __1716155166326_1716155166326_00000002eb7e47150fb1c1656ea9be15
    └── __enumerations

8 directories, 5 files

Read the array metadata again and observe that nothing changed. The same results are present there. Consolidation (in combination with vacuuming), helps significantly boost performance in the presence of multiple write operations. For more details, visit the Performance: Tuning Writes section.

Python
R

# Read all metadata
with tiledb.open(array_uri, "r") as A:
    for key, value in A.meta.items():
        print(f"{key}: {value}")

a: 1
b: 2

arr <- tiledb_array_open(arr, type = "READ")

print(tiledb_get_all_metadata(arr))

arr <- tiledb_array_close(arr)

a:  1
b:  2

Clean up in the end by deleting the array.

Python
R

# Delete the array
if os.path.exists(array_uri):
    shutil.rmtree(array_uri)

# Delete the array
if (file.exists(array_uri)) {
  unlink(array_uri, recursive = TRUE)
}

Note

For more advanced configuration for consolidation, visit the Key Concepts: Consolidation section.