Array Schema

Learn how to view various aspects of a TileDB array schema, such as its members, domain, dimensions, attributes, and filters.
How to run this tutorial

You can run this tutorial in two ways:

  1. Locally on your machine.
  2. On TileDB Cloud.

However, since TileDB Cloud has a free tier, we strongly recommend that you sign up and run everything there, as that requires no installations or deployment.

This tutorial shows how you can inspect the internals of an array schema. For more details, visit the Key Concepts: Array Schema section.

First, import the necessary libraries, set the array URI (that is, its path, which in this tutorial will be on local storage), and delete any previously created arrays with the same name.

# Import necessary libraries
import tiledb
import numpy as np
import shutil
import os.path

# Set array URI
array_uri = os.path.expanduser("~/array_schema")

# Delete array if it already exists
if os.path.exists(array_uri):
    shutil.rmtree(array_uri)
# Import necessary libraries
library(tiledb)

# Set array URI
array_uri <- path.expand("~/array_uri_r")

# Delete array if it already exists
if (file.exists(array_uri)) {
  unlink(array_uri, recursive = TRUE)
}

Next, create an array by specifying its schema. This example focuses on a dense array, but the case of a sparse array is similar.

# Create the two dimensions
d1 = tiledb.Dim(name="d1", domain=(1, 4), tile=2, dtype=np.int32)
d2 = tiledb.Dim(name="d2", domain=(1, 4), tile=2, dtype=np.int32)

# Create a domain using the two dimensions
dom = tiledb.Domain(d1, d2)
# Order of the dimensions matters when slicing subarrays.
# Remember to give priority to more selective dimensions to
# maximize the pruning power during slicing.

# Create an attribute
a = tiledb.Attr(name="a", dtype=np.int32)

# Create the array schema, setting `sparse=False` to indicate a dense array
sch = tiledb.ArraySchema(domain=dom, sparse=False, attrs=[a])

# Create the array on disk (it will initially be empty)
tiledb.Array.create(array_uri, sch)
# Create the two dimensions
d1 <- tiledb_dim("d1", c(1L, 4L), 2L, "INT32")
d2 <- tiledb_dim("d2", c(1L, 4L), 2L, "INT32")

# Create a domain using the two dimensions
dom <- tiledb_domain(dims = c(d1, d2))

# Create an attribute
a <- tiledb_attr("a", type = "INT32")

# Create the array schema, setting `sparse = FALSE` to indicate a dense array
sch <- tiledb_array_schema(dom, a, sparse = FALSE)

# Create the array on disk (it will initially be empty)
arr <- tiledb_array_create(array_uri, sch)

You can load the array schema, either directly or after opening an array.

# Load the array schema directly.
# Optionally pass `key=...` for encrypted arrays
schema = tiledb.ArraySchema.load(array_uri)

# or access the schema from an open array object
A = tiledb.open(array_uri)
print(A.schema)
A.close()
ArraySchema(
  domain=Domain(*[
    Dim(name='d1', domain=(1, 4), tile=2, dtype='int32', filters=FilterList([ZstdFilter(level=-1), ])),
    Dim(name='d2', domain=(1, 4), tile=2, dtype='int32', filters=FilterList([ZstdFilter(level=-1), ])),
  ]),
  attrs=[
    Attr(name='a', dtype='int32', var=False, nullable=False, enum_label=None),
  ],
  cell_order='row-major',
  tile_order='row-major',
  sparse=False,
)

schema <- schema(array_uri)
print(schema)

# You can check the schema for correctness
# TRUE means schema is valid
# FALSE means schema is invalid
schema_check(schema)
tiledb_array_schema(
    domain=tiledb_domain(c(
        tiledb_dim(name="d1", domain=c(1L,4L), tile=2L, type="INT32", filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1)))),
        tiledb_dim(name="d2", domain=c(1L,4L), tile=2L, type="INT32", filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1))))
    )),
    attrs=c(
        tiledb_attr(name="a", type="INT32", ncells=1, nullable=FALSE)
    ),
    cell_order="COL_MAJOR", tile_order="COL_MAJOR", capacity=10000, sparse=FALSE, allows_dups=FALSE,
    coords_filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1))),
    offsets_filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1))),
    validity_filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("RLE"),"COMPRESSION_LEVEL",-1)))
)
TRUE

TileDB offers a variety of accessors for the array schema members.

# Check if array is sparse
print(schema.sparse)

# Get tile capacity
print(schema.capacity)

# Get tile order
print(schema.tile_order)

# Get cell order
print(schema.cell_order)

# Get coordinates filter list
print(schema.coords_filters)

# Get offsets filter list
print(schema.offsets_filters)

# Get validity filter list
print(schema.validity_filters)

# Get the array domain
print(schema.domain)

# Get number of attributes
print(schema.nattr)

# Get attribute by index (0 <= idx < attr_num)
idx = 0
print(schema.attr(idx))

# Check if the named attribute exists
print(schema.has_attr("a"))

# Get attribute by name
print(schema.attr("a"))

# Check if the schema allows duplicates (sparse arrays only)
if schema.sparse:
    print(schema.allows_duplicates)
False
10000
row-major
row-major
FilterList([ZstdFilter(level=-1)])
FilterList([ZstdFilter(level=-1)])
FilterList([RleFilter()])
Domain(Dim(name='d1', domain=(1, 4), tile=2, dtype='int32', filters=FilterList([ZstdFilter(level=-1), ])),
       Dim(name='d2', domain=(1, 4), tile=2, dtype='int32', filters=FilterList([ZstdFilter(level=-1), ])))
1
Attr(name='a', dtype='int32', var=False, nullable=False, enum_label=None)
True
Attr(name='a', dtype='int32', var=False, nullable=False, enum_label=None)
# Check if array is sparse
print(is.sparse(schema))

# Get tile capacity
print(capacity(schema))

# Get tile order
print(tile_order(schema))

# Get cell order
print(cell_order(schema))

# Get coords filter lists associated with the schema
print(filter_list(schema)$coords)

# Get coords filter lists associated with the schema
print(filter_list(schema)$offsets)

# Get coords filter lists associated with the schema
print(filter_list(schema)$validity)

# Get all filter lists associated with the schema
print(filter_list(schema))

# Get the array domain
print(domain(schema))

# Get attributes
print(attrs(schema))

# Get attribute by index (1 <= idx <= attr_num)
print(attrs(schema, idx = 1))

# Check if the named attribute exists
print(has_attribute(schema, "a"))

# Check if the schema allows duplicates (sparse arrays only)
if (is.sparse(schema)) {
  print(allows_dups(schema))
}
[1] FALSE
[1] 10000
[1] "COL_MAJOR"
[1] "COL_MAJOR"
tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1))) 
tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1))) 
tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("RLE"),"COMPRESSION_LEVEL",-1))) 
$coords
tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1))) 

$offsets
tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1))) 

$validity
tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("RLE"),"COMPRESSION_LEVEL",-1))) 

tiledb_domain(c(
        tiledb_dim(name="d1", domain=c(1L,4L), tile=2L, type="INT32", filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1)))),
        tiledb_dim(name="d2", domain=c(1L,4L), tile=2L, type="INT32", filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1))))
    )) 
$a
tiledb_attr(name="a", type="INT32", ncells=1, nullable=FALSE) 

tiledb_attr(name="a", type="INT32", ncells=1, nullable=FALSE) 
[1] TRUE

You can inspect the members of the domain.

# Get the domain data type (i.e., the datatype of all dimensions)
# note: types are automatically converted to NumPy dtypes
print(schema.domain.dtype)

# Get number of dimensions
print(schema.domain.ndim)

# Get dimension by index (0 <= idx < dim_num)
idx = 0
print(schema.domain.dim(idx))

# Get dimension by name
print(schema.domain.dim("d1"))

# Check if domain has named dimension
print(schema.domain.has_dim("d1"))
int32
2
Dim(name='d1', domain=(1, 4), tile=2, dtype='int32', filters=FilterList([ZstdFilter(level=-1), ]))
Dim(name='d1', domain=(1, 4), tile=2, dtype='int32', filters=FilterList([ZstdFilter(level=-1), ]))
True
# Get the domain of the schema
domain <- domain(schema)
print(domain)

# Get the number of dimensions from the domain
print(tiledb_ndim(domain))

# Get dimensions from the domain
dimensions <- dimensions(domain)
print(dimensions)
tiledb_domain(c(
        tiledb_dim(name="d1", domain=c(1L,4L), tile=2L, type="INT32", filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1)))),
        tiledb_dim(name="d2", domain=c(1L,4L), tile=2L, type="INT32", filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1))))
    )) 
[1] 2
[[1]]
tiledb_dim(name="d1", domain=c(1L,4L), tile=2L, type="INT32", filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1)))) 

[[2]]
tiledb_dim(name="d2", domain=c(1L,4L), tile=2L, type="INT32", filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1))))

You can also inspect the members of the dimensions.

# Get a dimension
d1 = schema.domain.dim(0)

# Get dimension name
print(d1.name)

# Get dimension datatype
print(d1.dtype)

# Get dimension domain
print(d1.domain)

# Get tile extent
print(d1.tile)

# Get dimension filter list
print(d1.filters)
d1
int32
(1, 4)
2
FilterList([ZstdFilter(level=-1)])
library(tiledb)
# Get the domain of the schema
domain <- domain(schema)
print(domain)

# Get all dimension and attribute names
print(tiledb_schema_get_names(schema))

# Get all dimension and attribute types
print(tiledb_schema_get_types(schema))

# Get the number of dimensions for the schema
print(tiledb_ndim(schema))

# You can also get the number of dimensions from the domain
print(tiledb_ndim(domain))

# Get a dimension
dim <- dimensions[[1]]

# Get dimension name
print(name(dim))

# Get dimension datatype
print(datatype(dim))

# Get dimension domain
print(domain(dim))

# Get tile extent
print(tile(dim))

# Get dimension filter list
print(filter_list(dim))
tiledb_domain(c(
        tiledb_dim(name="d1", domain=c(1L,4L), tile=2L, type="INT32", filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1)))),
        tiledb_dim(name="d2", domain=c(1L,4L), tile=2L, type="INT32", filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1))))
    )) 
[1] "d1" "d2" "a" 
[1] "INT32" "INT32" "INT32"
[1] 2
[1] 2
[1] "d1"
[1] "INT32"
[1] 1 4
[1] 2
tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1)))

You can also inspect the members of the attributes.

# Get an attribute
attr = schema.attr(0)

# Get attribute name
print(attr.name)

# Get attribute datatype
print(attr.dtype)

# Get filter list
print(attr.filters)

# Check if attribute is variable-length
print(attr.isvar)

# Get number of values per cell
# note: for variable-length attributes, ncells == typemax(np.uint32)
print(attr.ncells)

# Get datatype size (bytes) for this attribute
print(attr.dtype.itemsize)
a
int32
FilterList([])
False
1
4
# Get all dimension and attribute names
print(tiledb_schema_get_names(schema))

# Get all dimension and attribute types
print(tiledb_schema_get_types(schema))

# Get an attribute
attr <- attrs(schema)[[1]]

# Get attribute name
print(name(attr))

# Get attribute datatype
print(datatype(attr))

# Get filter list
print(filter_list(attr))

# Check if attribute is variable-length
print(tiledb_attribute_is_variable_sized(attr))

# Check if attribute has an enumeration
print(tiledb_attribute_has_enumeration(attr))

# Get number of values per cell
print(cell_val_num(attr))

# Get datatype size (bytes) for this attribute
print(tiledb_attribute_get_cell_size(attr))
[1] "d1" "d2" "a" 
[1] "INT32" "INT32" "INT32"
[1] "a"
[1] "INT32"
 
[1] FALSE
[1] FALSE
[1] 1
[1] 4

Finally, you can inspect the members of the filters.

# Get a filter list
filter_list = d1.filters

# Get number of filters
print(len(filter_list))

# Get the maximum tile chunk size
print(filter_list.chunksize)

# Get a filter by index (0 <= idx < num_filters)
idx = 0
filter = filter_list[idx]

# Get filter type
print(filter.__class__)

# Get filter option (depends on the filter: `filter.{option_name}`)
print(filter.level)
1
65536
<class 'tiledb.filter.ZstdFilter'>
-1
# Get a filter list
filter_list <- filter_list(dim)

# Get number of filters
print(length(filter_list))

# Get the maximum tile chunk size
print(max_chunk_size(filter_list))

# Get a filter by index (0 <= idx < num_filters)
idx <- 0
filter <- filter_list[idx]

# Get filter type
print(tiledb_filter_type(filter))

# Get filter option (depends on the filter)
print(tiledb_filter_get_option(filter, "COMPRESSION_LEVEL"))
[1] 1
[1] 65536
[1] "ZSTD"
[1] -1

Clean up in the end by deleting the array.

# Delete the array
if os.path.exists(array_uri):
    shutil.rmtree(array_uri)
if (file.exists(array_uri)) {
  unlink(array_uri, recursive = TRUE)
}
Tip

On TileDB Cloud, you can visually inspect the schema of your array once you locate it under the Assets page. For more information, visit the Catalog: Arrays section.