Add Tile Filters to Arrays

arrays

tutorials

python

tile filters

Tile filters allow you to compress and encrypt attribute and dimension data for your arrays.

How to run this tutorial

You can run this tutorial in two ways:

Locally on your machine.
On TileDB Cloud.

However, since TileDB Cloud has a free tier, we strongly recommend that you sign up and run everything there, as that requires no installations or deployment.

This tutorial describes how to use tile filters for attributes and dimensions. Two types of unique filters exist in TileDB: compression and encryption. For tutorials on each, visit the Compress Array Data and Encrypt an Array with an AES-256-GCM Key tutorials. This tutorial is very similar to the one on compression. For more information on all the supported tiled filters, visit the Key Concepts: Tile Filters section.

Note

In this context, a tile filter is a transformation (that is, a processing step) applied to the data in a tile before TileDB writes the tile to storage. TileDB applies the filter to each tile individually, and the transformation is reversible.

First, import the necessary libraries, set the array URI (that is, its path, which in this tutorial will be on local storage), and delete any previously created arrays with the same name.

Python
R

# Import necessary libraries
import os.path
import shutil

import numpy as np
import tiledb

# Set array URI
array_uri = os.path.expanduser("~/tile_filters")

# Delete array if it already exists
if os.path.exists(array_uri):
    shutil.rmtree(array_uri)

# Import necessary libraries
library(tiledb)

# Set array URI
array_uri <- path.expand("~/tile_filters_r")

# Delete array if it already exists
if (file.exists(array_uri)) {
  unlink(array_uri, recursive = TRUE)
}

Next, create some filters as follows.

Python
R

# Set up some filter lists to pass to dimensions and attributes
# tiledb.FilterList accepts an iterable of zero or more filters
filter_list_1 = tiledb.FilterList(
    [tiledb.PositiveDeltaFilter(window=1000)], chunksize=10000
)
filter_list_2 = tiledb.FilterList([tiledb.BitWidthReductionFilter(window=10)])

# Set up some filter lists to pass to dimensions and attributes
# tiledb_filter_list() accepts a vector of zero or more filters
pd_filter <- tiledb_filter("POSITIVE_DELTA")
pd_filter <- tiledb_filter_set_option(
  pd_filter,
  "POSITIVE_DELTA_MAX_WINDOW",
  1000
)
filter_list_1 <- tiledb_filter_list(pd_filter)
set_max_chunk_size(filter_list_1, 10000)

bwr_filter <- tiledb_filter("BIT_WIDTH_REDUCTION")
bwr_filter <- tiledb_filter_set_option(
  bwr_filter,
  "BIT_WIDTH_MAX_WINDOW",
  10
)
filter_list_2 <- tiledb_filter_list(bwr_filter)

Then create an array and pass the desired filters as arguments in each dimension and attribute.

Python
R

# Create the two dimensions
# The filter list we created above is passed into the `filters` parameter
d1 = tiledb.Dim(name="d1", domain=(0, 3), tile=2, dtype=np.int32, filters=filter_list_1)
d2 = tiledb.Dim(name="d2", domain=(0, 3), tile=2, dtype=np.int32, filters=filter_list_1)

# Create a domain using the two dimensions
dom = tiledb.Domain(d1, d2)

# Create an attribute
# The filter list we created above is passed into the `filters` parameter
a = tiledb.Attr(name="a", dtype=np.int32, filters=filter_list_2)

# Create the array schema with `sparse=True`
sch = tiledb.ArraySchema(domain=dom, sparse=True, attrs=[a])

# Create the array on disk (it will initially be empty)
tiledb.Array.create(array_uri, sch)

# Create the two dimensions
d1 <- tiledb_dim("d1", c(0L, 3L), 2L, "INT32", filter_list = filter_list_1)
d2 <- tiledb_dim("d2", c(0L, 3L), 2L, "INT32", filter_list = filter_list_1)

# Create a domain using the two dimensions
dom <- tiledb_domain(dims = c(d1, d2))

# Create an attribute
a <- tiledb_attr("a", type = "INT32", filter_list = filter_list_2)

# Create the array schema with `sparse = TRUE`
sch <- tiledb_array_schema(dom, a, sparse = TRUE)

# Create the array on disk (it will initially be empty)
arr <- tiledb_array_create(array_uri, sch)

Inspect the schema of the array to see how TileDB set the filters you passed.

Python
R

print(tiledb.ArraySchema.load(array_uri))

ArraySchema(
  domain=Domain(*[
    Dim(name='d1', domain=(0, 3), tile=2, dtype='int32', filters=FilterList([PositiveDeltaFilter(window=1000), ])),
    Dim(name='d2', domain=(0, 3), tile=2, dtype='int32', filters=FilterList([PositiveDeltaFilter(window=1000), ])),
  ]),
  attrs=[
    Attr(name='a', dtype='int32', var=False, nullable=False, enum_label=None, filters=FilterList([BitWidthReductionFilter(window=10), ])),
  ],
  cell_order='row-major',
  tile_order='row-major',
  capacity=10000,
  sparse=True,
  allows_duplicates=False,
)

schema(array_uri)

tiledb_array_schema(
    domain=tiledb_domain(c(
        tiledb_dim(name="d1", domain=c(0L,3L), tile=2L, type="INT32", filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("POSITIVE_DELTA"),"POSITIVE_DELTA_MAX_WINDOW",1000)))),
        tiledb_dim(name="d2", domain=c(0L,3L), tile=2L, type="INT32", filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("POSITIVE_DELTA"),"POSITIVE_DELTA_MAX_WINDOW",1000))))
    )),
    attrs=c(
        tiledb_attr(name="a", type="INT32", ncells=1, nullable=FALSE, filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("BIT_WIDTH_REDUCTION"),"BIT_WIDTH_MAX_WINDOW",10))))
    ),
    cell_order="COL_MAJOR", tile_order="COL_MAJOR", capacity=10000, sparse=TRUE, allows_dups=FALSE,
    coords_filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1))),
    offsets_filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1))),
    validity_filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("RLE"),"COMPRESSION_LEVEL",-1)))
)

For convenience, you can pass the same filters to all dimensions by passing them as an argument to the array schema object.

Python
R

# ... create domain dom
# ... create attribute a
# ... create filter list filter_list_1

# Create the schema setting the coordinates filter list.
# This is applicable only to sparse arrays.
schema = tiledb.ArraySchema(
    domain=dom, sparse=False, attrs=[a], coords_filters=filter_list_1
)

# ... create (or retrieve) array schema sch

# assign filter list to an existing schema
sch <- tiledb_array_schema_set_coords_filter_list(sch, filter_list_1)

# Alternatively create a new schema and set the coordinates filter list
sch <- tiledb_array_schema(dom, a, coords_filter_list = filter_list_1)

You can also set filters for the variable-length attribute and dimension offsets.

Python
R

# Create the schema setting the offsets filter list

# ... create domain dom
# ... create attribute a
# ... create filter list filter_list_2

# Create the schema setting the offsets filter list
schema = tiledb.ArraySchema(
    domain=dom, sparse=False, attrs=[a], offsets_filters=filter_list_2
)

# ... create (or retrieve) array schema sch

# assign filter list to an existing schema
sch <- tiledb_array_schema_set_offsets_filter_list(sch, filter_list_2)

# Alternatively create a new schema and set the offsets filter list
sch <- tiledb_array_schema(dom, a, offsets_filter_list = filter_list_2)

The same applies to the validity vectors for fixed-length or variable-length attributes.

Python
R

# Create the schema setting the offsets filter list

# ... create domain dom
# ... create attribute a
# ... create filter list filter_list_3

# Create the schema setting the offsets filter list
schema = tiledb.ArraySchema(
    domain=dom, sparse=False, attrs=[a], validity_filters=filter_list_1
)

# ... create (or retrieve) array schema sch

# assign filter list to an existing schema
sch <- tiledb_array_schema_set_validity_filter_list(sch, filter_list_1)

# Alternatively create a new schema and set the validity filter list
sch <- tiledb_array_schema(dom, a, validity_filter_list = filter_list_1)

Clean up in the end by deleting the array.

Python
R

# Delete the array
if os.path.exists(array_uri):
    shutil.rmtree(array_uri)

if (file.exists(array_uri)) {
  unlink(array_uri, recursive = TRUE)
}