Arrays Quickstart

arrays

quickstart

tutorials

python

reads

writes

If you’re starting your journey into the world of multi-dimensional arrays, start with this Quickstart guide.

How to run this tutorial

You can run this tutorial in two ways:

Locally on your machine.
On TileDB Cloud.

However, since TileDB Cloud has a free tier, we strongly recommend that you sign up and run everything there, as that requires no installations or deployment.

This section explains how to get started quickly on your local machine or TileDB Cloud. It also covers two basic examples, one on dense arrays and one on sparse arrays. Do not worry at all if some concepts feel foreign to you. You will uncover all the array secrets and make you a power user as you go through the Foundation and Tutorials sections.

Getting started

Local installation

You can choose your preferred API to install from the tabs below. Although these tutorials are in multiple languages, Python is the most common one and the easiest to start with. You can always visit the API Reference section for the detailed usage of all supported language APIs.

Conda
PyPI

conda install -c conda-forge tiledb-py

Note

Conda will install pre-built TileDB-Py and TileDB core binaries for Windows, macOS, or Linux.

pip install tiledb

install.packages("tiledb")
library(tiledb)   # displays versions used

Installing TileDB-R on macOS

If you want to install TileDB-R on macOS, you must also install XQuartz.

Conda
Binaries

conda install -c conda-forge tiledb

You can download the release binaries from the TileDB-Inc/TileDB Releases page on GitHub.

Use the TileDB.CSharp NuGet package to create a .NET console application with the .NET CLI. The TileDB NuGet package is currently compatible with .NET 5 and later.

# Create a new .NET solution
dotnet new sln -o TileDB-Project
cd TileDB-Project
# Create a new console project using .NET CLI
dotnet new console -o ConsoleApp
dotnet sln add ConsoleApp/ConsoleApp.csproj

# Add TileDB.CSharp NuGet package to the project
dotnet add ConsoleApp/ConsoleApp.csproj package TileDB.CSharp

After running the commands above, TileDB-Project/ConsoleApp/ConsoleApp.csproj will have the following configuration, providing access to the TileDB-CSharp API in the project:

Warning

You must declare a runtime identifier (RID). Otherwise, the native TileDB Embedded binaries will not be imported, and the app might fail at runtime.

To learn more about RIDs and see a list of known RIDs, review .NET Runtime Identifier

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net8.0</TargetFramework>
    <ImplicitUsings>enable</ImplicitUsings>
    <!-- The following RID corresponds to a 64-bit Windows installation: -->
    <RuntimeIdentifier>win-x64</RuntimeIdentifier>
    <Nullable>enable</Nullable>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="TileDB.CSharp" Version="5.8.0" />
  </ItemGroup>

</Project>

To test your project, edit TileDB-Project/ConsoleApp/Program.cs and obtain the version of TileDB core currently in use by TileDB.CSharp. For more examples using the C# API, see the TileDB-Inc/TileDB-CSharp repository on GitHub.

using System;
using TileDB.CSharp;

namespace TileDB_Project
{
    class Program
    {
        static void Main(string[] args)
        {
            var version = CoreUtil.GetCoreLibVersion();
            Console.WriteLine($"TileDB Core version: {version}");
            // TileDB Core version: 2.17.0
        }
    }
}

# Go Get
go get -v github.com/TileDB-Inc/TileDB-Go@v0.23.3

# Go modules
go mod init github.com/<github_username>/<repository_name>

Here is a sample go.mod file:

module github.com/<github_username>/<repository_name>

go 1.25.1

require (
    github.com/TileDB-Inc/TileDB-Go v0.23.3
)

docker pull tiledb/tiledb
docker run -it tiledb/tiledb

TileDB Cloud

Sign up to the free tier of TileDB Cloud and follow the instructions in the Get Started section, where you will learn how to set up your account.

Dense array quickstart

In this tutorial, you will learn how to create, write, and read a small dense array on your local disk.

Setup

First, import the necessary libraries:

Python
R

import os.path
import shutil

import numpy as np
import tiledb

print("TileDB core version: {}".format(tiledb.libtiledb.version()))
print("TileDB-Py version: {}".format(tiledb.version()))

library(tiledb)
print(tiledb_version())

Select an array path and remove the array if it already exists.

Python
R

dense_array = os.path.expanduser("~/dense_array")
if os.path.exists(dense_array):
    shutil.rmtree(dense_array)

dense_array <- path.expand("~/dense_array_r")
if (file.exists(dense_array)) {
  unlink(dense_array, recursive = TRUE)
}

Create array

Create a 2D dense array, with dimensions d1 and d2, each with domain [1,4]. The array will also have a single integer attribute a.

Python
R

# Create the two dimensions
d1 = tiledb.Dim(name="d1", domain=(1, 4), tile=2, dtype=np.int32)
d2 = tiledb.Dim(name="d2", domain=(1, 4), tile=2, dtype=np.int32)

# Create a domain using the two dimensions
dom = tiledb.Domain(d1, d2)

# Create an integer attribute
a = tiledb.Attr(name="a", dtype=np.int32)

# Create the array schema, setting `sparse=False` to indicate a dense array
schema = tiledb.ArraySchema(domain=dom, sparse=False, attrs=[a])

# Create the array on disk (it will initially be empty)
tiledb.Array.create(dense_array, schema)

# Create the two dimensions
d1 <- tiledb_dim("d1", c(1L, 4L), 2L, "INT32")
d2 <- tiledb_dim("d2", c(1L, 4L), 2L, "INT32")

# Create a domain using the two dimensions
dom <- tiledb_domain(dims = c(d1, d2))

# Create an attribute
a <- tiledb_attr("a", type = "INT32")

# Create the array schema, setting `sparse = FALSE` to indicate a dense array
sch <- tiledb_array_schema(dom, a, sparse = FALSE)

# Create the array on disk (it will initially be empty)
arr <- tiledb_array_create(dense_array, sch)

This creates an array folder on your local storage, at the path you specified in dense_array above. The array does not contain any data yet.

Python
R

/Users/stavrospapadopoulos/dense_array
├── __commits
├── __fragment_meta
├── __fragments
├── __labels
├── __meta
└── __schema
    ├── __1715630210421_1715630210421_eca4ec20b1c24b049c8bca8dc36859f3
    └── __enumerations

8 directories, 1 file

/Users/stavrospapadopoulos/dense_array
├── __commits
├── __fragment_meta
├── __fragments
├── __labels
├── __meta
└── __schema
    ├── __1715630210421_1715630210421_eca4ec20b1c24b049c8bca8dc36859f3
    └── __enumerations

8 directories, 1 file

You can inspect the array schema as follows.

Python
R

# Read the array schema
schema = tiledb.ArraySchema.load(dense_array)
print(schema)

ArraySchema(
  domain=Domain(*[
    Dim(name='d1', domain=(1, 4), tile=2, dtype='int32', filters=FilterList([ZstdFilter(level=-1), ])),
    Dim(name='d2', domain=(1, 4), tile=2, dtype='int32', filters=FilterList([ZstdFilter(level=-1), ])),
  ]),
  attrs=[
    Attr(name='a', dtype='int32', var=False, nullable=False, enum_label=None),
  ],
  cell_order='row-major',
  tile_order='row-major',
  sparse=False,
)

# Read the array schema
sch <- schema(dense_array)
print(sch)

tiledb_array_schema(
    domain=tiledb_domain(c(
        tiledb_dim(name="d1", domain=c(1L,4L), tile=2L, type="INT32", filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1)))),
        tiledb_dim(name="d2", domain=c(1L,4L), tile=2L, type="INT32", filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1))))
    )),
    attrs=c(
        tiledb_attr(name="a", type="INT32", ncells=1, nullable=FALSE)
    ),
    cell_order="COL_MAJOR", tile_order="COL_MAJOR", capacity=10000, sparse=FALSE, allows_dups=FALSE,
    coords_filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1))),
    offsets_filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1))),
    validity_filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("RLE"),"COMPRESSION_LEVEL",-1)))
)

Write array

Next, write some data to the TileDB array, using a 2D input array:

Python
R

# Prepare some data in a numpy array
data = np.array(
    [[1, 2, 3, 4], [5, 6, 7, 8], [9, 10, 11, 12], [13, 14, 15, 16]], dtype=np.int32
)

# Open the array in write mode and write to the whole array domain
with tiledb.open(dense_array, "w") as A:
    A[:] = data

# Prepare some data in an array
data <- t(array(1:16, dim = c(4, 4)))

# Open the array for writing and write data to the array
arr <- tiledb_array(uri = dense_array, query_type = "WRITE", return_as = "data.frame")
arr[] <- data

# Close the array
arr <- tiledb_array_close(arr)

Read array

You can read the array as follows:

Python
R

# Open the array in read mode
A = tiledb.open(dense_array, "r")

# Open the array in read mode
arr <- tiledb_array_open(arr, type = "READ")

Python
R

# Read the whole array
print(A[:])  # dictionary of 2D numpy arrays, one for each attribute
print(A[:]["a"])  # numpy array

OrderedDict([('a', array([[ 1,  2,  3,  4],
       [ 5,  6,  7,  8],
       [ 9, 10, 11, 12],
       [13, 14, 15, 16]], dtype=int32))])
[[ 1  2  3  4]
 [ 5  6  7  8]
 [ 9 10 11 12]
 [13 14 15 16]]

# Read the whole array
print(arr[])

# Return the attribute a
print(arr[]["a"])

   d1 d2  a
1   1  1  1
2   2  1  5
3   3  1  9
4   4  1 13
5   1  2  2
6   2  2  6
7   3  2 10
8   4  2 14
9   1  3  3
10  2  3  7
11  3  3 11
12  4  3 15
13  1  4  4
14  2  4  8
15  3  4 12
16  4  4 16
    a
1   1
2   5
3   9
4  13
5   2
6   6
7  10
8  14
9   3
10  7
11 11
12 15
13  4
14  8
15 12
16 16

You can also efficiently slice a portion of the array (very useful when the arrays are too big to fit in main memory):

Python
R

print(A[1:3, 1:2]["a"])

[[1]
 [5]]

print(arr[1:2, 2:4]["a"])

You can even slice a multi-range subarray (note that multi_index uses closed ranges, unlike Python half-open intervals).

Python
R

print(A.multi_index[[slice(1, 2), 4], slice(1, 3)]["a"])

[[ 1  2  3]
 [ 5  6  7]
 [13 14 15]]

# Define selected ranges
selected_ranges <- list(
  d1 = matrix(c(
    1, 2,
    4, 4
  ), 2, 2, byrow = TRUE),
  d2 = cbind(1, 3)
)

# Open the array using the selected ranges
arr <- tiledb_array(
  dense_array,
  selected_ranges = selected_ranges,
  return_as = "data.frame"
)
print(arr[])

Remember to always close the array when you are done with it.

Python
R

A.close()

arr <- tiledb_array_close(arr)

Congratulations! You successfully completed your first dense array tutorial!

Sparse array quickstart

In this tutorial, you will learn how to create, write, and read a small sparse array on your local disk.

Setup

First, import the necessary libraries:

Python
R

import os.path
import shutil

import numpy as np
import tiledb

print("TileDB core version: {}".format(tiledb.libtiledb.version()))
print("TileDB-Py version: {}".format(tiledb.version()))

library(tiledb)
print(tiledb_version())

Select an array path and remove the array if it already exists.

Python
R

sparse_array = os.path.expanduser("~/sparse_array")
if os.path.exists(sparse_array):
    shutil.rmtree(sparse_array)

sparse_array <- path.expand("~/sparse_array_r")
if (file.exists(sparse_array)) {
  unlink(sparse_array, recursive = TRUE)
}

Create array

Create a 2D sparse array, with dimensions d1 and d2, each with domain [0,3]. The array will also have a single integer attribute a.

Python
R

# Create the two dimensions
d1 = tiledb.Dim(name="d1", domain=(0, 3), tile=2, dtype=np.int32)
d2 = tiledb.Dim(name="d2", domain=(0, 3), tile=2, dtype=np.int32)

# Create a domain using the two dimensions
dom = tiledb.Domain(d1, d2)

# Create an integer attribute
a = tiledb.Attr(name="a", dtype=np.int32)

# Create the array schema, setting `sparse=True` to indicate a sparse array
schema = tiledb.ArraySchema(domain=dom, sparse=True, attrs=[a])

# Create the array on disk (it will initially be empty)
tiledb.Array.create(sparse_array, schema)

# Create the two dimensions
d1 <- tiledb_dim("d1", c(0L, 3L), 2L, "INT32")
d2 <- tiledb_dim("d2", c(0L, 3L), 2L, "INT32")

# Create a domain using the two dimensions
dom <- tiledb_domain(dims = c(d1, d2))

# Create an attribute
a <- tiledb_attr("a", type = "INT32")

# Create the array schema with `sparse = TRUE`
sch <- tiledb_array_schema(dom, a, sparse = TRUE)

# Create the array on disk (it will initially be empty)
arr <- tiledb_array_create(sparse_array, sch)

This creates an array folder on your local storage, at the path you specified in sparse_array above. The array does not contain any data yet.

Python
R

/Users/stavrospapadopoulos/sparse_array
├── __commits
├── __fragment_meta
├── __fragments
├── __labels
├── __meta
└── __schema
    ├── __1714261379869_1714261379869_00000002a6ed8ee8e935087ed6ccd4b2
    └── __enumerations

8 directories, 1 file

/Users/stavrospapadopoulos/sparse_array
├── __commits
├── __fragment_meta
├── __fragments
├── __labels
├── __meta
└── __schema
    ├── __1714261379869_1714261379869_00000002a6ed8ee8e935087ed6ccd4b2
    └── __enumerations

8 directories, 1 file

You can inspect the array schema as follows.

Python
R

# Read the array schema
schema = tiledb.ArraySchema.load(sparse_array)
print(schema)

ArraySchema(
  domain=Domain(*[
    Dim(name='d1', domain=(0, 3), tile=2, dtype='int32', filters=FilterList([ZstdFilter(level=-1), ])),
    Dim(name='d2', domain=(0, 3), tile=2, dtype='int32', filters=FilterList([ZstdFilter(level=-1), ])),
  ]),
  attrs=[
    Attr(name='a', dtype='int32', var=False, nullable=False, enum_label=None),
  ],
  cell_order='row-major',
  tile_order='row-major',
  capacity=10000,
  sparse=True,
  allows_duplicates=False,
)

# Read the array schema
sch <- schema(sparse_array)
print(sch)

tiledb_array_schema(
    domain=tiledb_domain(c(
        tiledb_dim(name="d1", domain=c(0L,3L), tile=2L, type="INT32", filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1)))),
        tiledb_dim(name="d2", domain=c(0L,3L), tile=2L, type="INT32", filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1))))
    )),
    attrs=c(
        tiledb_attr(name="a", type="INT32", ncells=1, nullable=FALSE)
    ),
    cell_order="COL_MAJOR", tile_order="COL_MAJOR", capacity=10000, sparse=TRUE, allows_dups=FALSE,
    coords_filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1))),
    offsets_filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("ZSTD"),"COMPRESSION_LEVEL",-1))),
    validity_filter_list=tiledb_filter_list(c(tiledb_filter_set_option(tiledb_filter("RLE"),"COMPRESSION_LEVEL",-1)))
)

Write array

Next, write some data to the array, using a set of 1D NumPy arrays (one for each dimension coordinate and one for the attribute):

Python
R

# Prepare some data in numpy arrays, simulating the COO format
d1_data = np.array([2, 0, 3, 2, 0, 1], dtype=np.int32)
d2_data = np.array([0, 1, 1, 2, 3, 3], dtype=np.int32)
a_data = np.array([4, 1, 6, 5, 2, 3], dtype=np.int32)

# Open the array in write mode and write the data
with tiledb.open(sparse_array, "w") as A:
    A[d1_data, d2_data] = a_data

# Prepare some data in an array
d1_data <- c(2L, 0L, 3L, 2L, 0L, 1L)
d2_data <- c(0L, 1L, 1L, 2L, 3L, 3L)
a_data <- c(4L, 1L, 6L, 5L, 2L, 3L)

# Open the array for writing and write data to the array
arr <- tiledb_array(uri = sparse_array, query_type = "WRITE", return_as = "data.frame")
arr[d1_data, d2_data] <- a_data

# Close the array
invisible(tiledb_array_close(arr))

Read array

You can read the array as follows:

Python
R

# Open the array in read mode
A = tiledb.open(sparse_array, "r")

# Open the array in read mode
invisible(tiledb_array_open(arr, type = "READ"))

Python
R

# Read the whole array
A[:]

OrderedDict([('a', array([1, 2, 3, 4, 6, 5], dtype=int32)),
             ('d1', array([0, 0, 1, 2, 3, 2], dtype=int32)),
             ('d2', array([1, 3, 3, 0, 1, 2], dtype=int32))])

# Read the whole array
print(arr[])

Observe that the result is an ordered dictionary of NumPy arrays: two for the coordinates and one the attribute.

You can also efficiently slice a portion of the array (very useful when the arrays are too big to fit in main memory):

Python
R

# Read an array slice
A[0:2, 0:3]

OrderedDict([('a', array([1], dtype=int32)),
             ('d1', array([0], dtype=int32)),
             ('d2', array([1], dtype=int32))])

print(arr[0:2, 0:3]["a"])

You can even slice a multi-range subarray (note that multi_index uses closed ranges).

Python
R

# Read a multi-range slice
A.multi_index[[slice(0, 1), 3], slice(0, 2)]

OrderedDict([('d1', array([0, 3], dtype=int32)),
             ('d2', array([1, 1], dtype=int32)),
             ('a', array([1, 6], dtype=int32))])

# Define selected ranges
selected_ranges <- list(
  d1 = matrix(c(
    0, 1,
    3, 3
  ), 2, 2, byrow = TRUE),
  d2 = cbind(0, 2)
)

# Open the array using the selected ranges
arr <- tiledb_array(
  sparse_array,
  selected_ranges = selected_ranges,
  return_as = "data.frame"
)
print(arr[])

  d1 d2 a
1  0  1 1
2  3  1 6

Remember to always close the array when you are done with it.

Python
R

A.close()

invisible(tiledb_array_close(arr))

Congratulations! You successfully completed your first sparse array tutorial!