Data Structures

Learn about the key data structures in TileDB-SOMA in this page.

This section provides a high-level overview of the key data structures in TileDB-SOMA that encapsulate both in-memory representations and persistent storage formats on disk.

TileDB-SOMA is designed to manage large-scale bulk and single-cell omics data efficiently. Central to its functionality are several foundational data structures that facilitate the storage, query, and management of complex datasets. These data structures are essential for handling data both in-memory and on disk.

Foundational data structures

The foundational types represent the core data structures used to store and index data. They are intended to be general-purpose, and to serve as building blocks for the composed data structures, which codify domain-specific use cases (e.g., single-cell experimental datasets).

SOMACollection

A SOMACollection is a container built on TileDB Groups that allows you to organize and manage multiple SOMA objects in a structured way. It is a string-keyed map of values, where each value can be any SOMA foundational or composed data structure. You can think of it as a folder that can store different types of data objects, each identified by a unique name.

Each value in a SOMACollection is referred to by a URI (Uniform Resource Identifier). These URIs can be:

  • Absolute These are complete paths to the data objects and remain unchanged if the collection is moved.
  • Relative: These paths are relative to the collection’s location. If the collection is moved, the relative paths are updated accordingly.

SOMADataFrame

A SOMADataFrame is a multi-column table similar to a spreadsheet. It allows you to store and manipulate tabular data with a user-defined schema, which specifies the columns and their data types. Each SOMADataFrame must have a column called soma_joinid that uniquely identifies each row, acting as a key for linking with other data objects.

Key features

  • User-defined schema: Define the structure of your data with specific column names and types.
  • Integration with other tools: Easily convert between SOMADataFrames and data structures like a pandas DataFrame in Python or a data.frame in R.

Default schema and data types

  • soma_joinid: int64 - A unique identifier for each row.
  • Column data types: User-defined, but can include various types like int32, float32, string, etc.

SOMASparseNDArray

SOMASparseNDArrays are used to store sparse, multi-dimensional arrays. This structure allows efficient storage and retrieval of the non-zero elements, saving space and computational resources.

Default schema and data types

  • Dimensions (soma_dim_N): int64 - Zero-based integer indexing for each dimension.
  • Elements (soma_data): User-defined primitive type, such as int64, float32, etc.

Composed data structures

Composed data structures combine the foundational types to support specific use cases, such as omics data. These structures are designed to encapsulate the complexity of real-world data and provide a unified interface for managing and analyzing it.

SOMAExperiment

A SOMAExperiment is a specialized SOMACollection used to represent multi-modal data from a set of observations. It contains two pre-defined components:

  • obs: A SOMADataFrame containing primary annotations on the observation axis (e.g., cells). The soma_joinid column in this data frame defines the observation index domain.
  • ms: A SOMACollection of SOMAMeasurement objects, each representing a distinct measurement layer.

SOMAMeasurement

A SOMAMeasurement is another specialized SOMACollection used to represent the measurements collected on a single set of variables (e.g., transcripts), along with any results derived from these measurements. It contains the following pre-defined fields:

  • var: A SOMADataFrame containing annotations for the set of variables measured in this modality. The soma_joinid column in this data frame defines the unique identifier for each variable (e.g., gene).
  • X: A SOMACollection of SOMASparseNDArrays containing the measured values, indexed by observation and variable IDs.
  • obsm: A SOMACollection of SOMASparseNDArrays with additional annotations for observations, indexed by obsid.
  • varm: A SOMACollection of SOMASparseNDArrays with additional annotations for variables, indexed by varid.
  • obsp: A SOMACollection of SOMASparseNDArrays with pairwise annotations for observations, indexed by [obsid_1, obsid_2].
  • varp: A SOMACollection of SOMASparseNDArrays with pairwise annotations for variables, indexed by [varid_1, varid_2].

Conclusion

By understanding these foundational and composed data structures, you can leverage TileDB-SOMA to efficiently manage and analyze large-scale single-cell and spatial omics datasets. These tools provide the flexibility and efficiency needed for cutting-edge research, helping you to organize, query, and interpret your data with ease.