Chunked Ingestion of Biomedical Images

Learn how to ingest biomedical images in chunks into TileDB.

Biomedical images are often massive, making it sometimes impossible to load an entire image into memory for processing. Chunked ingestion breaks down these large images into smaller, manageable pieces called chunks. This enables efficient processing and offers the following benefits:

• Reduced memory footprint: Only the chunk being processed needs to live in memory.
• Parallel processing: Different chunks can be processed concurrently, speeding up computations.
• Efficient storage: Chunks can be stored and retrieved independently, optimizing data management.

TileDB offers chunked ingestion capabilities to all the supported formats. In this tutorial, you will learn how to enable chunked ingestion while ingesting different formats into TileDB.

Note

The OpenSlideConverter may not be 100% lossless with chunked=True for levels greater than the highest resolution level, even though the converted images look visually identical to the original ones.

For chunked ingestion, you’ll follow similar steps as in the Biomedical Imaging Quickstart with only one slight change: the addition of an extra argument, chunked=True.

Start by importing the required packages and creating the directory to hold the files used in this tutorial:

Python

import os
import shutil

from tiledb.bioimg import Converters, from_bioimg
from tiledb.bioimg.openslide import TileDBOpenSlide

root_dir = os.path.expanduser("~/tiledb-bioimg-chunked-ingestion")

if os.path.exists(root_dir):
    shutil.rmtree(root_dir)

os.makedirs(root_dir)

Next, fetch the image data and define your ingestion paths.

Python

import requests

url = "https://github.com/libvips/libvips/raw/refs/heads/master/test/test-suite/images/CMU-1-Small-Region.svs"
response = requests.get(url, stream=True)

data_home = os.path.join(root_dir, "data.svs")
data_dest = os.path.join(root_dir, "data.tdb")

with open(data_home, "wb") as out_file:
    shutil.copyfileobj(response.raw, out_file)

Now run the ingestion.

Python

from_bioimg(data_home, data_dest, converter=Converters.OMETIFF, chunked=True)

tiledb.bioimg.converters.ome_tiff.OMETiffConverter

You can read the image data similarly to the Biomedical Imaging Quickstart.

Python

# Construct OpenSlide API wrapper
img = TileDBOpenSlide(data_dest)
region_data = img.read_region(location=(0, 0), level=0, size=(100, 100))
region_data

array([[[221, 182, 221],
        [255, 172, 196],
        [158, 141, 164],
        ...,
        [246, 246, 243],
        [246, 246, 243],
        [246, 246, 243]],

       [[230, 172, 226],
        [255, 189, 212],
        [218, 160, 197],
        ...,
        [246, 246, 243],
        [246, 246, 243],
        [246, 246, 243]],

       [[222, 170, 184],
        [230, 210, 186],
        [231, 189, 189],
        ...,
        [246, 246, 243],
        [246, 246, 243],
        [246, 246, 243]],

       ...,

       [[140,  78, 128],
        [164, 102, 124],
        [195, 138, 136],
        ...,
        [243, 243, 243],
        [243, 243, 243],
        [243, 243, 243]],

       [[122,  75, 120],
        [157,  89, 125],
        [189, 106, 138],
        ...,
        [243, 243, 243],
        [243, 243, 243],
        [243, 243, 243]],

       [[188, 139, 171],
        [184, 107, 171],
        [180,  85, 171],
        ...,
        [243, 243, 243],
        [243, 243, 243],
        [243, 243, 243]]], dtype=uint8)

Visualize the stored data with Pillow:

Python

import PIL

img = PIL.Image.fromarray(region_data)
img

Clean up in the end by removing the data directory.

Python

if os.path.exists(root_dir):
    shutil.rmtree(root_dir)

The TileDB converter streamlines biomedical image ingestion by automatically optimizing the process based on your available resources. It adjusts parameters behind the scenes to maximize performance, efficiently use resources, and minimize memory footprint, ensuring a smooth and efficient ingestion workflow.

TileDB ingestion

To ingest biomedical images with the TileDB API, include the same extra argument in your ingest function call. You can find a full tutorial on ingesting biomedical images into TileDB in the Biomedical Imaging Ingestion page.

This will ensure TileDB correctly processes and stores your changes.