Split VCF

Split multi-level VCF files for compatibility with TileDB-VCF ingestion.

How to run this tutorial

You can run this tutorial in two ways:

1. Locally on your machine.
2. On TileDB Cloud.

However, since TileDB Cloud has a free tier, we strongly recommend that you sign up and run everything there, as that requires no installations or deployment.

TileDB supports the ingestion of single-sample VCF files into TileDB-VCF datasets. Multi-sample or project-level VCF files must be split as a prerequisite for ingestion into TileDB. To accommodate multi-sample VCFs, TileDB offers utilities to split them into individual, single-sample files. The most useful of these utilities is a distributed splitting function to enable simultaneous splitting of numerous samples from a single multi-sample VCF. Once single-sample VCF files are isolated, subsequently ingest them into TileDB datasets or combine the entire process into a broader task graph for standardized processing.

Follow this tutorial to learn how to split multi-sample VCF files stored in public or private S3 buckets.

Setup

First, import the necessary libraries for this tutorial.

Python

from tiledb.cloud.vcf import ls_samples, split_vcf

Next, set some global variables:

• project_vcf: S3 path of bgzipped VCF to be split. No index file is required for splitting.
• dest: S3 directory to write single-sample VCFs.
• namespace: Namespace in which to execute task graph in.
• acn: TileDB access credential name to authenticate task graph to S3.
• config: Parameters for accessing S3, such as region.

More notes concerning these variables:

• Split VCF files are named after the sample name listed in the source multi-sample VCF file. If a VCF file already exists at the S3 URI, it will be overwritten.
• The task graph in tiledb.cloud.vcf.split.split_vcf is authenticated by the specified TileDB access credential name (acn).
• For non-task graph access to S3, either pass credentials to config or set them as environment variables (recommended).

Python

# bgzipped target project VCF file
project_vcf = (
    "s3://tiledb-inc-demo-data/vcf/canine/vcf_split_demo_yorkshire_terrier.vcf.gz"
)
# s3 destination to deposit single-sample VCF files
dest = "s3://tiledb-spencer/projects/canine/split-demo/singles"
# TileDB namespace to execute task graph in
namespace = "spencerseale"
# TileDB access credential to authenticate task graph to read/write to S3
acn = "ss_tiledb_cloud"  # update with your ACN
# tiledb config settings
config = {
    "vfs.s3.region": "us-east-1",  # update to S3 region holding your data
}

Split by sample names

To identify all samples within a multi-sample VCF file for splitting, use tiledb.cloud.vcf.split.ls_samples.

Python

all_samples = ls_samples(vcf_uri=project_vcf, config=config)
all_samples

[2024-11-26 17:28:19,081] [split] [ls_samples] [INFO] Samples Identified:
    >>> YorkshireTerrier01
    >>> YorkshireTerrier02
[2024-11-26 17:28:19,082] [split] [ls_samples] [INFO] 
2 samples aggregated in s3://tiledb-inc-demo-data/vcf/canine/vcf_split_demo_yorkshire_terrier.vcf.gz.

['YorkshireTerrier01', 'YorkshireTerrier02']

This dataset consists of two samples. To split just one sample, use tiledb.cloud.vcf.split.split_vcf and pass its name to the samples: Sequence[str] argument.

Python

# indicate sample to split from VCF
graph_a = split_vcf(
    vcf_uri=project_vcf,
    output_uri=dest,
    namespace=namespace,
    acn=acn,
    config=config,
    samples=[all_samples[0]],
)

[2024-11-26 17:28:42,838] [split] [split_vcf] [INFO] Task Graph submitted - https://cloud.tiledb.com/activity/taskgraphs/spencerseale/db43aa71-fc70-48ce-a6c2-854ac4b0cdda

This launches a single-node task graph in the indicated namespace since only one sample is passed to samples. The resultant single-sample VCF for the indicated sample is saved to the location specified to split_vcf’s output_uri argument.

After the task graph completes, the URI of the isolated VCF file is returned. The file is named based on the sample name provided in tiledb.cloud.vcf.split.split_vcf.

Python

graph_a.end_results()

{UUID('02388050-ecdf-45bc-b238-5c115c9421be'): 's3://tiledb-spencer/projects/canine/split-demo/singles/YorkshireTerrier01.vcf.gz'}

Split all samples

Split all samples in a multi-sample VCF into single-sample VCF files by not specifying a value for samples.

Python

# split all samples
graph_b = split_vcf(
    vcf_uri=project_vcf,
    output_uri=dest,
    namespace=namespace,
    acn=acn,
    config=config,
)

[2024-11-25 16:23:07,242] [split] [split_vcf] [INFO] Task Graph submitted - https://cloud.tiledb.com/activity/taskgraphs/spencerseale/215c8897-ac3b-4bec-a838-31ac2cba14a1

This starts a two-stage task graph in the specified namespace. The first stage identifies all samples in the VCF. The second stage dynamically creates nodes for each sample identified in the first stage. This results in parallel isolation of each sample from the multi-sample VCF, expediting the splitting process.

Next steps

Now that the multi-sample VCF file has been isolated into single-sample VCF files, proceed with ingestion. To learn how to ingest the newly created single-sample VCF files, visit the ingestion tutorial.