Amazon S3

arrays

tutorials

amazon s3

configuration

backends

object stores

Learn how to integrate Amazon S3 with TileDB.

This section covers all you need to know about how to configure and use TileDB on Amazon S3.

Quickstart

Assuming you have installed TileDB and you are already set up on AWS, the quickest way to get started with TileDB on Amazon S3 is by going through the Tutorials: Basic S3 Example with Arrays section.

After configuring TileDB to work with Amazon S3 (more details on that below), your TileDB programs will function properly without any API change! Instead of using local file system paths for referencing files (e.g. arrays, groups, VFS files) use must format your URIs to start with s3://. For instance, if you wish to create (and subsequently write/read) an array on Amazon S3, you use URI s3://<your-bucket>/<your-array-name> for the array name.

AWS setup

First, you need to set up an AWS account and generate access keys.

Create a new AWS account.
Visit the AWS console and sign in.
On the AWS console, select the Services drop-down menu and select Storage -> S3. You can create S3 buckets there.
On the AWS console, select the Services drop-down menu and select Security, Identity & Compliance -> IAM.
Select Users from the left-hand side menu, and then select the Add User button. Provide the email or username of the user you wish to add, select the Programmatic Access checkbox and select Next: Permissions.
Select the Attach existing policies directly button, search for the S3-related policies and add the policy of your choice (e.g., full-access, read-only, etc.). Select Next and then Create User. Using TileDB with an existing bucket requires at least the following S3 permissions:
```
Copy
s3:ListBucket
s3:GetObject
s3:PutObject
s3:ListBucketMultipartUploads
s3:AbortMultipartUpload
s3:ListMultipartUploadParts
s3:DeleteObject
```
Upon successful creation, the next page will show the user along with two keys: Access key ID and Secret access key. Write down both these keys.

AWS security credentials

AWS supports many ways to access its resources. You can either request access with long-term access credentials (for example, Access Keys) or temporary ones by using the AWS Security Token Service.

Access keys

Access keys are long-term credentials for an IAM user or the AWS account root user. To be able to access AWS resource this way, you need to follow the next steps.

Export these keys to your environment from a console:

export AWS_ACCESS_KEY_ID=<your-access-key-id>
export AWS_SECRET_ACCESS_KEY=<your-secret-access-key>

$env:AWS_ACCESS_KEY_ID = "<your-access-key-id>"
$env:AWS_SECRET_ACCESS_KEY = "<your-secret-access-key>"

set AWS_ACCESS_KEY_ID=<your-access-key-id>
set AWS_SECRET_ACCESS_KEY=<your-secret-access-key>

Or, set the following keys in a configuration object (see Configuration):

Parameter	Default value
`"vfs.s3.aws_access_key_id"`	`""`
`"vfs.s3.aws_secret_access_key"`	`""`

Session tokens

TileDB (version 1.8+) supports authentication with temporary credentials from the AWS Session Token. This method of acquiring temporary credentials is preferred in case you want to maintain permissions solely within your organization.

Parameter	Values
`"vfs.s3.aws_session_token"`	session token corresponding to the configured key/secret pair

Assume role

TileDB (version 2.1+) supports authentication with temporary credentials from the AWS AssumeRole API. If you prefer to maintain permissions within AWS, this method’s base permissions for the temporary credentials will be derived from the policy on a role. You can use them to access AWS resources to which you might not normally have access. In this case, you will need to configure the following parameters:

Parameter	Values
`"vfs.s3.aws_role_arn"`	Required - The Amazon Resource Name (ARN) of the role to assume
`"vfs.s3.aws_session_name"`	Optional - An identifier for the assumed role session
`"vfs.s3.aws_external_id"`	Optional - A unique identifier that might be required when you assume a role in another account
`"vfs.s3.aws_load_freq"`	Optional - The duration, in minutes, of the role session

Note

Using this method, the IAM user credentials used by your proxy server only requires the ability to call sts:AssumeRole. You must also create a new role and attach a trust policy to it, which isn’t the case with the Session Tokens approach.

Now you are ready to start writing TileDB programs! When creating a TileDB context or a VFS object, you need to set up a configuration object with the following parameters for Amazon S3.

Parameter	Default value
`"vfs.s3.scheme"`	`"https"`
`"vfs.s3.region"`	`"us-east-1"`
`"vfs.s3.endpoint_override"`	`""`
`"vfs.s3.use_virtual_addressing"`	`"true"`

Note

The above configuration parameters are currently set as shown in TileDB by default. However, you should always check whether the default values are the desired ones for your application.

Physical organization

So far, you learned that TileDB stores arrays and groups as directories. S3 has no concept of a directory, similar to other object stores. However, S3 uses character / in the object URIs which allows the same conceptual organization as a directory hierarchy in local storage. At a physical level, TileDB stores on S3 all the files it would create locally as objects. For instance, for array s3://bucket/path/to/array, TileDB creates array schema object s3://bucket/path/to/array/schema/__<timestamp>_<timestamp>_<uuid>, along with other files and objects. Since S3 has no concept of a directory, nothing distinctive persists on S3 for directories (for example, s3://bucket/path/to/array/meta/ doesn’t exist as an object).

With the AWS CLI, you can sync (that is, download) the S3 objects having a common URI prefix to local storage, organizing them into a directory hierarchy based on the use of / in the object URIs. You can clone TileDB arrays or entire groups locally from S3 by using the aws s3 sync command. For instance, given an array my_array you created and wrote on an S3 bucket my_bucket, you can clone it locally to an array my_local_array with the following command from your console:

aws s3 sync s3://my_bucket/my_array my_local_array

After downloading an array locally, your TileDB program will function properly by changing the array name from s3://my_bucket/my_array to my_local_array, without any other modification.

Performance

TileDB writes the various fragment files as append-only objects using the multi-part upload API of the AWS C++ SDK. In addition to enabling appends, this API renders the TileDB writes to S3 particularly amenable to optimizations via parallelization. Since TileDB updates arrays only by writing (appending to) new files (i.e., it never updates a file in-place), TileDB does not need to download entire objects, update them, and re-upload them to S3. This leads to excellent write performance.

TileDB reads utilize the range GET request API of the AWS SDK, which retrieves only the requested (contiguous) bytes from a file/object, rather than downloading the entire file from the cloud. This results in extremely fast subarray reads, especially because of the array tiling. Recall that a tile (which groups cell values that are stored contiguously in the file) is the atomic unit of I/O. The range GET API enables reading each tile from S3 in a single request. Finally, TileDB performs all reads in parallel using multiple threads, which is a tunable configuration parameter.

Advanced

This section includes more advanced settings regarding using a proxy server, as well as logging.

Proxy server settings

The AWS backend supports several settings for proxy servers:

Parameter	Default value	Description
`"vfs.s3.proxy_host"`	`""`	The S3 proxy host.
`"vfs.s3.proxy_port"`	`"0"`	The S3 proxy port.
`"vfs.s3.proxy_scheme"`	`"https"`	The S3 proxy scheme.
`"vfs.s3.proxy_username"`	`""`	The S3 proxy username.
`"vfs.s3.proxy_password"`	`""`	The S3 proxy password.

Note

It is necessary to override "vfs.s3.proxy_scheme" to http for most proxy setups. TileDB 2.0.8 and later uses the default setting, which will be updated in the next release.

Logging

TileDB uses the AWS C++ SDK and cURL for access to S3. The AWS SDK logging level is a process-global setting. The configuration of the most recently constructed context will set the process state. Log files are written to the process working directory.

Parameter	Values
`"vfs.s3.logging_level"`	`"[OFF], TRACE, DEBUG"`

Certificate paths

Linux has no universal location for SSL/TLS certificates. While TileDB searches for the default CA store on several major distributions, other systems or custom certificates may require path specification. to SSL/TLS certificate file to be used by cURL for S3 HTTPS encryption. These parameters follow cURL conventions: https://curl.haxx.se/docs/manpage.html

Parameter	Values
`"vfs.s3.ca_file"`	File path
`"vfs.s3.ca_path"`	Directory path

For debugging purposes only, it is possible to disable SSL/TLS certificate verification:

Parameter	Values
`"vfs.s3.verify_ssl"`	`[false], true`