Compression
You can compress attributes, dimensions (in sparse arrays), or variable-length attribute offsets for more optimal storage.
Note
It is strongly recommended to read the following sections before you learn about compression.
TileDB suppports several filters performing generic compression. Compression is specified per attribute (or dimension in sparse arrays, or offsets for variable-length attributes) and applies to each physical data tile. TileDB further partitions each physical data tile into chunks of size typically equal to the L1 cache, which are streamed into the compression process.
The table below summarizes the supported compression filters in TileDB.
| Compressor | Description | Option (type) |
|---|---|---|
TILEDB_FILTER_GZIP |
GZIP | level (np.int32) |
TILEDB_FILTER_ZSTD |
Zstandard | level (np.int32) |
TILEDB_FILTER_LZ4 |
LZ4 | level (np.int32) |
TILEDB_FILTER_RLE |
Run-length encoding | level (np.int32) |
TILEDB_FILTER_BZIP2 |
BZIP2 | level (np.int32) |
TILEDB_FILTER_DOUBLE_DELTA |
Double Delta | None |
| Compressor | Description | Option (type) |
|---|---|---|
tiledb.GzipFilter() |
GZIP | level (np.int32) |
tiledb.ZstdFilter() |
Zstandard | level (np.int32) |
tiledb.Lz4Filter() |
LZ4 | level (np.int32) |
tiledb.RleFilter() |
Run-length encoding | level (np.int32) |
tiledb.Bzip2Filter() |
BZIP2 | level (np.int32) |
tiledb.DoubleDeltaFilter() |
Double Delta | None |
| Compressor | Description | Option (type) |
|---|---|---|
"GZIP" |
GZIP | level (np.int32) |
"ZSTD" |
Zstandard | level (np.int32) |
"LZ4" |
LZ4 | level (np.int32) |
"RLE" |
Run-length encoding | level (np.int32) |
"BZIP2" |
BZIP2 | level (np.int32) |
"DOUBLE_DELTA" |
Double Delta | None |
| Compressor | Description | Option (type) |
|---|---|---|
"GzipFilter" |
GZIP | level (np.int32) |
"ZstdFilter" |
Zstandard | level (np.int32) |
"LZ4Filter" |
LZ4 | level (np.int32) |
"RleFilter" |
Run-length encoding | level (np.int32) |
"Bzip2Filter" |
BZIP2 | level (np.int32) |
"DoubleDeltaFilter" |
Double Delta | None |
| Compressor | Description | Option (type) |
|---|---|---|
Gzip |
GZIP | level (np.int32) |
Zstandard |
Zstandard | level (np.int32) |
Lz4 |
LZ4 | level (np.int32) |
RunLengthEncoding |
Run-length encoding | level (np.int32) |
Bzip2 |
BZIP2 | level (np.int32) |
DoubleDelta |
Double Delta | None |
The default for the Option value is -1.