Tuning Writes

Learn how to improve write performance through choosing the appropriate write layout, consolidating fragments, and utilizing TileDB Cloud’s distributed compute.

When tuning the performance during writing data into TileDB arrays, there are a few main factors you need to consider.

Write layout

The most efficient layout to write in is global order, as TileDB won’t need to sort and re-organize the written cells internally. Row-major, column-major and unordered layouts require sorting and, thus, more work for TileDB. In general, the write layout and the sub-domain in which the write occurs must coincide as much as possible with the space tiles and the global order.

Write size

The larger the writes, the better for performance. This is because TileDB can take advantage of parallelism and perform better parallel tile filtering and I/O. Moreover, larger (and more infrequent) writes can lead to fewer fragments, thus affecting read performance.

Fragments

What affects read performance regarding fragments is:

  • The number of fragments.
  • The overlap of fragment non-empty domains.

The fewer and more “spatially disjoint” the fragments are, the better TileDB can prune fragments during reads and achieve better performance. Make sure that each write operation focuses on a separate sub-domain.

Consolidation

The best scenario for maximizing the performance of reads is to have a single fragment after writing. The ways you can get a single fragment when ingesting into your TileDB arrays are:

  1. By performing a single write (which may not be possible in applications where the data is much larger than RAM).
  2. By writing in global order, appending data to your fragments (which may not be possible in applications where the data don’t arrive in global order).
  3. By consolidating your fragments often, which is the most reasonable choice for most applications. However, properly tuning consolidation for an application may be challenging.

Here are a few tips for maximizing the consolidation performance.

  • In dense arrays, perform dense writes in subarrays that align with the space tile boundaries. This will prevent TileDB from filling with empty cell values, which may make finding consolidatable fragments more difficult.

  • Update a dense array by trying to rewrite the same dense subarrays. This helps the preprocessing clean up process, which will try to delete older fully overwritten fragments rapidly without performing consolidation.

  • For sparse arrays, perform writes of roughly equal sizes. This will lead to balanced consolidation.

  • It may be a good idea to invoke consolidation after every write, tuning sm.consolidation.step_min_frags, sm.consolidation.step_max_frags, and sm.consolidation.steps to emulate the way log-structured merge (LSM) trees work. Specifically, choose a reasonable value for sm.consolidation.step_min_frags and sm.consolidation.step_max_frags (for example, 2-20). This will ensure that TileDB consolidates only a small number of fragments per step. Then you can set the number of steps (sm.consolidation.steps) to something large, so that consolidation proceeds recursively until only a single fragment (or very few fragments) remains. If you invoke consolidation after each write, the consolidation cost will be amortized over all ingestion processes in the lifetime of your system. Note that in that case, the consolidation times will vary. Sometimes, you won’t need consolidation at all. Other times, TileDB may perform a few fast consolidation steps (involving a few small fragments). In some situations (although less often), consolidation may take much longer, as it may be consolidating large fragments. Still, this approach leads to a great amortized average ingestion time, resulting in only a few fragments and, hence, fast reads.

  • Read performance is inversely related to fragment size (that is, the larger the fragment, the worse the read performance will be). This is because larger metadata and indexes may have to be fetched in main memory during the reads. For that reason, you can change the sm.consolidation.max_fragment_size parameter to split the result of the consolidation into many fragments that will be smaller in size than the set value in bytes.

Visit the Configuration: Consolidation and Vacuuming doc for information on how to set the configuration parameters mentioned earlier.

Distributed compute

You can maximize performance if you use TileDB Cloud and take advantage of distributing compute. For more information, visit the Key Concepts: Distributed Compute and Tutorials: Distributed Compute sections.