Why TileDB for Machine Learning
Machine Learning (ML), at its core, starts with data.
This single term embodies the challenge of storing machine learning data in a common, shareable, and efficient data format. These data typically come into many forms, such as images, audio, video, tables, and so on. Data storage and manipulation are integral components of the machine learning workflow and offer the following benefits:
- Data governance
- Ensuring data quality
- Feature engineering
- Scalability
- Efficient data preprocessing
- Model training and evaluation
- Model deployment and monitoring
Different groups are researching what are and should be the essential features, functionalities, and requirements of a storage system that would overcome common challenges in a typical machine learning application. Although this research is ongoing, TileDB satisfies most of the requirements for a successful ML application.
In March 2020, the OpenML community published a blog post, Finding a standard dataset format for machine learning, outlining a list of these requirements, in their quest to find a standard dataset format for machine learning.
Requirements
The blog post outlines the following 17 data format requirements for ML applications.
The blog post outlines the following soft format requirements for ML applications:
- The format should be stable and fully maintained by an active community.
- Parsers in various programming languages, including well-maintained and stable libraries.
- Streaming reads and writes, for easy conversion and memory efficiency.
- Version control, some way to see differences between versions.
- Ideally, there is a way to detect bitflip errors during storage or transmission.
- Ideally, fast reads and writes and efficient storage.
- Ideally, there should be support for storing sparse data.
- Support for storing binary blobs and vectors of different lengths.
- If possible, support for multiple “resources” (for example, collections of files or relational tables).
- Potentially, store some metadata inside the file.
A stable format fully supported by an active community
The TileDB team is 100% committed to supporting and improving upon the open source core of TileDB, and to growing TileDB’s user and developer community.
Parsers in different programming languages
Not just parsers, but instead a single, efficient C++ core implementation, along with numerous language APIs and integrations with popular SQL engines and data science tools.
TileDB offers more than just parsers. It has a single, efficient C++ core implementation, along with many supported language APIs and integrations with popular SQL engines
Streaming reads and writes, for easy conversion and memory efficiency
We firmly believe that no data is unstructured. With TileDB’s multi-dimensional array storage format, you can store many types of data, including images, video, and audio, that don’t align with what’s understood as tabular data. Also, with its use of immutable fragments, TileDB avoids rewriting existing data, which improves performance.
Version control
TileDB supports version control of data through immutable, time-stamped fragments.
Detection of bitflip errors during storage or transmission
TileDB supports a variety of filters that you can apply on a per-data tile basis, including MD5 and SHA256 checksums.
Fast reads and writes and efficient storage
TileDB’s storage engine is optimized for cloud object stores, allowing for fast read and write operations and for efficient storage and retrieval. When users add, update or delete cells in a TileDB array, TileDB generates a fragment with only the applied changes, yielding efficient writes. Through consolidation and vacuuming, you can improve read performance at the cost of granularity for time traveling.
Support for storing sparse data
TileDB supports storing both dense and sparse data by design.
Support for storing binary blobs and vectors of different lengths
TileDB allows storing variable-sized attributes of any data type.
Support for multiple resources
You can create TileDB groups to simulate a file system on cloud object stores, which don’t have a concept of a file hierarchy. Groups can have any number of assets of any TileDB asset type within them, including other groups.
File metadata
The TileDB multi-dimensional array model includes arbitrary key-value metadata, as well as labels for axes.
Summary
Efficient data storage ensures that data is stored securely and is readily accessible for all machine learning operations. TileDB as an Adaptive Intelligence database is a natural fit for the domain of machine learning. It supports fostering research across domains with different integrations and its multi-modal nature, with applications in imaging, healthcare, geospatial, finance, defense, telecommunications, and more.