Should you use a file-system or a database
Whether to use a file-system or a database to store the data of your application has been a contentious issue since the 80s. It was something we worried about even when I was doing my Ph.D. in Databases in the 90s. Now Jaspreet Singh, of Pune-based startup Druvaa has weighed in on this issue on Druvaa’s blog. His post is republished here with permission.
This topic has been on my plate for some time now. It’s interesting to see how databases have come a long way and have clearly out-shadowed file-systems for storing structured or unstructured information.
Technically, both of them support the basic features necessary for data access. For example both of them ensure -
- Data is managed to ensure its integrity and quality
- Allow shared access by a community of users
- Use of well defined schema for data-access
- Support a query language
But, file-systems seriously lack some of the critical features necessary for managing data. Lets take a look at some of these feature.
Atomic transactions guarantee complete failure or success of an operation. This is especially needed when there is concurrent access to same data-set. This is one of the basic features provided by all databases.
But, most file-systems don’t have this features. Only the lesser known file-systems – Transactional NTFS(TxF), Sun ZFS, Veritas VxFS support this feature. Most of the popular opensource file-systems (including ext3, xfs, reiserfs) are not even POSIX compliant.
Databases allow indexing based on any attribute or data-property (i.e. SQL columns). This helps fast retrieval of data, based on the indexed attribute. This functionality is not offered by most file-systems i.e. you can’t quickly access “all files created after 2PM today”.
The desktop search tools like Google desktop or MAC spotlight offer this functionality. But for this, they have to scan and index the complete file-system and store the information in a internal relational-database.
Snapshot is a point-in-time copy/view of the data. Snapshots are needed for backup applications, which need consistent point-in-time copies of data.
The transactional and journaling capabilities enable most of the databases to offer snapshots without shopping access to the data. Most file-systems however, don’t provide this feature (ZFS and VxFS being only exceptions). The backup softwares have to either depend on running application or underlying storage for snapshots.
Advanced databases like Oracle (and now MySQL) also offer clustering capabilities. The “g” in “Oracle 11g” actually stands for “grid” or clustering capability. MySQL offers shared-nothing clusters using synchronous replication. This helps the databases scale up and support larger & more-fault tolerant production environments.
File systems still don’t support this option :( The only exceptions are Veritas CFS and GFS (Open Source).
Replication is commodity with databases and form the basis for disaster-recovery plans. File-systems still have to evolve to handle it.
Relational View of Data
File systems store files and other objects only as a stream of bytes, and have little or no information about the data stored in the files. Such file systems also provide only a single way of organizing the files, namely via directories and file names. The associated attributes are also limited in number e.g. – type, size, author, creation time etc. This does not help in managing related data, as disparate items do not have any relationships defined.
Databases on the other hand offer easy means to relate stored data. It also offers a flexible query language (SQL) to retrieve the data. For example, it is possible to query a database for “contacts of all persons who live in Acapulco and sent emails yesterday”, but impossible in case of a file system.
File-systems need to evolve and provide capabilities to relate different data-sets. This will help the application writers to make use of native file-system capabilities to relate data. A good effort in this direction was Microsoft WinFS.
The only disadvantage with using the databases as primary storage option, seems to be the additional cost associated. But, I see no reason why file-systems in future will borrow features from databases.
Druvaa inSync uses a proprietary file-system to store and index the backed up data. The meta-data for the file-system is stored in an embedded PostgreSQL database. The database driven model was chosen to store additional identifiers withe each block – size, hash and time. This helps the filesystem to -
- Divide files into variable sized blocks
- Data deduplication – Store single copy of duplicate blocks
- Temporal File-system – Store time information with each block. This enables faster time-based restores.
Related articles by Zemanta
- Cloud Database Standard Needed (continuations.com)
- The Commoditization of Massive Data Analysis (radar.oreilly.com)
- Speeding Up Database Piggies (arnoldit.com)
- Programmers take to the clouds (theregister.co.uk)
- Understanding Data De-duplication (punetech.com)