Comments on: Should you use a file-system or a database

By: navin

navin — Wed, 24 Mar 2010 04:14:54 +0000

@Swapan,

As I said in an earlier comment, the question to be answered is this: “When I have some data to store, what are the conditions under which I choose to store it in a file system, and when should I use a database”. Now file-systems and databases start looking comparable. There are a very large number of applications that don’t really need all the complexity, and overheads of a database, but use databases instead of file-systems because they need one feature (maybe indexing, or maybe transactions) that is missing from their file-system.

Why don’t they use ZFS, VxFS etc? My guess is this: MySQL, Oracle, or Sybase will give me a way to get the feature I want without forcing the hardware/OS on me, or without forcing my customers’ admins to learn an obscure filesystem (come on, VxFS _is_ obscure compared to the databases I listed). Also, I can get the features I need from the database in a “standard” way, where as in a file-system I’m stuck with proprietary interfaces for getting any of these features.

I think what Jaspreet is arguing for (and others have been saying since early 80s) is that having a “standard” way to do some of these things in file-systems will be a much better solution for many app vendors than being forced to use databases.

By: Swapan

Swapan — Tue, 23 Mar 2010 21:05:31 +0000

Hi Navin and Jaspreet.. Nice article.. Quite informative ..

But, at the same time, I sense that you have missed out on the complete holistic picture from the FS perspective. Just adding my bits on it.

Lets have a look on the features on which you have compared FS and DB:

1. Transaction Support: As you yourself said that ZFS, VxFS, AdVFS are the industry standards which support this feature very well.

2. Snapshots: For a FS, it is mostly expected from the backup application running on top of it to do the snapshotting. At the same time, VxFS, AdVFS etc have taken it to the FS level.

3. Clustering: You have yourself given the examples of FS that support it.

4. Replication:
How about the different RAID levels.?

PS: Navin can you please explain the motivation behind trying to compare FS and Db when they have been exiting for serving altogether a different organization of data.

Just trying to understand if I am missing on some trivial point. Or, are you just concentrating on the limitation of most of the opensource OS. Check for AdVFS (tru64 UNIX) which is too opensource.

And what are we trying to build actually siting the limitations of FS ?

By: Farheen

Farheen — Sun, 25 Oct 2009 20:24:28 +0000

I like the articles as well as every comments. I want to clarify that i myself is an engineering student and i want to know the pros and cons of flat file system and database system . Bcoz this article don’t clear the pros and cons . So if somebody reading this blog know the answer plz write an article on this topic .

By: Himanshu Ubale

Himanshu Ubale — Fri, 06 Feb 2009 09:18:33 +0000

Most of the applications tend to use both file system as well as database system for various reasons. It is equally important to abstract operations. Consider using something similar to Apache Jackrabbit (JCR implementation); we had excellent results while accessing half-GB size images (on file system) along with typical application data from relational databases thru JCR. Just for clarification, we were not storing webpages; we were storing typical application data.

By: Mitul

Mitul — Thu, 05 Feb 2009 08:35:25 +0000

@Navin: I sent a mail on your smriti.com mail id. Did you get it?

By: Mitul

Mitul — Wed, 04 Feb 2009 07:18:27 +0000

@Dhananjay: GemFire has been there for more than 6 years as well and Tangosol’s Coherence is our direct competitor. By the way, Tangosol was acquired by Oracle and it is now Oracle Coherence. You are right about TerraCotta, though it is more like a shared JVM as opposed to a typical distributed data management solution but definitely in a similar space.

By: Dhananjay Nene

Dhananjay Nene — Wed, 04 Feb 2009 07:11:13 +0000

@mitul, you are right these solutions have been proprietary for long (hence no links to them). A distributed, transactional, in-memory caching solution I know has been around since 2004 at least is Tangosol. An open source solution I am aware of which is moving parts of data management from the database into a distributed in memory storage is TerraCotta. These may not serve exactly the same set of purposes as GemFire would, but the general class of distributed transactional in memory storage has been active for some time.

Shall look forward to your post.

By: Nilesh Sane

Nilesh Sane — Wed, 04 Feb 2009 07:08:39 +0000

Good article, always nice to refresh oneself on the “classic” db terms.
I’d like to add my two cents to the discussion.
The author mentioned Replication as a means to have disaster recovery plans, but replication can and more often does go beyond it. Replication is a very essential indgredient for managing distributed databases. DoubleD’s are common now a days and file system db’s will find it hard to match this requirement.
I wonder how XML’s would be rated then? Just another means of EDI?

By: Jaspreet

Jaspreet — Wed, 04 Feb 2009 06:38:32 +0000

Navin,

There is a dark side to the cloud as well. We investigated a lot in this space. And I plan to write on this soon.

My reasons -

1. Its something which is critical to your business and you can’t control – I know 3 startups which went out of business because of cloud downtimes and data losses.

Unlike hosted email, which is there for the past 20 years. This space would take a long time to mature. Besides,no one minds if the email server goes down for some time.

2. Applications need to be changed to adjust to cloud API.

3. Performance – Its a big question here.

4. Its a vendor lockin. You can’t change options tomorrow.

We investigated on a lot of options and benchmarked – SQLite, MSSQL, MySQL, PostgreSQL and Oracle. Then finally settled on PostgreSQL. Will share this investigation sometime.

@sashi, look at the disclosure.

Jaspreet

By: navin

navin — Wed, 04 Feb 2009 06:37:32 +0000

@Dhananjay, Agreed that doing a good in-depth analysis is easily a one-semester course! However, I think there is still value in giving a 30,000-foot view of this area. I mean one paragraph on each of the major approaches that appear to be emerging.

@Mitul, what you are describing sounds fine – can you write an article for PuneTech on that topic?