That certainly makes sense. I had a couple of follow up questions:

* How do you get your data (the input data) onto the local storage of your large EC2 instances?

* The EC2 instances are not persistent i.e. when they go down any local storage disappears. How do plan for such an eventuality i.e. some of your EC2 instances which are part of the Hadoop cluster go down?

Regards
Abhijit

That said it is possible to run Hadoop on EC2, many people are doing that. The way would you would do the hardware configuration is – use a large EC2 instance, and use local storage that comes with the server. Then you can run HDFS on top of the local storage.

I hope that helps.

Thanks,
Mukul.

It was indeed a very interesting session particularly the interaction and the discussions around various points of interest. I seem to recollect that you were investigating use of Hadoop to process the humongous amount of data that you get from the activity (clicks etc) at the various websites where the ads get displayed. I was wondering whether you have considered using Amazon Elastic MapReduce offering http://aws.amazon.com/elasticmapreduce/

It seems that this requires the data to be in S3 – Would loading data in S3 be a prohibitive cost? Are there any other concerns that you would have using the cloud for this use case as opposed to your own hosted servers ?

Regards
Abhijit

I just posted my presentation on ‘Web Scalability & Performance’ at the following URL:

http://mukulblog.blogspot.com/2009/09/web-scalability-performance-real-life.html

Thanks again for the great response and questions. I will be happy to answer more questions. You can send me an email or a message on Twitter @mukulneetika .

Thanks,
Mukul.