Category Archives: In Depth

Supply Chain Management in Consumer Goods – An In-Depth Look

In Depth, Overviews,

Amit Paranjape, a regular contributor and primary adviser to PuneTech, had earlier written an article giving an overview of Supply Chain Management, and companies in Pune that develop software products in this area. This article, the next in the series, goes into details of the problems that SCM software products need to tackle in a consumer goods supply chain. This is a longer-than-usual article, hence posted on a Friday so you can read it over the weekend (assuming you are not attending one of the various tech activities happening in Pune this weekend.)

Here is a story about a packet of ‘Star Glucose Biscuits’ in ‘SuperMart’ on FC Road in Pune, told from the point of view of Supply Chain Management. Buckle up your seat belts because this story has tension, drama, emotion, and suspense (will the biscuits reach the shops in time for the T20 World Cup Promotion?)

Overview

The story begins at the Star Biscuits Factory in Bangalore where flour, sugar and other raw material are converted to the finished cases of biscuits. From Bangalore, the biscuits are shipped to a regional Distribution Center on the outskirts of Pune. This center then ships the biscuits to the local depots in different parts of cities such as Mumbai, Pune and from there they ultimately end up at the neighboring retail store, such as SuperMart on FC Road, Pune. In this seemingly simple journey are hidden a host of difficult business decisions and issues that arise on a daily basis. And to complicate matters further, we will throw in a few ‘interesting’ challenges as well! Throughout this story, we will take a deeper look at how the various business processes, and software programs associated with planning this entire supply chain network work in concert to bring you the extra energy and extra confidence of Star Glucose Biscuits.

This chain, from the raw materials all the way to the finished product sitting on the retail shelves is called the supply chain, and managing it efficiently is called supply chain management. Supply chain management is one of the most important aspects of running a manufacturing business, and doing it well has been the key to the phenomenal success of such giants as Walmart and Dell. The basic conflict that SCM is trying to tackle is this: you must have the right quantity of goods at the right place at the right time. Too few biscuits in the store on Sunday, and you lose money because have to turn customers away. Too many biscuits in the store and you have excess inventory. This is bad in a number of ways: 1. It eats up shelf space in the store, or storage space in your warehouse. Both of these cost money. 2. Your money, your working capital is tied up in excess inventory which is sitting uselessly in the warehouse. 3. If the biscuits remain unsold, you lose a lot of money. The same trade-off is repeated with intermediate goods at each step of the supply chain.

The Supply Chain in detail

Schematic of a supply chain. From bottom to top: multiple suppliers supply raw materials to multiple factories. Finished goods are then sent to regional distribution centers. From there it goes to smaller regional depots, and finally to individual stores.
Schematic of a supply chain. From bottom to top: multiple suppliers supply raw materials to multiple factories. Finished goods are then sent to regional distribution centers. From there it goes to smaller regional depots, and finally to individual stores.

At the Star Biscuit factory in Bangalore, they are gearing up to meet forecasted production requirements that were recently communicated by the Star Biscuits headquarters (HQ) in Mumbai. This is the ‘demand’ placed on this factory. These production requirements consist of weekly quantities spread over next 12 weeks. The factory planning manager now has to plan his factory to meet these requirements on time.

Let us see what all he needs to take into account. First, he needs to figure out the raw material requirements – wheat flour, oil, sugar, flavors, etc. as well as packaging material. Each of them has different procurement lead-times and alternative suppliers. He needs to pick the time to place orders with the right suppliers so that the material is available on time for the manufacturing process.

The manufacturing process itself consists of two primary steps – making the biscuits from the flour, and packaging the biscuits into individual boxes and cases. Typically, multiple parallel making and packing lines work together to achieve the desired output. The packing process scheduling is often complicated further by a series of different sizes and packaging configurations.

Ensuring that the right amount of material is available at the right time is called Material Requirements Planning (MRP), and in the old days, that was good enough. However, this can no longer be done purely in isolation. Even if the amounts of the different raw materials required are predicted precisely, it can be problematic if the various making and packing machines do not have the capacity to handle the load of processing the raw materials. Hence, another activity, called capacity planning, needs to be undertaken, and the capacity plan needs to be synchronized with the materials requirement plan, otherwise excess raw material inventory will result, due to sub-optimal loading of the machines and excessive early procurement of raw material. Excess inventory translates to excess working capital cash requirement; which in the current hyper competitive world is not good! Luckily, today there are sophisticated APS (Advanced Planning & Scheduling) software tools that are far superior to traditional MRP systems that enable him to simultaneously do material and capacity planning.

Happy with the production plan for the next 12 weeks, the factory planner then makes sure that individual making and packing lines have their detailed production schedules for the next two weeks. The finished cases leave his factory on truck loads. But where do they go from here? The journey to SuperMart where our customer wants to purchase the final product is still far from over!

The next stop is a big distribution center (DC) for the western region that sits on the outskirts of Pune. This distribution center is housed in a large warehouse with multi-level stacking pallets (each pallet contains multiple cases) of multiple different products from the manufacturer. A set of conveyors and fork-lifts enable material to flow smoothly from inbound truck docks to stocking area, and from the stocking area, to the outbound truck docks. These products come not only from our Star Biscuits factory in Bangalore, but from various other Star Biscuits factories located all over India. In fact, some of these products could also be directly imported from the parent company of Star Biscuits in the UK (the ones with the bitter, dark chocolate!). This Pune distribution center stocks and stores this material in proximity to the western region – with specific emphasis on the large Greater Mumbai and Pune markets. How are all these warehousing related activities smoothly managed? The DC manager takes full advantage of a Warehouse Management System Software (WMS). The truck loading and load containerization is managed by a Transportation Management Module.

From here outbound shipments are sent to smaller regional depots that are located in the cities, nearer to the stores. From these depots, the biscuits are finally shipped to the stores to meet the end customer demand. How is this demand calculated? Clearly, it is impossible to predict the demands coming from individual customers at the store, few weeks in advance! Hence it is necessary to ‘forecast’ the demand.

Forecasting demand and determining stock levels

Who decides how much material to stock? And how is it calculated? Clearly, as we briefly indicated earlier, keeping too much product is costly and keeping too little results in stockouts (material unavailability on store shelf) at the stores, thereby resulting in unhappy customers and lost sales. Too much product equals excess working capital (similar to the excess raw material problem) and is not good for the company’s financial performance. Too little, and we will run out if there are any major demand swings (commonly referred to as ‘demand variability’). To achieve the optimum level of the desired stock on hand to buffer against demand variability a ‘safety stock quantity’ is maintained in the warehouse. This quantity is computed by the central Supply Chain Management (SCM) team at HQ.

The actual computation of the safety stock for different products in the distribution center is calculated using a statistical computation (In some cases, a manual override is also done over the computed value). The most common technique consists of using Poisson distribution, demand & supply variability historical data, demand & supply lead time data, and desired customer service levels. Customer service levels are assigned based on an “ABC” classification of the products. ‘A’ category items are the fast movers and have a high revenue share and are typically assigned a 99% customer service level. Roughly speaking, a ‘99%’ customer service level implies that the safety stock quantity is adequate to guard against demand variability signals 99 times out of 100. Proactive planning on a daily basis that involves daily monitoring of stocks of all products is done, based on actual outbound shipments, can many times help in even reacting to that ‘1 in 100’ cases with rapid corrective measures.

The forecasting process for all the products is done at Star Biscuits Head Quarters. Let us continue with our example of ‘Star Glucose Biscuits’. The modern forecasting process is more accurately referred to as a ‘Demand Planning’ process. Statistical forecast is one input to the overall process. Statistical forecasts are derived from shipment history data and other input measures such as seasonality, competitor data, macro-economic data, etc. Various statistical algorithms are used to come up with a technique that reduces the forecasting error. Forecasting error is typically measured in ‘MAPE’ (Mean Absolute Percentage Error) or ‘MAD’ (Mean Absolute Deviation).

The statistical forecast is then compared with the sales forecast and the manufacturing forecast in a consensus planning process. This is often done as part of a wider ‘Sales & Operations Planning Process’ in many companies. Often times, a ‘top-down’ and ‘bottom-up’ forecasting technique is used. Here, individual forecasts at the product level are aggregated up the product hierarchy into product group forecasts. Similarly, aggregate product group level forecasts are disaggregated down the same hierarchy to the individual product level. These are then compared and contrasted and the expert demand planner then takes the final decision. Aggregated forecasting is important, since often times this reduces the forecast error. In case of Star Glucose Biscuits, the aggregated product hierarchy would first combine all sizes (e.g. 100 gm, 200 gm), then aggregate along sugar based biscuits type, and then into ‘All biscuits’ groups.

The end result of the demand planning process is the final consensus forecast that is calculated in weekly time intervals (commonly referred to in planning terminology as ‘buckets’) for a time horizon of 8-12 months. The demand forecasts drive the entire Star Biscuits Supply Chain. To simplify the overall Demand Planning process, modern DP software tools provide great help in statistical forecasting, OLAP based aggregation/disaggregation, and in facilitating interactive collaborative web-based workflows across different sets of users.

Managing the Supply Chain

It was easy if all we had was one DC in Pune and one factory in Bangalore, supplying to one store. But Star Biscuits has a much more extensive network! They have multiple factories throughout India and the same biscuits can be produced by each factory. How much of the demand to allocate to which factory? This problem is addressed by the Supply Chain Management (SCM) team that works in close concert with the Demand Planning team. The allocation is made based on various criteria such as shipment times, capacities, costs, etc. In coming up with the sourcing, transportation and procurement decisions – minimizing costs and maximizing customer service are amongst the top business objectives.

Now, getting back to Star Biscuits, they have over 300 products across 5 factories and 4 DCs. The DCs in turn, receive demand from nearly 100 depots that are supplying to thousands of stores all over India. Determining the optimal allocation of demands to factories, safety stocks to DCs and all the transportation requirements is beyond the abilities of humans. Such a complicated scale and decision problem needs computer help! Luckily, advanced SCM software tools can help the SCM team make these decisions fairly efficiently. Good SCM tools allow user interactivity, optimization, and support for business specific rules & heuristics. The SCM process thus determines the 12 week demand in weekly buckets for our factory in Bangalore, where we started.

To summarize, the overall supply chain – we saw how the product demand gets forecasted at HQ by the demand planning group. The SCM group then decides on how to allocate and source this demand across the different factories. They also decide on the ideal safety stock levels at the DCs. The WMS group ensures the efficient management of the distribution center activities. The factory planner team decides on the most efficient way to produce the biscuits demand allocated to their plant. The transportation management team is assigned the task of shipping material across this network in the best possible way to reduce cost and cut down on delivery times.

Dealing with drastic changes

And all of this is just to do “normal” business in “normal” times.
All the processes described earlier are great if the business works at a stable, reasonably predictable pace. Our safety stock policies guard against the day to day variability. But what about drastic changes? Unfortunately in the current environment, the only thing that is constant is ‘change’.

Here is what happened at Star Biscuits. One day, out of the blue, the entire planning team was thrown into a mad scramble by a new request from the marketing department. In order to react to a marketing campaign launched by one of their top competitors, the marketing department had a launched a new cricket promotion of its own for the next month.

Promotions are extremely important in the consumer goods industry. They entail targeted customer incentives, advertising spending and custom packaging – all in a synchronized fashion. The success of promotions often times make or break the annual year performance of a Consumer Goods Company. Promotions driven sales often times contribute large double digit percentage of total sales of consumer goods companies.

This particular cricket promotion involved a special packing requirement with the star logos on the packet. The target customer demand was not only upped by 50%, the offer also had a ‘Buy 1 Get 1 Free’ incentive. As a result, the total demand was going up by nearly 300%.

The SVP in charge of Supply Chain was trying his best to get a handle of the problem. He was getting irritated by the constant pressure he was under from the SVP Marketing, and the CEO.

The demand planning team had to quickly alter its demand numbers to meet the new targets. The real trouble spot was brewing at the SCM team. The team had to rapidly make decisions on where to source this sudden demand spike. While cost optimization is important, meeting customer demand at ‘all costs’ is the key. The Bangalore factory was already running at 90% capacity and was in no position to produce much more. Luckily for the SCM team, their SCM tool quickly ran a series of scenarios and presented possible alternatives. These scenarios looked at various alternatives such as contract packing, new factories, expedited raw material shipments, direct shipments from the factories to the stores, etc. One of the resulting scenarios seemed to fit the bill. It was decided that bulk of the extra demand be routed to the alternative factory in Faridabad which had some spare capacity. From here, the product was going to be shipped directly (where feasible) to the Mumbai and Pune depots, where a large chunk of the promotion driven demand was expected. The rest of the country’s demand was going to be met by the conventional approach, from the Bangalore factory. The new package also resulted in demand for new packaging material with the cricket logos. New scenarios were generated that source this material from packaging material suppliers from the middle-east. (Interesting to note, that in some time crunched promotions, packaging material often times ends up being the bottle neck!)

Satisfied with this approach, the SVP Supply Chain ordered his team to come up with process improvements to prevent such scrambles in future. Luckily there was an easy solution. The Demand Planning software tool had a nice capability to support an integrated promotions planning & demand planning workflow. Such workflows look at various promotions related data, such as timing, costs, volume, competitor strategies and efficiently plan future promotions – instead of reacting to them at the last minute. In turn, such effective promotion planning can not only drive revenues, but also further improve supply chain efficiencies.

The SVP is happy, but what happened to our end customer on FC Road Pune? Well, she walked away happy with her promotion pack of Star Glucose biscuits, completely oblivious of what had happened behind the scenes!

About the Author – Amit Paranjape

Amit Paranjape is one of the driving forces behind PuneTech. He has been in the supply chain management area for over 12 years, most of it with i2 in Dallas, USA. He has extensive leadership experience across Product Management/Marketing, Strategy, Business Development, Solutions Development, Consulting and Outsourcing. He now lives in Pune and is an independent consultant providing consulting and advisory services for early stage software ventures. Amit’s interest in other fields is varied and vast, including General Knowledge Trivia, Medical Sciences, History & Geo-Politics, Economics & Financial Markets, Cricket.

What is multi-core architecture and why you need to understand it

In Depth, Technology, ,

Dhananjay Nene has just written a brilliant article in which he gives a detailed overview of multi-core architectures for computer CPUs – why they came about, how they work, and why you should care. Yesterday, Anand Deshpande, CEO of Persistent Systems, while speaking at the IndicThreads conference on Java Technologies exhorted all programmers to understand multi-core architectures and program to take advantage of the possibilities they provide. Dhananjay’s article is thus very timely for both, junior programmers who wish to understand why Anand was attaching so much importance to this issue, and what they need to do about it, and also for managers in infotech to understand how they need to deal with that issue.

Dhananjay sets the stage with this lovely analogy where he compares the CPU of your computer with superman (Kal-El) and then multi-core is explained thus:

One fine morning Kal’s dad Jor-El knocked on your door and announced that Kal had a built in limitation that he was approaching, and that instead of doubling his productivity every year, he shall start cloning himself once each year (even though they would collectively draw the same salary). Having been used to too much of the good life you immediately exclaimed – “But thats preposterous – One person with twice the standard skill set is far superior to 2 persons with a standard skill set, and many years down the line One person with 64 times the standard skill sets is far far far superior to 64 persons with a standard skill set”. Even as you said this you realised your reason for disappointment and consternation – the collective Kal family was not going to be doing any lesser work than expected but the responsibility of ensuring effective coordination across 64, 128 and 256 Kals now lay upon you the manager, and that you realised was a burden extremely onerous to imagine and even more so to carry. However productive the Kal family was, the weakest link in the productivity was now going to be you the project manager. That in a nutshell is the multicore challenge, and that in a nutshell is the burden that some of your developers shall need to carry in the years to come.

What is to be done? First is to understand which programs are well suited to take advantage of a multi-core architecture, and which ones:

if Kal had been working on one single super complex project, the task of dividing up the activities across his multiple siblings would be very onerous, but if Kal was working on a large number of small projects, it would be very easy to simply distribute the projects across the various Kal’s and the coordination and management effort would be unlikely to increase much.

Dhananjay goes into more detail on this and many other issues, that I am skimming over. For example:

Some environments lend themselves to easier multi threading / processing and some make it tough. Some may not support multi threading at all. So this will constrain some of your choices and the decisions you make. While Java and C and C++ all support multi threading, it is much easier to build multi threaded programs in Java than in C or C++. While Python supports multi threading building processes with more than a handful of threads will run into the GIL issue which will limit any further efficiency improvements by adding more threads. Almost all languages will work with multi processing scenarios.

If you are a programmer or a manager of one, you should read the entire article.  In fact, as we mentioned in  a previous PuneTech post (Why Python is better than Java), you should really subscribe to his blog. He writes detailed and insightful articles that, as a techie, you would do well to read. If you are interested in programming languages, I would recommend reading “Contrasting java and dynamic languages”, and “Performance Comparison – C++ / Java / Python / Ruby/ Jython / JRuby / Groovy”. And if you are a blogger, check out his tips for software/programming blogging.

Dhananjay is a Pune-based software Engineer with 17 years in the field. Passionate about software engineering, programming, design and architecture. For more info, check out his PuneTech wiki profile.

Introducing Ask PuneTech – Ask us anything and we’ll get you the answer

In Depth,

Update: Ask PuneTech is being replaced by ForPune.com, a site that is much better suited to answer questions you might have about Pune. Please ask a question there, and we (PuneTech and the rest of the tech community in Pune) will monitor that site and try to get an answer for you. To ensure that the right people read your question, please make sure to tag your question “punetech” so that it gets read by the right people.

The rest of this page is kept for historical purposes, but we prefer that you use ForPune.com instead of the mechanisms described below.

Quick Summary for those too lazy to read the whole page: Ask PuneTech is a One-Stop Shop for all the information needed by infotech professionals and infotech startups in Pune. Ask us a question, or for a referral, or for some information and we find it for you, or we will put you in touch with person who has previous experience in that area to guide you. Just send an email to ask@punetech.com.

Details: PuneTech has been running for 8 months now, and it has been very successful in filling the information gap that existed in the tech ecosystem in Pune. The PuneTech blog reaches about 700 people on a daily basis. The PuneTech wiki is slowly adding more and more useful data (see, for example, the page on tech user groups and organizations in Pune). The PuneTech calendar has become the most comprehensive source of information about tech events in the city.

However, one of the more unexpected ways in which we are helping people is in finding answers to our readers’ questions, or connecting them to the right people. In doing this, we have realized that there is a a need of a service where people who are facing some problem need to be connected with someone who has solved that problem in the past. The internet and google are supposed to solve this problem, but it doesn’t work that way. The web is not particularly useful if you are looking for specific, local information (like, who is a good CA for STPI registration). And sometimes the problem complicated and you need to talk to a human who can guide you. We noticed that we have been doing more and more of that in recent times, and we decided to formalize the concept and announce it. Thus was born the “Ask PuneTech” service.

Through PuneTech, we now know enough experienced people in the field of infotech that for any question you might have as a techie in Pune, or an infotech startup in Pune, we can connect you to the right person who can help you. More importantly, we know enough people who would happily help others like this for free – just for the satisfaction of preventing others from going through the same pain that they went through. So send the question to ask@punetech.com and we’ll get back to you with the answer.

This is a free service, but if you ask a question, you need to promise that when you get a solution to you problem, you will write it up in an article and send it to us for publishing on the PuneTech wiki for the benefit of all the others in the future. This way, the knowledgebase will continue to grow.

The rest of this page has more details in a Q&A format to make you think that you are talking to yourself:

Q: How does it work?

A: If you are a techie in Pune, or a Pune-based info tech startup, and you have some question or difficulty send us the question at ask@punetech.com. If we already know the answer (i.e. someone has asked that question before) you’ll get the answer right away. If not, we’ll tap our network of volunteer experts, and try to get the answer for you. Or, we’ll put you in touch with a person who we believe will be able to guide you appropriately, and whose judgment we have some confidence in. We’ll provide you with of the source of the information, and our confidence in the validity of the information, and/or our confidence in the reputation of the source.

This will be a completely free service, but there is one rule you have to follow: If you find a good enough solution to your problem (and assuming that the question / answer happens to be of general interest), you must write it up and send it to us so that we can post it to the PuneTech wiki for the benefit of the rest of the world. You must promise us this as a “payment” for using our service.

Sample questions:

1. Can you suggest a good consultant / service provider for the following:

a. Accounting

b. Company registration

c. STPI registration

d. Branding / marketing

e. Design (logo, website)

f. Intellectual property (copyright, trademarks, patents)

g. Legal questions (cyber law, privacy)

h. Facilities (rental accommodation, furniture, etc.)

2. Can you suggest a good free / paid service for

a. Web hosting

b. Broadband internet

c. Website development

d. Search Engine Optimization

3. Where can I find a Ph.D. in statistics / maths / data-mining in Pune to help me with xyz

4.

Warning: Ask PuneTech is not intended to be a jobs board. Do not post about open positions you might have. Do not advertise yourself as a candidate. Such requests will be summarily deleted. Requests for persons with specific skills will be entertained only if the skills required are very specific (e.g. good: looking for help with Erlang; bad: looking for Java expert) and for a short consultation (e.g. good: need help with installing SuSE on a Dell laptop, bad: need a QA person).

Q: How much do you charge for this?

A: This is a free service. We are a group of volunteers doing this because we really want the tech ecosystem in Pune to improve, become more vital, become more active, and in general increase the productivity of Pune’s techies. We personally benefit from this indirectly: vastly improved networking, increase in visibility, increase in exposure, having a finger on the pulse of Pune, and the undying gratitude of some of the smartest people in Pune.

Q: Do you take money from services/experts whom you are recommending?

No. PuneTech and “Ask PuneTech” are community-driven, non-commercial services that do not accept any money from any source for any purpose whatsoever. Services/sources will be recommended purely on the basis of quality of service provided.

Q: Why not create a wiki for this and be done? Why is this service needed?

A: We tried and it doesn’t work. For some reason, people don’t add content to a wiki. There isn’t enough critical mass, and/or visibility for this to work. So, to tackle that problem, we are trying this, where we manually connect the “askers” to the “askees”, and “force” the “askers” to update the wiki after the question has been answered.

Q: Why not have an open mailing list where anybody can post questions and answers?

A: One problem is that there are many people who are not aware of the appropriate mailing list or website or forum where the question can be asked. In such cases, the Ask PuneTech service will simply point you towards the appropriate forum where you should ask your question. And in case, the question doesn’t get answered there, then we try to find an actual person to answer the question for you.

That brings us to the second problem: Sometimes posting to a mailing list works, but often it doesn’t. In other cases, there are many questions which do not get answers on the mailing list, in spite of the fact that the list does contain members who know the answers. It is often the case that the person who knows the answer would be happy to guide you if you called him up on the phone, but doesn’t have the time to type out the answer in email (or doesn’t read the mailing list so closely.) We are hoping to overcome this problem by introducing a personal touch.

Where possible, we will redirect you to an appropriate mailing list. e.g. if you have problems with connecting to BSNL broadband from your Ubuntu system, we’ll send you to the PLUG mailing list.

Q: How do you keep tabs on the quality / reliability of the information you provide?

A: We intend to tackle this in two ways. First, for all the suggestions you get, we’ll provide you with of the source of the information, and our confidence in the validity of the information, and/or our confidence in the reputation of the source. Hopefully this will be enough for you to take a slightly more informed decision. We feel that this will be better than having no information at all.

Second, over time, we intend to build a database of information sources and our confidence in the accuracy of their information. After we suggest something to someone, we expect them to provide us feedback as to how well it worked out. We’ll collate this information over time so that the quality of the information improves.

Q: Why are you doing this manually? Shouldn’t you use technology / wisdom of the crowds / automation to do this more efficiently?

OR

Q: Is this the same as Yahoo!Answers / LinkedIn Answers / xyz Forums / CraigsList?

A: We believe that technological / wisdom-of-the-crowds solutions only work at a large scale (like Wikipedia / Yahoo!Answers). It does not work at a smaller scale (like Pune). Otherwise, a viable technology solution would have emerged by now. The lack of that has forced us to try this out.

That said, we definitely intend to use leverage technology as much as possible in this endeavor. We just feel that initially this has to be done manually until it gathers enough of a critical mass.

Also, if you feel that a technology solution will beat this, we would encourage you to try. This is one argument we’d love to lose. We don’t care whether our approach wins or some other approach. We just want a solution to the problem of lack of reliable information.

Q: If this is free, it is not sustainable! This service will shut down after 6 months!

A: We have been doing http://punetech.com as a free service for almost an year now, and we have a pretty good feel for what is sustainable and what isn’t. In any case, we’ll worry about sustainability a little later. For now we are focusing on proving that there is a need, and people will use a service like this, and it will be useful. (We are convinced of that, but we need to prove it.) Once that has been proved, we can worry about how to sustain it.

Q: Since this is volunteer driven, it is not scalable!

A: See answer to previous question.

Q: How can I help?

A:

  1. Start using the service. The more people use it, the more useful the service becomes (and the knowledgebase grows)
  2. Make yourself available to us as an expert who can provide the answer (in whatever happens to be your area of expertise). We promise to not bother you too much. We’ll only forward you as many questions are you are willing to handle. If we don’t know you, we’ll initially put you down as an “untrusted source” and over time, your designation will (hopefully!) change to “reliable”.
  3. Tell all your friends about this service. (Actually, while you are at it, also tell your friends to subscribe to PuneTech at http://punetech.com/subscribe/ to get information about all the latest news, events, tech groups, startups and technologies in Pune.

Q: Who is behind this service?

A: The questions asked to “Ask PuneTech” are answered by a network of volunteers across Pune, who all share a passion for technology, or a passion for Pune, or, in most cases, both.

The service is coordinated by the people who run PuneTech, the most comprehensive source of information about technology in Pune (if you find a better source please let us know!) It was started by Navin Kabra. Amit Paranjape is a key advisor and evangelist for PuneTech, and one of the top volunteers. Another special mention goes to the Pune OpenCoffee Club, which has a close symbiotic relationship with PuneTech. There are a number of other people, too numerous to mention, who contribute in varying degrees.

Both, PuneTech and “Ask PuneTech” are intended to be community-driven, non-commercial services. To ensure that the content remains free from bias and vested interests, we do not accept any money from any source for any purpose whatsoever.

The next wave of entrepreneurship in India

In Depth, ,

Monish Darda, Director of Product Development at BMC BladeLogic and co-founder of WebSym, wrote an article on the next wave of entrepreneurship in India for CSI Pune’s quarterly newsletter Desktalk. Parts of that article have been reproduced here with permission. You should be able to download the the newsletter (contain the full article) from CSI Pune website’s download section. (It’s the July-Aug-Sept 2008 newsletter.)

So let us take a look at tomorrow’s entrepreneur – I tend to see him (or her – the masculine is just convenience) in two colors – Mr E the risk taker, and Mr E, the man with the plan. Tomorrow’s risk taker is a person fresh out of college, with a few like-minded close friends and a couple of mentors, who want to do the next cool thing. The man with the plan is a youngish guy, probably back from the USofA, his future secured, with a plan that will leverage India for his next successful startup. Is anyone building up a services company? Well, yes and no. What I see in the future is services sold as a product. One in three entrepreneurs are going to be thinking about leveraging the labor cost differential at the low end of the value chain along with innovation that takes the returns to the high end of the value chain.

I see social networking sites that will spawn with better ideas in India, quickly gaining eyeballs, cheaper and faster than anywhere else except China perhaps. I see quicker and richer integration of media, with infotainment at the core. Indian entrepreneurs will be driving mid range technology applications with a larger audience and higher success rates. You saw youtube and facebook become the rage and build value worth billions in a short span of time – hold your breath for the Indian versions; they are not too far away in the future.

I see mobile middle-ware and products – if every other Indian is going to have a mobile phone soon, the apps are not far away. Indian apps, in Indian languages, closer to the Indian psyche, driven by Indian technology entrepreneurs. And it is not far in the future that we will have our own Nokias and Sonys and Ericssons – hardware is sure to follow.

I am not a betting person, but if I were, I would be betting on small, consumer shareware coming out of India in the near future – apps for the phone, the iPhone and the PC, that makes practical use of the now ubiquitous personal computer. I see enterprise software being developed in India, owned in India, but still managed and sold principally outside India. The entrepreneurs are going to be eyeing the small, high volume software for cash market, where services coupled with technology vying to increase the quality of life.

How will Technology Entrepreneurship benefit India?

My bet on the man with the plan is that he will drive the “real” technology – high tech technology creation and adaptation for grand socio-economic experiments, for logistics and the growing manufacturing industry; selling to corporates, multi-nationals and governments. He will be the guy attracting large investments and innovation dependent on infrastructure.

We are culturally a very adaptable, and very tolerant, people – the technology infusion of the future is going to bring about sweeping social change; most of which has already started. Look at what we did with the mobile phone – with SMS and the “missed calls” syndrome. We adapted the technology to suit our way of working, and kept driving costs to the ground. We will keep doing that with automobiles, phones, software, nuclear power and any other technology that we adapt as our own. We are very good at converting luxuries to needs, and that to me is key – the future of the Indian entrepreneur is dependent on this one factor. And I think this mass market has the potential to make billionaires out of ordinary people, with the spark to adapt and profit. And I believe with all its perceived ineptness, corruption and mismanagement, the government is somehow going to be the catalyst to make this happen, believe it or not. Perhaps all the impossible traffic and the constant load-shedding is already starting an entrepreneur somewhere on the road to his empire …

About the Author – Monish Darda

Monish is Director, Product Development at BMC BladeLogic. He set up the India operations of Storability Software, an East coast storage startup and was heading the group at Sun Microsystems when Storability underwent two acquisitions.

Monish is also the cofounder and Director of Websym Technologies.

He did his Master’s at Florida Atlantic University in Boca Raton, FL. He has architected large systems in the areas of customer acquisition, manufacturing and finance on J2EE and Microsoft platforms. He has also shared his experience in leading technologies in implementation and design through mentoring programs for senior developers and designers in top national and international software houses. He has implemented innovative processes and tools for distributed design and development across the US and Europe

Reblog this post [with Zemanta]

Stop Virtual Machine Sprawl with Colama

In Depth, Overviews, , ,

This is a product-pitch for Colama, a product built by Pune-based startup Coriolis. For the basics of server virtualization, see our earlier guest posts: Introduction to Server Virtualization, and Why do we need server virtualization. Virtualization is fast emerging as a game-changing technology in the enterprise computing space, and Colama is trying to address an interesting new pain-point that making its presence felt. 

Virtualization is fast becoming an accepted standard for the IT infrastructure. While it comes as a boon to the development and QA communities, the IT practitioners are dealing with the pressing issue of virtual machine sprawl that surfaces due to adhoc and hence uncontrolled adoption of virtualization. While describing the problem and its effects, this paper outlines a solution called Colama, as offered by Coriolis Technologies.

 

Virtual machines have been slipping in under the covers everywhere in the IT industry. Software developers like virtual machines because they can easily mimic a target environment. QA engineers prefer virtual machines because they can simultaneously test the software on different configurations. Support engineers can ensure reproducibility of an issue by pointing to an entire machine rather than detailing on the individual steps and/or configuration requirement on a physical host. In many cases, adoption of virtual machines has been primarily driven by users’ choice rather than any coherent corporate strategy. The ad-hoc and uncontrolled use of virtual machines across the organization has resulted in to a problem called Virtual Machine sprawl, which has become critical for today’s IT administrators.

Virtual machine sprawl is an unpleasant side effect of server virtualization and its near exponential growth over the years. Here are the symptoms:

  • At any given point, the virtual machines running in the organization are un-accounted for. Information like who created them and when, who used them, what configuration/s they have, what licensed software they use, whether security patches have been applied, whether the data is backed up etc are not maintained and tracked anywhere.
  • Most commonly, people freely copy each other’s virtual machines and no usage tracking and access control is in place.
  • Because of cheap storage, too many identical or similar copies of the same machines are floating across the organization. But reduction in storage cost does not reduce the operational cost of storage management, search, backup, etc. Data duplication and redundancy is a problem even if storage is plentiful.
  • Because there is no mechanism to keep track of why an image was created, it is hard to figure out when it should be destroyed. People tend to forget what each image was created for, and keep them around just in case they are needed. This increases the storage requirements.
  • Licensing implications: Virtual machine copied from one with a licensed (limited) software needs to tracked for its life span in order to put a control on the use of licensed software.

    •  

    There are many players in the industry who address this problem. Most of the virtual lab management products are tied to one specific virtualization technology. For example, the VMWare Lab Manager works for only VMWare virtualization technology. In a heterogeneous virtualization environment that is filled with Xen, VMWare, VirtualBox and Microsoft virtual machines, such an approach falls short.

    Colama is Coriolis Technologies solution to address this problem. Colama manages the life cycle of virtual machines across an organization. While tracking and virtual machines, Colama is agnostic to the virtualization technology.

     

    Here are some of the features of Colama:

  • Basic SCM for virtual machine: You can Checkin/checkout/clone/tag/comment virtual machine/s for tracking revisions of virtual machine.
  • Image inspection: Colama provides automatic inspection, extraction and display of image-related data, like OS version, software installed, etc and also facilitates “search” on the extracted data. For example, you can search for the virtual machines that have got Windows 2003 server with service pack 4 and Oracle 10g installed!
  • Web based interface: Navigate through the virtual machine repository of your organization using just a web browser.
  • Ownership and access control: • Create a copy of a machine for yourself and decide who can use “your” machine.
  • De-duplication: Copying/Cloning virtual machines happens without any additional storage requirement.
  • Physical machine provisioning (lab management): Spawn a virtual machine of your choice on a physical host available and ‘ready’.
  • Management reports: auditing and compliance User activity reports, virtual machine history, health information (up/down time) of virtual machines, license reports of the virtual machines etc.
  • Virtualization agnostic: works for virtual machines from all known vendors. 

    • Please note: This product-pitch, by Barnali Ganesh, co-founder of Coriolis, is featured on PuneTech because the editors found the technology interesting (or more accurately, the pain-point which it is addressing). PuneTech is a purely non-commercial website and does not take any considerations (monetary or otherwise) for any of the content on the site. If you would like your product to be featured on the front page, send us an article and we’ll publish it if we fell it’s interesting to our readers. You can always add a page to the PuneTech wiki by yourself, as long as you follow the “No PR” guidelines.

    Advice for entrepreneurs – Gireendra Kasmalkar

    Featured, In Depth, ,

    Gireendra Kasmalkar (Giri), whose testing startup VeriSoft was recently acquired by SQS, was interviewed for CSI Pune’s quarterly newsletter Desktalk. Some quotes from that interview that should be especially inspirational are reproduced here with permission. You should be able to download the the newsletter (containing the full article) from CSI Pune website’s download section. (It’s the July-Aug-Sept 2008 newsletter.)

    About how your attitude towards something makes all the difference:

    Initially, testing was a new area and it was a tough to convince people about it. Even today testing is sometimes looked down upon. My response: “I am glad you think of testing that way. That is what creates the opportunities for us!”.

    About how to deal with commoditization in your domain:

    But the testing industry was maturing. Every company now had an ‘independent’ testing practice. To maintain our leadership, we had to specialize further. We did that horizontally in such specialized areas as automation, load / performance, security and usability testing, even publishing papers in international conferences.

    But we also realized we had to develop a vertical. It had to be a vertical right for our size and where we could have a shot at global leadership. After due consideration, we decided that Games was that vertical. It was not at all easy to enter this vertical. But today we test for top 10 publishers in the video games industry and have authorization with all 3 major console manufacturers – Nintendo, Sony and Microsoft. In the on-line games industry, we not only test for game developers / publishers, but also for regulatory bodies for their compliance.

    On how a small startup can make an impact:

    For an entrepreneur, focusing on a niche is crucially important. One can hope to compete and beat the bigger, already established players only in a niche of your strength. More often than not, this niche would be a new area, and the entrepreneur only has his / her own data and analysis to see the hidden potential in the new area.

    On the importance of networking for success:

    This is where networking plays a very important role. Networking in the industry can bring you data about the trends in the industry. And you can even get your analysis of that data, and your ideas validated using your network. I was lucky to have the IIT network, the Jnana Prabodhini network, the TCS network because of my earlier background. But I also actively worked to build new networks through forums such as SPIN, CSI and MCCIA. In fact, there was a group of 8-10 successful industry seniors, whom I met once a year, by formal appointment, specifically for discussing my activities and seeking their suggestions. To their credit, none of these people ever refused my request. I definitely became a better entrepreneur as a result of these meetings.

    From my own experience, I strongly recommend active participation in forums such as CSI.

    I particularly like the suggestion about meeting 8 to 10 successful industry seniors on an yearly basis. I had started doing this with about 2, and based on Giri’s suggestion, I plan on increasing that to 8 to 10.

    And I certainly agree with active participation in forums. Giri suggests CSI-Pune. I would strongly suggest getting involved in the Pune OpenCoffee Club. There are a whole bunch of other groups and organizations where you could get involved. Check out the Groups and Organizations Page from the PuneTech wiki.

    Final word from Giri:

    To me, entrepreneurship is less about making money and more about making a difference. (There must be easier ways of making money .) An entrepreneur provides a viable solution to a problem (need). On the flip side, the entrepreneur’s capabilities are stretched to the maximum extent possible, which I find very satisfying.

    About the Interviewee – Gireendra Kasmalkar

    Gireendra Kasmalkar (Giri) is the Founder Director of Prabodhan and Verisoft InfoSystems, and after the recent acquisition of Verisoft, is the MD and CEO of SQS India InfoSystems Pvt. Ltd. Giri has been the Chairman of CSIPune chapter for 2007-2008 and is also actively associated with other relevant industry forums like SPIN and MCCIA.

    Despite having his hands full, professionals working with Giri will vouch for the energy, responsiveness and maturity that he brings forth to any activity.

    For a full profile and links, see Giri’s profile on the PuneTech wiki.

    Reading list for startup-founders

    In Depth, , ,

    Earlier, we published a list of free web services that a company can use to better run their business. However, tools do not really make or break a company. Having the right attitude, and making the right decisions at the right time is much more important. If you are a startup founder, it is not necessary that you learn everything from your mistakes at the school of hard knocks. You can try to benefit from the experiences of countless others who have learnt from their own successes and failures, and have survived to write blog posts about them. 

    To educate you, Sukshma.net (the blog of Santosh and Anjali, founders of Bookeazy and Lipikaar) have put together this list of articles and books that they found useful in their own journey. Here is a full list, shamelessly plagiarized from there, (to save my lazy readers from one extra click, I hope they don’t mind):

    Think Big!

    1st Month: Bullet-proof your idea.

    2nd Month: Raise Money, or scrape some together.

    3rd Month: Define your value proposition.

    4th Month: Iterative Innovation.

    5th Month: Advertising.

    7th Month: Momentum

    • 7 months in to my 2nd stint as Startup CEO – Jason Goldberg. Sorry. This link has gone off the internet. It used to be at http://blog.socialmedian.com/2008/08/7_months_into_my_2nd_stint_as.html but the whole blog disappeared after Jason sold social|median to Xing. If you’re able to find a copy of this article, please let us know. Lot’s of Pune people want to read it.

    9th Month: Profit for sustainability.

    Raising serious money from Professional Investors/Venture Capitalists.

    Master your disruptive contribution.

    Managing up and down.

    Other books worth reading.

    The Economics of Media and E-Commerce businesses.

    I need inspiration. I need to know this has happened to others. It’s never too late to start over.

    Read Founders@Work. Just turn to any story, any page and start reading. This is not to be read in one go, it’s most useful when you think you’re a complete idiot. Also, don’t forget to read Guy Kawasaki’s list of best pages from the book. Make your own list. We both have our list that we’ll put on the blog someday.

    Startup Resources from around the web.

    Remember to check the source, as they are promising to keep updating that list based on your suggestions (leave a comment).

    Pune’s too expensive, outsource to Nashik – Interview of Sushrut Bidwai (StartupForStartups)

    In Depth, Interview, , ,

    Nashik based StartupForStartups (SFS) is marketing itself as a “facility to help early stage companies with limited resources to build first cut of the product (V0.5/V1.0)”. Founder Sushrut Bidwai is a regular fixture at Pune startup events, and is trying to convince Pune-based startups to outsource work to his programmers in Nashik, promising that it will be cheaper than doing it themselves. PuneTech interviewed Sushrut to get a first-hand take on SFS’ value proposition.

    Can you give an overview of StartupForStartups?
    Many times people have good ideas, but dont have guts OR are shouldering family responsibilities, which does not allow them to pursue these ideas further. In some cases none of the founding team members are from tech background, so even getting a good CTO is difficult for them (salary wise as well skill wise) . StartupForStartups (SFS for short) is meant for such teams. Its better to have something ready before taking the risk of quitting (high paying) jobs. It gives you more insight into product you are building as well domain you are targeting.
    What SFS does is, it provides you resources required to build that beta version which you will show to investors (if its a big product) or will launch to limited audience and see how market reacts to it. After having this beta ready and some reactions from market or investors it becomes easier to take the risk and pursue it further full time.

    Why are you doing this in Nashik? I would have thought that being in Pune or Bangalore (near all the startups, who are your customers) would make more sense for you?
    Problems with Pune and Banglore are operational costs and resource costs. Also Nasik will have lower attrition rates and keeping people happy is easier. With technologies pushing the boundaries, we have so many tools available which makes working in distributed teams far easier. We even can do pair programming with two people sitting 5000 miles away from each other using WebEx/dimdim/Skype. Also it provides lot of cost advantages to startups we are working with.

    As a customer, one of the worries I would have with StartupForStartups, is the availability of quality talent in Nashik. How are you tackling this?
    We have developed a unique training program called “Implementing Concepts” which all our engineers go through before joining any startup team. So even if a particular engineer has gone through it once for a particular project, he/she will go through it again using the technologies which are going to be used in new project. This kind of a very elaborate HelloWorld for a project. Also, my experience in working on tech products is 80% of work is trivial and 20% is core work which is complex. So even if a startup is working on a product which is complex, they can take help of our resource in rest of 80% work. Though this does not mean we do not have expertise to take up complex work, it just means we are flexible and are okay with working as part of larger team. Also this is not outsourcing model, it is collaboration model. So you know who is working on your product and what that persons skills are and you can choose from the pool available. Also, to keep high availability of quality talent we are in process of collaborating with colleges here in Nasik. In this we will take the training program to colleges and have students go through them while working on final year projects. Please note that we do not assign interns on the projects.

    Considering that most of your customers are early-stage startups who are strapped for cash, how do you plan on charging them for your services?
    Charging is transparent. We send details about salaries we are paying to engineers assgined assigned to work on a particular startup and plus typically 20% operational costs. Now if you consider Nasik and typical salaries engineers expect and are more than happy with are much lesser than in Pune/Banglore. We already have the infrastructure and are building team. We are planning to build a team of 12-15 people by Dec end.

    Could you give us an idea of what kind of savings I can expect compared to outsourcing to a company in Pune? (Where are the savings coming from: lower salaries, or other factors too?)
    Lower costs for resources.
    Lower operational costs. Just to give you indicator of savings, an Entry level GWT/J2EE programmer will draw salaries in the range of 17-22K in Pune/Banglore same programmer if he/she is from Nasik will be more than content to work on a good startup team if given around 9-10K salary. Plus you have to keep him/her happy so there is no attrition. Spend money on infrastructure like office space/ furniture/ hardware/ software / electricity / lunch facilities and many non-tangible costs like FBTs/Mediclaim facilities etc etc. you save these costs by almost 70%.

    What do you see as your key competencies?

    • Experience of working in large scale product companies as well as early stage startups.
    • Top management has excellent problems solving and product designing skills.
    • Understand working on an Idea and processes involved in the same.
    • We are young, enthu, full of energy and love working on good ideas. (This is probably the most important quality.)

    In the context of StartupForStartups, I’ve heard you talk about having a startup ecosystem. Can you elaborate on that?
    This eco system is for people who are still in jobs and want pursue ideas. Though part of it can be used by full time entrepreneurs.

    • We are signing an MoU with a financial and legal service provider company having experienced in handling these services for startups.
    • We are creating pool of consultants (Architects, Performance engineering, Marketing/Advertising, HR, Viral Video creators etc).
    • We are collaborating with engineering colleges here in Nasik. We have designed an unique training program “Implementing Concepts” which is focused on training engineering students with latest technology and early stage product engineering.
    • We are talking to people who have the expertise of providing mentoring to early stage companies even before product is built OR evolved. Though we are in early stages of discussions with these people, but hopefully it will happen.
    • We are building a tool (looking to raise funds for this tool) which is designed around a process called Super-Agile, which I will be publishing shortly. This tool and process are targeted for early stage product development. The tool will make writing code almost a trivial thing and even non-tech background people after a little training can build the products first cut on their own.
    • Network. Not all startup founders are well connected. It takes lot of time to connect with people who can provide you help in building the startup. We can help there by connecting you with people we know, so its some starting point.
    • Knowledge-base. Startup entrepreneurs does not have time to go through lengthy tax stuff etc. Or does not have time to design their documents like Offer letters, seperation letters, NDAs. Over a period of time we will collect such documents and put them in an inventory. This inventory can be very valuable to startups.

    Note that all the services mentioned above does not necessarily come with a price tag. Some are out of goodwill some are for money 🙂 . Our main problem is we are young, have the skillsets necessary to pursue an idea and make it to successful business. But we do not have the idea. So we want to work with people who do have it.

    For those interested in meeting, Sushrut is in Pune today (25th September 2008) and is likely to attend the CSI Pune Seminar on Entrepreneurship. I know that there are a bunch of PuneTech readers who have an idea for a startup but haven’t made much progress yet because they haven’t been able to quit their job and take the plunge. If you are one of those, would you be willing to outsource some of this development to StartupForStartups or a similar company? Do you think this model will work? Let us know in the comments section below.

    Related Links

    Use Google Insights to find a niche market for your (non-web) product

    In Depth, , ,
    Image representing Google Labs as depicted in ...Image via CrunchBase, source unknown

    (In this interesting article, Trevas of Druvaa uses keyword search trending data from Google Insights and Google Labs Experimental Search to fine-tune his idea of what exactly is the market niche into which his products are most likely to have a demand.

    While search term analysis is a very common technique used by web-based companies for search engine optimization and finding long-tail customers, what is surprising in this case is that the products Trevas wants to sell have nothing to do with the web. He is using the keyword analysis to simply get a feel for which needs of users seem to have been met in the last few years, and which needs seem to be increasingly unmet. That gives him ideas for potential niche markets in which to position his products. Even if you have no interest in laptop backup and disaster recovery and the other terms used in this article, you should still read the article to get a hang of the technique, which can be applied in other fields. This article first appeared at Druvaa’s blog and is reproduced with permission. For more information about Druvaa and its technology, see this in-depth punetech article.)

    While doing some keyword research for Druvaa it began to become clear how interesting search engine statistic can be when you look closely at the data. From simple keyword suggestion tools, and graphs you can ascertain information that you never thought possible.

    The terms “backup” or “recovery”, for instance, get over 300,000 searches per month each with Google. In other words people are searching for good solutions to keep their data safe. That information by itself is useful (at least to us), but it’s when you begin to look at more specific search terms that things really get interesting. In fact, you can even begin to clearly see trends within the industry when you compare specific terms over any given length of time.

    With a look at some simple charts, you can begin to see things like:

    • Interest in laptop backup solutions has greatly increased over the past 10 years.
    • Some users are finding solutions to their data backup needs and disaster recovery isn’t as much of a problem as it was 4 years ago (but it still is a problem).
    • Enterprise users who have laptops in the office are still seeking a suitable solution to their backup needs.
    • Enterprise users who have offsite backup needs are still seeking a solution to business continuity.

    To demonstrate how I can get all of that from a few search terms, let’s take a closer look at some charts.

    A Look at Trends Using Search Engine Statistics

    Using Google Labs and their experimental search tool you come up with the following charts for the terms “data backup” and “laptop backup”.  This particular tool uses search volumes, online news statistics, number of websites, and more to show interest in any given topic. The charts clearly show that, while data backup has retained the same amount of interest over the past 10 years, interest in laptop backup has (and is) increasing.

    Of course, this idea makes sense. Laptops have decreased greatly in price since 1998, and as such have become a more common tool both for enterprise users and at home. On the other hand, data integrity has been a problem for business users for a couple of decades now, so interest in the topic of “data backup” have remained relatively the same.

    This information alone isn’t necessarily new. It’s the reason we created Druvaa InSync in the first place. The industry needed a reliable data backup solution, which is also fast enough to work well with computers that are on the go.  To further look at what’s needed let’s look at some more charts. This time based on search volumes alone.

    Laptop Backup as Important as Ever

    Search volumes for any given term are an easy way to see what is happening within an industry, to gauge interest for a product or service, or even to see how one product relates to another. In the developed world more than 73% of the population has internet access, and over 88% of internet users go online when they seek a solution to a problem.

    With that in mind let’s briefly look at some search engine statistic.  In this case I have used Google Insights to compare related search terms. The charts are based on normalized data, over time. If you looked at the actual search volumes they would have increased with time (since Internet use has grown). To get a more accurate look, Insights uses normalized data displayed on a scale of 1 – 100.

    Click Here to See the Chart for Yourself

    The first chart compares the terms data backup and disaster recovery. There are two things that can be gained from this chart.

    • 1. Since search volumes for both terms have declined over the past few years, it shows that some users are finding solutions to their backup needs, and disaster recovery is less of a problem today than it was in 2004/2005.
    • 2. As the lines of the chart come together, they begin to show a direct correlation to each other. Very likely this is due to the fact that proper data backup is becoming the solution to disasters in the office. It really was only a few years ago that disaster recovery often meant taking that broken hard drive to have the data extracted. In the past couple of years, enterprise users have begun to see that simple backups are a cheaper (and more reliable) solution.

    Since the term data backup may also relate to home users, with the next chart I used the term “enterprise backup” and compared it to “laptop backup”. Again we can see a couple of things from this chart. Once again we see a slight decline in the search volumes for enterprise backup. This confirms the idea that some enterprise users are finding a suitable solution to their backup needs.

    Click Here to See the Chart for Yourself

    By adding the term laptop backup though, something else begins to become clear. The term started the chart off at 61 and finished three years later at 62. There have been slight ups and downs in search volumes, but overall they have remained relatively the same. The two terms also begin to correspond closely with each other as the chart moves through 2007 and into 2008. To me this says that these terms are also beginning to become synonymous.  In other words, although some enterprise users are finding a backup solution, those with laptops in the office aren’t.

    I could repeat these same results with terms like “offsite backup” or “remote backup”.

    With a simple look at search engine statistics we begin to see that enterprise users have a need for a laptop backup solution that works. With our own product, which provides 10x faster laptop backup and a 90% reduction in storage and bandwidth, there is a solution to suit.

    Reblog this post [with Zemanta]

    Why do we need server virtualization

    In Depth, Technology, , , ,

    Virtualization is fast emerging as a game-changing technology in the enterprise computing space. What was once viewed as a technology useful for testing and development is going mainstream and is affecting the entire data-center ecosystem. This article on the important use-cases of server virtualization by Milind Borate, is the second in PuneTech’s series of articles on virtualization. The first article gave an overview of server virtualization. Future articles will deal with the issue of management of virtual machines, and other types of virtualization.

    Introduction

    Is server virtualization a new concept? It isn’t, because traditional operating systems do just that. An operating system provides a virtual view of a machine to the processes running on it. Resources are virtualized.

    • Each process gets a virtual address space.
    • A process’ access privileges control what files it can access. That is storage virtualization.
    • The scheduler virtualizes the CPU so that multiple processes can run without conflicting with each other.
    • Network is virtualized by providing multiple streams (for example, TCP) over the same physical link.

    Storage and network are weakly virtualized in traditional operating systems because some global state is shared by all processes. For example, the same IP address is shared by all processes. In case of storage, the same namespace is used by all processes. Over time, some files/directories become de-facto standards. For example, all process look at the same /etc/passwd file.

    Today, the term “server virtualization” means running multiple OSs on one physical machine. Isn’t that just adding one more level of virtualization? An additional level generally means added costs, lower performance, higher maintenance. Why then is everybody so excited about it? What is it that server virtualization provides in addition to traditional OS offerings? An oversimplified version of the question is: If I can run two processes on one OS, why should I run two OSs with one process each? This document enumerates the drivers for running multiple operating systems on one physical machine, presenting a use case, evaluating the virtualization based solution, suggesting alternates where appropriate and discussing future trends.

    Application Support

    Use case: I have two different applications. One runs on Windows and the other runs on Linux. The applications are not resource intensive and a dedicated server is under-utilized.

    Analysis: This is a weak argument in an enterprise environment because enterprises want to standardize on one OS and one OS version. Even if you find Windows and Linux machines in the same department, the administrators are two different people. I wonder if they would be willing to share a machine. On the other hand, you might find applications that require conflicting versions of, say, some library, especially on Linux.

    Alternative solution: Wine allows you to run Windows applications on Linux. Cygwin allows you to run Linux applications on Windows. Unfortunately, it’s not the same as running the application directly on the required OS. I won’t bet that a third party application would run out of the box under these virtual environments.

    Future: Some day, developers will get fed up of writing applications for a particular OS and then port them to others. JAVA provides us with an host/OS independent virtual environment. JAVA wants programmers to write code that is not targetted for a particular OS. It succeeded in some areas. But, still there is a lot of software written for a particular OS. Why did everybody not move to JAVA? I guess, because JAVA does not let me do everything that I can do using OS APIs. In a way, that’s JAVA’s failure in providing a generic virtual environment. In future, we will see more and more software developed over OS independent APIs. Databases would be the next target for establishing generic APIs.

    Conflicting Applications

    Use case: I have two different applications. If I install both on the same machine, both fail to work. In fact, they might actually work but it’s not a supported by my vendor.

    Analysis: In the current state of affairs, an OS is not just hardware virtualization. The gamut of libraries, configuration files, daemons is all tied up with an OS. Even though an application does not depend on the exact kernel version, it very much depends on the library versions. It’s also possible that the applications make conflicting changes to some configuration file.

    Alternative solution: OpenVZ modifies Linux to provide multiple “containers” inside the same OS. The machine runs a single kernel but provides multiple isolated environments. Each isolated environment can run an application that would be oblivious to the other containers.

    Future: I think, operating systems need to support containers by default. The process level isolation provided at memory and CPU level needs to be extended storage and network also. On the other hand, I also hope that application writers desist from depending on shared configuration and shared libraries pay some attention to backward compatiblity.

    Fault Isolation

    Use case: In case an application or the operating system running the application faults, I want my other applications to run unaffected.

    Analysis: A faulty application can bring down entire server especially if the application runs in a priviledged mode and if it could be attacked over a network. A kernel driver bug or operating system bug also brings down a server. Operating systems are getting more stable and servers going down due to operating system bug is rare now a days.

    Alternative solution: Containers can help here too. Containers provide better isolation amongst applications running on the same OS. But, bugs in kernel mode components cannot be addressed by containers. Future: In near future, we are likely see micro-kernel like architectures around virtual machines monitors. Light weight operating systems could be developed to work only with virtual machine monitors. Such a solution will provide fault isolation without incurring the overheads of a full opearting system running inside a virtual machine.

    Live Application Migration

    Use case: I want to build a datacenter with utility/on-demand/SLA-based computing in mind. To achieve that, I want to be able to live-migrate an application to a different machine. I can run the application in a virtual machine and live-migrate the virtual machine.

    Analysis: The requirement is to migrate an application. But, migrating a process is not supported by existing operating systems. Also, the application might do some global configuration changes that need to be available on the migration target.

    Alternative solution: OpenVZ modifies Linux to provide multiple “containers” inside the same OS. OpenVZ also supports live migration of a container.

    Future: As discussed earlier, operating systems need to support containers by default.

    Hardware Support

    Use case: My operating system does not support the cutting edge hardware I bought today.

    Analysis: Here again, I’m not bothered about the operating system. But, my applications run only on this OS. Also, enterprises like to use the same OS version throughout the organization. If an enterprise sticks to an old OS version, it does not work with new hardware to be bought. If an enterprise is willing to move to the newer OS, it does not work with the existing old hardware.

    But, the real issue here is the lack of standardization across hardware or driver development models. I fail to understand why every wireless LAN card needs a different driver. Can all hardware vendors not standardize the IO ports and commands so that one generic driver works for all cards? On the other hand, every OS and even OS version has a different drivers development model. That means every piece of hardware requires a different driver for each OS version. Alternative solution: I cannot think of a good alternative solution. One specific issue, unavailability of wireless LAN card drivers for Linux, is addressed by NdisWrapper. NdisWrapper allows us to access a wirelss card on Linux by loading a Windows driver.

    Future: We either need hardware level standardization or the ability to run the same driver on all verions on all operating systems. It would be good to have wrappers, like NdisWrapper, for all types of drivers and all operating systems. A hardware driver should write to a generic API provided by the wrapper framework. The generic API should be implemented by the operating system vendors.

    Software Development Environment

    Use case: I want to manage hardware infrastructure for software development. Every developer and QA engineer needs dedicated machines. I can quickly provision a virtual machine when the need arises.

    Analysis: Under development software fails more often than a released product. Software developers and QA engineers want an isolated environment for the tests to correctly attribute bugs to the right application. Also, software development envinronments require frequent reprovisioning as the product under development needs to be tested under different operating systems.

    Alternative solution: Containers would work for most software development. I think, the exception is kernel level development.

    Future: Virtual machines found an instant market in software QA labs. Virtual machines will continue to flourish in this market.

    Application Configuration Replica

    Use case: I want to ship some data to another machine. Instead of setting up identical application enviroment on the other machine to access the data, I want to ship the entire machine image itself. Shipping physical machine image does not work because of hardware level differences. Hence, I want to ship virtual machine image.

    Analysis: This is another hard reality of life. Data formats are not compatible across multiple versions of a software product. Portable data formats are used by human readable documents. File-system data formats are also stable to a large extent and you can mount a FAT file-system or ISO 8529 file-system on virtually any version of any operating system. The same level of compatiblity is not established for other structured data. I don’t see that happening in near future. Even if this hurdle is crossed, you need to bother about correctly shipping all the application configuration, which itself could be different for the same software running on different OSs.

    Alternative solution: OpenVZ container could be a light-weight alternative to a complete virtual machine.

    Future: The future seems inclined towards “computing in a cloud”. The network bandwidth is increasing and so is the trend towards is outsourced hosting. Mail and web services are outsourced since a long time. Oracle-on-demand allows us to outsource database hosting. Google (writely) wants us to outsource document hosting. Amazon allows us to outsource storage and compuation both. In future, we will be completely oblivious to the location of our data and applications. The only process running on your laptop would be an improved a web browser. In that world, only system software engineers, who build these datacenters, would be worried about hardware and operating system compatibilities. But, they also will not be overly bothered because the data-center consolidations will reduce the diversity in hardware and OS.

    Thin Client Desktops

    Use case: I want to replace desktop PCs with thin clients. A central server will run a VM for each thin client. The thin client will act as a display server.

    Analysis: Thin clients could bring down the maintenance costs substantially. Thin client hardware is more resilient than a desktop PC. Also, it’s easier to maintain the software installed on a central server than managing several PCs. But, it’s not required to run a full virtual machine for each thin client. It’s sufficient to allow users to run the required applications from a central server and make the central storage available.

    Alternative solution: Unix operating systems are designed to be server operating systems. Thin X terminals are still prevalent in Unix desktop market. Microsoft Windows, the most prevalent desktop OS, is designed as a desktop OS. But, Microsft also has added substantial support for server based computing. Microsft’s terminal services allows multiple users to connect to a Windows server and launch applications from a thin client. Several commercial thin clients can work with Microsoft terminal services or similar services provided by other vendors.

    Future: Before the world moves to computing in a global cloud, an intermediate step would be enterprise-wide desktop application servers. Thin-clients would become prevalent due to reduced maintenance costs. I hope to see Microsoft come up with better licensing for server based computing. On Unix, floating application licenses is the norm. With a floating application licence, a server (or a cluster of servers) can run only fixed application instances as per the license. It does not matter which user or thin client launches the application. Such a floating licensing from Microsoft will help.

    Conclusion

    Server virtualization is a “heavy” solution for the problems it addresses today.These problems could be adddressed by operating systems in a more efficient manner with following modifications:

    • Support for containers.
    • Support for live migration of containers.
    • Decoupling of hardware virtualization and other OS functionalities.

    If existing operating systems muster enough courage to deliver these modifications, server virtualization will have tough time. It’s unrealistic to expect complete overhauls of existing operating systems. It’s possible to implement containers as a part of OS but decoupling hardware virtualizatoin from OS is a hard job. Instead, we are likely to see new light weight operating systems designed to run only in server virtualization environment. The light weight operating system will have following characteristics:

    • It will do away with functionality already implemented in virtual machine monitor.
    • It will not worry about hardware virtualization.
    • It might be a single user operating system.
    • It might expect all co-operative processes.
    • It will have a minimal kernel mode component. It will be mostly composed of user mode libraries providing OS APIs.

    Existing virtual machine monitors would also take up more responsiblity in order to support light weight operating systems:

    • Hardware support: The hardware supported by a VMM will be of primary importance. The OS only needs to support the virtual hardware made visible by VMM.
    • Complex resource allocation and tracking: I should get a finer control over resources allocated to virtual machines and be able to track resource usage. This involves CPU, memory, storage and network.

    I hope to see a light weight OS implementation targetted at server virtualization in near future. It would a good step towards modularizing the operating systems.

    Acknowledgements

    Thanks to Dr. Basant Rajan and V. Ganesh for their valuable comments.

    About the Author – Milind Borate

    Milind Borate is the CTO and VP of Engineering at Pune-based continuous data protecting startup Druvaa. He has over 13 years experience in enterprise product development and delivery. He worked at Veritas Software as Technical Director for SAN-FS and served on board of Veritas patent filter committee. Milind has filed over 15 patent applications (4 alloted) and co-authored “Undocumented Windows NT” in 1998. He holds a BE (CS) degree from University of Pune and MTech (CS) degree from IIT, Bombay.

    This article was written when Milind was at Coriolis, a startup he co-founded before Druvaa.

    Related articles by Zemanta
    Reblog this post [with Zemanta]