PlanetAikon is a forum that allows “community powered innovation”. It is a Pune-based startup which allows its members to “connect, collaborate and co-create”. It strives to be a unique innovation ecosystem bringing together a community of ideators, contributors and sponsors – in other words, it connects the guys with the bright ideas, the guys with the money, and the guys with skills and time to implement the idea.
This is the story of TZen, one of PlanetAikon’s success stories. It is written by Prateek Dubey and originally appeared on the PlanetAikon Blog, and is republished here with permission.
TZen – The Tool whose Time has Come
Hi- I am Prateek Dubey and I have a Story to share with You – The Story of Tzen
The Inspiration and the Rationale
It all started when Zensar Technologies Testing Center of Excellence decided about 2 yrs ago to develop a test tool which will be used for Manual Testing for Projects in Zensar and to encourage use of Test Tools among associates. Add to that the prohibitive cost of branded Testing tools and the Training & Upgrade costs associated with it were the significant business drivers which strengthened our resolve to build such a tool.
We developed our first Test Tool prototype which was a Test Management Layer over an Opensource Defect Management Tool called “Bugzilla”, which was based on technologies like CGI, perl and mysql.
We soon realized that our prototype was not going to fulfil the future requirements of Test Management, as CGI was slow and handling large data in an efficient way was a challenge. This was the time when we started a new prototype which was based on new AJAX Technology. An article about a newly introduced web services concept called REST sparked further ideas. After a month of design and re-design we finally had a draft design ready. We chose php for implementing our web services to support the development of a quick, easy to understand and highly responsive tool. We wanted this application to be a full-fledged WebTop Application, keeping in mind that we were designing it for the future. We came up with a prototype which was named “TZen” by our project manager Vishal Wani. Vishal encouraged us to do a Demo of our Prototype to some internal projects. The Demo was an instant hit as nobody had ever seen a Web Application which loaded only once and then ran on the browser as a Desktop Application.
With Vishal’s and Testing Practice Head Prem Apte’s continued support, we started development of our product with help from Testing Community in Zensar. As we developed TZen further we were facing serious technology challenges as “Ajax” was a new technology and we were at par with rest of the world in exploration of the same. Thus, every time we ran into an issue we had to resolve it mostly by ourselves. We were also using Technologies like Yahoo UI library and Dojo, it took us around 6 months to come up with a first fully functional TZen prototype. We deployed it at various projects and got a lot of feedback on our implementation, TZen was still very basic in functionality and we still had a lot of work to be done. During this time we had a lot of organizational changes in team and finally we were left as a 2 member team from a 4 member team.
Reinventing TZen with Community Powered Innovation
TZen went slow for about 6 months until recently when it got revived by a Zensar Opensource Innovation initiative with the help of Planetaikon Platform. We started with significant doubts about what we could achieve through an Open Community involvement. But with support of NASSCOM (under their Innovation Initiatives) and the leadership at Zensar we soon found ourselves inundated by requests from individuals from other companies to join this initiative.
We had the usual initial teething issues as folks were adapting to working on a on-line collaborative co-creation platform (as against meeting face-to-face or the “business-as-usual” of e-mail exchanges).
We got a lot of feedback from participants which led us to shortlist 27 (twenty seven) new features to the existing version of our tool.
Those who contributed did so because of their passion to create something and did so in after-office hours and on weekends. In approximately 5 months time we were able to come up with our Release 1.1 with those 27 additional features which we have again contributed back to the community.
Our Vision is to develop TZen into the most successful Opensource Test Management Platform. Our Team size has now grown to 66 (Sixty Six)!
TZen is designed ground up to be Opensource and Agile development compliant product. It can be developed quickly and tested easily. This was recently re-integrated when a small set of students with only a few days of TZen exposure, created a “Commenting System” for TZen , which is going to be featured in the next TZen release again a validation of the value that communities can provide
TZen – A Snapshot
TZen provides most features that a Test Management Tools must have. These include:
Requirement Management
Test Plan Management
Test Case Management
Test Execution Management
Defect Management (integrated with Mantis)
Reports and Exports at various levels
The Road Ahead
As we progress, we look forward to the community to help us take this product to new heights. Some of the features we plan to include are
Extensive graph and Chart support
Flexible Imports
Custom Fields
Server pull services and more….
TZen has got a lot of business, technical and innovation opportunity ready to be explored and taken forward, improvements at every level are in progress.
We are now throwing open another call to professionals who are keen to participate in taking TZen to its next high. We are looking for Php 5 developers, Testing Domain Experts, Usability Domain Professionals and Testers.
Come add to our adrenalin and let us show the world how the audacity of Ideas can be supported by the passion of communities.
For Dr. Narayan Venkatasubramanyan’s detailed bio, please click here. For the full series of articles, click here.)
this is a follow-up to optimization: a case study. frequent references in this article to details in that article would make this one difficult to read for someone who hasn’t at least skimmed through that.
the problem of choice
the wikipedia article on optimization provides a great overview of the field. it does a thorough job by providing a brief history of the field of mathematical optimization, breaking down the field into its various sub-fields, and even making a passing reference to commercially available packages that help in the rapid development of optimization-based solutions. the rich set of links in this page lead to detailed discussions of each of the topics touched on in the overview.
i’m tempted to stop here and say that my job is done but there is one slight problem: there is a complete absence of any reference to helicopter scheduling in an offshore oil-field. not a trace!
this brings me to the biggest problem facing a young practitioner in the field: what to do when faced with a practical problem?
of course, the first instinct is to run with the technique one is most familiar with. being among the few in our mba program that had chosen the elective titled “selected topics in operations research” (a title that i’m now convinced was designed to bore and/or scare off prospective students who weren’t self-selected card-carrying nerds), we came to the problem of helicopter scheduling armed with a wealth of text-book knowledge.
the lines represent the constraints. the blue region is the set of all “permissible values”. the objective function is used to choose one (“the most optimal”) out of the blue points. image via wikipedia
having recently studied linear and integer programming, we first tried to write down a mathematical formulation of the problem. we knew we could describe each sortie in terms of variables (known as decision variables). we then had to write down constraints that ensured the following:
any set of values of those decision variables that satisfied all the constrains would correspond to a sortie
any sortie could be described by a set of permissible set of values of those decision variables
this approach is one of the cornerstones of mathematical programming: given a practical situation to optimize, first write down a set of equations whose solutions have a one-to-one correspondence to the set of possible decisions. typically, these equations have many solutions.
the other cornerstone is what is called an objective function, i.e., a mathematical function in those same variables that were used to describe the set of all feasible solutions. the solver is directed to pick the “best” solution, i.e., one that maximizes (or minimizes) the objective function.
the set of constraints and the objective function together constitute a mathematical programming problem. the solution that maximizes (or minimizes) the objective function is called an optimal solution.
linear programming – an example
googling for “linear programming examples” leads to millions of hits, so let me borrow an example at random from here: “A farmer has 10 acres to plant in wheat and rye. He has to plant at least 7 acres. However, he has only $1200 to spend and each acre of wheat costs $200 to plant and each acre of rye costs $100 to plant. Moreover, the farmer has to get the planting done in 12 hours and it takes an hour to plant an acre of wheat and 2 hours to plant an acre of rye. If the profit is $500 per acre of wheat and $300 per acre of rye how many acres of each should be planted to maximize profits?”
the decisions the farmer needs to make are: how many acres of wheat to plant? how many acres of rye to plant? let us call these x and y respectively.
so what values can x and y take?
since we know that he has only 10 acres, it is clear that x+y must be less than 10.
the problem says that he has to plant at least 7 acres. we have two choices: we can be good students and write down the constraint “x+y >= 7” or we can be good practitioners and demand to know more about the origins of this constraint (i’m sure every OR professional of long standing has scars to show from the times when they failed to ask that question.)
the budget constraint implies that 200x + 100y <= 1200. again, should we not be asking why this farmer cannot borrow money if doing so will increase his returns?
finally, the time constraint translates into x + 2y <= 12. can he not employ farm-hands to increase his options?
the non-negativity constraints (x, y >= 0) are often forgotten. in the absence of these constraints, the farmer could plant a negative amount of rye because doing so would seem to get him more land, more money, and more time. clearly, this is practically impossible.
as you will see if you were to scroll down that page, these inequalities define a triangular region in the x,y plane. all points on that triangle and its interior represents feasible solutions: i.e., if you were to pick a point, say (5,2), it means that the the farmer plants 5 acres each of wheat and 2 acres of rye. it is easy to confirm that this represents no more than 10 acres, no less than 7 acres, no more than $1200 and no more than 12 hours. but is this the best solution? or is there another point within that triangle?
this is where the objective function helps. the objective is to maximize the profit earner, i.e., maximize 500x + 300y. from among all the points (x,y) in that triangle, which one has the highest value for 500x + 300y?
this is the essence of linear programming. LPs are a subset of problems that are called mathematical programs.
real life isn’t always lp
in practice, not all mathematical programs are equally hard. as we saw above, if all the constraints and the objective function are linear in the decision variables and if the decision variables can take on any real value, we have a linear program. this is the easiest class of mathematical programs. linear programming models can be used to describe, sometimes approximately,a large number of commercially interesting problems like supply chain planning. commercial packages like OPL, GAMS, AMPL, etc can be used to model such problems without having to know much programming. packages like CPLEX can solve problems with millions of decision variables and constraints and produce an optimal solution in reasonable time. lately, there have been many open source solvers (e.g., GLPK) that have been growing in their capability and competing with commercial packages.
integer programming problems constrain the solution to specific discrete values. while the blue lines represent the “feasible region”, the solution is only allowed to take on values represented by the red dots. this makes the problem significantly more difficult. image via wikipedia
in many interesting commercial problems, the decision variables is required to take on discrete values. for example, a sortie that carries 1/3 of a passenger from point a to point b and transports the other 2/3 on a second flight from point a to point b would not work in practice. a helicopter that lands 0.3 in point c and 0.7 in point d is equally impractical. these variables have to be restricted to integer values. such problems are called integer programming problems. (there is a special class of problems in which the decision variables are required to be 0 or 1; such problems are called 0-1 programming problems.) integer programming problems are surprisingly hard to solve. such problems occur routinely in scheduling problems as well as in any problem that involves discrete decisions. commercial packages like CPLEX include a variety of sophisticated techniques to find good (although not always optimal) solutions to such problems. what makes these problems hard is the reality that the solution time for such problems grows exponentially with the growth in the size of the problem.
another class of interesting commercial problems involves non-linear constraints and/or objective functions. such problems occur routinely in situations such refinery planning where the dynamics of the process cannot be described (even approximately) with linear functions. some non-linear problems are relatively easy because they are guaranteed to have unique minima (or maxima). such well-behaved problems are easy to solve because one can always move along an improving path and find the optimal solution. when the functions involved are non-convex, you could have local minima (or maxima) that are worse than the global minima (or maxima). such problems are relatively hard because short-sighted algorithms could find a local minimum and get stuck in it.
fortunately for us, the helicopter scheduling problem had no non-linear effects (at least none that we accounted for in our model). unfortunately for us, the discrete constraints were themselves extremely hard to deal with. as we wrote down the formulation on paper, it became quickly apparent that the sheer size and complexity of the problem was beyond the capabilities of the IBM PC-XT that we had at our disposal. after kicking this idea around for a bit, we abandoned this approach.
resorting to heuristics
we decided to resort to a heuristic approach, i.e., an approach that used a set of rules to find good solutions to the problem. the approach we took involved the enumeration of all possible paths on a search tree and then an evaluation of those paths to find the most efficient one. for example, if the sortie was required to start at point A and drop off m1 men at point B and m2 men at point C, the helicopter could
leave point A with the m1 men and proceed to point B, or
leave point A with the m2 men and proceed to point C, or
leave point A with the m1 men and some of the m2 men and proceed to point B, or
leave point A with the m1 men and some of the m2 men and proceed to point C, or
. . .
if we were to select the first possibility, it would drop off the m1 men and then consider all the options available to it (return to A for the m2 men? fly to point D to refuel?)
we would then traverse this tree enumerating all the paths and evaluating them for their total cost. finally, we would pick the “best” path and publish it to the radio operator.
at first, this may seem ridiculous. the explosion of possibilities meant that this tree was daunting.
there were several ways around this problem. firstly, we never really explicitly enumerated all possible paths. we built out the possibilities as we went, keeping the best solution until we found one that was better. although the number of possible paths that a helicopter could fly in the course of a sortie was huge, there were simple rules that directed the search in promising directions so that the algorithm could quickly find a “good” sortie. once a complete sortie had been found, the algorithm could then use it to prune searches down branches that seemed to hold no promise for a better solution. the trick was to tune the search direction and prune the tree without eliminating any feasible possibilities. of course, aggressive pruning would speed up the search but could end up eliminating good solutions. similarly, good rules to direct the search could help find good solutions quickly but could defer searches in non-obvious directions. since we were limited in time, so the search tree was never completely searched, so if the rules were poor, good solutions could be pushed out so late in the search that they were never found, at least not in time to be implemented.
one of the nice benefits of this approach was that it allowed the radio operator to lock down the first few steps in the sortie and leave the computer to continue to search for a good solution for the remainder of the sortie. this allowed the optimizer to continue to run even after the sortie had begun. this bought the algorithm precious time. allowing the radio operator the ability to override also had the added benefit of putting the user in control in case what the system recommended was infeasible or undesirable.
notice that this approach is quite far from mathematical programming. there is no guarantee of an optimal solution (unless one can guarantee that pruning was never too aggressive and that we exhaustively searched the tree, neither of which could be guaranteed in practical cases). nevertheless, this turned out to be quite an effective strategy because it found a good solution quickly and then tried to improve on the solution within the time it was allowed.
traditional operations research vs. artificial intelligence
this may be a good juncture for an aside: the field of optimization has traditionally been the domain of operations researchers (i.e., applied mathematicians and industrial engineers). even though the field of artificial intelligence in computer science has been the source of many techniques that effectively solve many of the same problems as operations research techniques do, OR-traditionalists have always tended to look askance at their lowly competitors due to the perceived lack of rigour in the AI techniques. this attitude is apparent in the wikipedia article too: after listing all the approaches that are born from mathematical optimization, it introduces “non-traditional” methods with a somewhat off-handed “Here are a few other popular methods:” i find this both amusing and a little disappointing. there have been a few honest attempts at bringing these two fields together but a lot more can be done (i believe). it would be interesting to see how someone steeped in the AI tradition would have approached this problem. perhaps many of the techniques for directing the search and pruning the tree are specific instances of general approaches studied in that discipline.
if there is a moral to this angle of our off-shore adventures, it is this: when approaching an optimization problem, it is tempting to shoot for the stars by going down a rigorous path. often, reality intrudes. even when making technical choices, we need to account for the context in which the software will be used, how much time there is to solve the problem, what are the computing resources available, and how it will fit into the normal routine of work.
other articles in this series
this article is the fourth in the series of short explorations related to the application of optimization. i’d like to share what i’ve learned over a career spent largely in the business of applying optimization to real-world problems. interestingly, there is a lot more to practical optimization than models and algorithms. each of the the links leads to a piece that dwells on one particular aspect.
Dr. Narayan Venkatasubramanyan has spent over two decades applying a rare combination of quantitative skills, business knowledge, and the ability to think from first principles to real world business problems. He currently consults in several areas including supply chain and health care management. As a Fellow at i2 Technologies, he tackled supply chains problems in areas as diverse as computer assembly, semiconductor manufacturer, consumer goods, steel, and automotive. Prior to that, he worked with several airlines on their aircraft and crew scheduling problems. He topped off his days at IIT-Bombay and IIM-Ahmedabad with a Ph.D. in Operations Research from the University of Wisconsin-Madison.
He is presently based in Dallas, USA and travels extensively all over the world during the course of his consulting assignments. You can also find Narayan on Linkedin at: http://www.linkedin.com/in/narayan3rdeye
For Dr. Narayan Venkatasubramanyan’s detailed bio, please click here. For the full series of articles, click here.)
this is a follow-up to optimization: a case study. frequent references in this article to details in that article would make this one difficult to read for someone who hasn’t at least skimmed through that.
organizational dynamics
most discussions of optimization tend to focus on the technical details of problem formulation, algorithm design, the use of commercially available software, implementation details, etc. a fundamental point gets lost in that approach to this topic. in this piece, we will focus on that point: organizational readiness for change.
the introduction of optimization in the decision-making process almost always requires change in that process. processes exist in the context of an organization. as such, when introducing change of this nature, organizations need to be treated much the same way a doctor would treat a recipient of an organ. careful steps need to be take to make sure that the organization is receptive to change. before the change is introduced, the affected elements in the organization need to be made aware of the need for change. also, the organization’s “immune system” needs to be neutralized while the change is introduced. the natural tendency of any organization to attack change and frustrate the change agent needs to be foreseen and planned for.
the structure of the client’s project organization is critical. in my experience, every successful implementation of optimization has required support at 3 levels within the client organization:
a project needs “air cover” from the executive level.
at the project level, it needs a champion who will serve as the subject-matter expert, evangelist, manager, and cheerleader.
at the implementation level, it needs a group of people who are intimately familiar with the inner workings of the existing IT infrastructure.
let me elaborate on that with specific emphasis on the first two:
an executive sponsor is vital to ensuring that the team is given the time and resources it needs to succeed even as changes in circumstances cause high-level priorities to change. during the gestation period of a project — a typical project tends to take several months — the project team needs the assurance that their budget will be safe, the priorities that guide their work will remain largely unchanged, and the team as a whole will remain free of distractions.
a project champion is the one person in the client organization whose professional success is completely aligned with the success of the project. he/she stands to get a huge bonus and/or a promotion upon the success of the project. such a person keeps the team focused on the deliverable, keeps the executive sponsor armed with all the information he/she needs to continue to make the case for the project, and keeps all affected parties informed of impending changes, in short, an internal change agent. in order to achieve this, the champion has to be from the business end of the organization, not from the IT department.
unfortunately, most projects tend to focus on the third of the elements. strength in the implementation team alone will not save project that lacks a sponsor or a champion.
ONGC’s HAL Dhruv Helicopters on sorties off the Mumbai coast. Image by Premshree Pillai via Flickr
it could be argued that executive sponsorship for this project came from the highest possible level. i heard once that our project had been blessed by the managing directors of the two companies. unfortunately, their involvement didn’t extend anywhere beyond that. neither managing director helped shape the project organization for success.
who was our champion? there was one vitally important point that i mentioned in passing in the original narrative: the intended users of the system were radio operators. they reported to an on-shore manager in the electronics & telecommunication department. in reality, their work was intimately connected to the production department, i.e., the department that managed the operations in the field. as such, they were effectively reporting to the field production supervisor. the radio operators worked very much like the engineers in the field: they worked all day every day for 14 days at a time and then went home for the next 2 weeks. each position was manned by two radio operators — more about them later — who alternately occupied the radio room. as far as their helicopter-related role was concerned, they were expected to make sure that they did the best they could do to keep operations going as smoothly as possible. their manager, the person who initiated the project, had no direct control over the activities of the radio operator. meanwhile, the field production supervisor was in charge of maintaining the efficient flow of oil out of the field. the cost of helicopter operations was probably a miniscule fraction of the picture they viewed. because no one bore responsibility for the efficiency of helicopter usage, no one in the client organization really cared about the success of our project. unfortunately, we were neither tasked nor equipped to deal with this problem (although that may seem odd considering that there were two fresh MBAs on the team).
in hindsight, it seems like this project was ill-structured right from the beginning. the project team soldiered on in the face of these odds, oblivious to the fact that we’d been dealt a losing hand. should the final outcome have ever been a surprise?
other articles in this series
this article is the third in a series of short explorations related to the application of optimization. i’d like to share what i’ve learned over a career spent largely in the business of applying optimization to real-world problems. interestingly, there is a lot more to practical optimization than models and algorithms. each of the the links below leads to a piece that dwells on one particular aspect.
Dr. Narayan Venkatasubramanyan has spent over two decades applying a rare combination of quantitative skills, business knowledge, and the ability to think from first principles to real world business problems. He currently consults in several areas including supply chain and health care management. As a Fellow at i2 Technologies, he tackled supply chains problems in areas as diverse as computer assembly, semiconductor manufacturer, consumer goods, steel, and automotive. Prior to that, he worked with several airlines on their aircraft and crew scheduling problems. He topped off his days at IIT-Bombay and IIM-Ahmedabad with a Ph.D. in Operations Research from the University of Wisconsin-Madison.
He is presently based in Dallas, USA and travels extensively all over the world during the course of his consulting assignments. You can also find Narayan on Linkedin at: http://www.linkedin.com/in/narayan3rdeye
Ajit Shelat, Senior Vice President of Engineering, Nevis Networks
Nevis Networks, a mostly-Pune-based-company (with “official” headquarters in the US, and an additional center in China), builds network switches and other network hardware that allows a company to secure it’s internal network from attacks and to enforce identity-based security policies. The company’s LANenforcer product family transparently protects the network from external malicious attacks, and also allows restricting access to different network resources based on users’ identities according to policies set by the system administrators. This can be customized to ensure different levels of access to different classes of users, employees, contractors, guests and other third parties. In addition, the product allows detailed reporting, auditing, employee activity reports that make it possible to analyze security breaches in very granular detail. And because it is hardware based, all of this is delivered in realtime with very low latency.
Nevis Networks’ customers range from financial services, healthcare, education and defense contractors and they deploy Nevis LANenforcers to protect sensitive network resources and assets, with an intention of reducing the overall costs and time to resolve security breaches and conduct network audits. The company is headquartered in Mountain View, CA, with additional R&D centers in Pune, India and Beijing, China.
The ongoing recession has hit Nevis Networks hard, and it downsized a very large fraction of its workforce late last year. On top of that, on Monday, in a report title “LSI Acquires Manpower Team of Navis Networking”, CXOToday implied that the company (which they alternately identified as Navis Networks or Nevis Networks in the article) had shutdown and the team taken over by LSI. Specifically, this is what CXOToday said:
With recession being an opportunity to invest for big MNCs, LSI Technologies, a provider of innovative silicon, systems and software technologies has acquired the team of Navis Networking based at Pune. With the R&D unit based out of Mountain View, California shutting down, LSI has acquired the manpower of the captive R&D centre in India.
After hearing from PuneTech readers that this report is misleading, we caught up with Ajit Shelat, Senior Vice President of Engineering for Nevis Networks, to learn that the reports of Nevis’ demise have been greatly exaggerated. Here is a quick report of the conversation we had with Ajit:
On the news that LSI has “acquired” the “manpower” of Nevis but not the company.
The report by CXOToday is misleading. What actually happened is much simpler. Due to the economic downturn last year, Nevis Networks was looking to downsize some of its workforce. A friendly interaction between the respective managements of Nevis and LSI led to movement of some of Nevis manpower to LSI. This was a simple case of Nevis ex-employees being hired by LSI en masse. It does not represent any sort of acquisition or even agreement between Nevis and LSI. And these are certainly not the entire team of Nevis Networks India, as implied by the CXOToday article.
In any case, Nevis networks is not shutting down. It continues to execute on a with strategy and focus.
On the current status of Nevis Networks
Nevis networks core team is still there and it is going strong. In fact, the last quarter was quite good and has been the best quarter for Nevis since the inception of the company.
What has happened is that due to the downturn, Nevis shifted its focus away from the US market to the India and China markets, reduced its workforce in the US and in India, and this new strategy appears to be working for them.
On the surprising fact that India/China are better markets than the US market
Since Nevis Networks is selling cutting edge technology, one would have expected US to be the logical market for these products. However, people really underestimate the extent of the effect the economic recession is having on the market there. While the markets really melted around September 2008, the signs have been obvious for at least an year before that, and starting Nov/Dec 2007, Nevis had started planning its strategy of shifting focus away from the US market to the India/China markets.
In tune with their new strategy, Nevis substantially reduced its India workforce. They continue to support existing customers in the US, but new customers are coming mainly from India – which is apparently not affected by the recession as much. In general, it is easier for a company with mainly Indian promoters to sell in India than in other countries.
China is another country where sales are expected to grow – Nevis is in the process of stengthening its sales presence in China. The Chinese market, having a significantly different character, takes a longer ramp up time to achieve its full potential – though a very good start has been made in terms of immediate sales. Like other markets, achieving full potential is really a function of getting the right people on the ground, and building the right relationships and customer confidence. All this effort is justified by the fact that the Chinese market has the potential to scale up dramatically.
More about Nevis Networks
Nevis Networks was founded in 2002 with the intention of building a network security solution with high speed and low latency, using its proprietary ASIC-based technology. As of last year, Nevis had raised a total of US$40 million in three rounds of funding from premier venture capital firms New Enterprise Associates, BlueRun Ventures (formerly Nokia Venture Partners) and New Path Ventures LLC. We are told that their funding situation has recently changed and an announcement to this effect is expected in the next couple of weeks.
Joomla! Day India, will be big. Toby Patterson, a member of the Joomla! Core team development workgroup member (Update: Sorry about misidentifying Toby – Live tweets from the event indicate that his keynote was great), will be giving the keynote speech. Members of the Joomla! bug squad will be there. And they are coming in from outside the country, so if you miss them this time, you are not going to get a chance to interact with them in the next run-of-the-mill barcamp that happens in Pune, or even Bombay/Bangalore. So what’s your excuse for not going? Earlier, due to the costs of having external visitors for this event, the price of entry was Rs. 1000. But thanks to the organizers finding appropriate sponsors, everything, including the prizes worth $1600, the T-shirts, the lunch/snacks and other goodies are FREE!(Update: The T-shirts, snacks and other goodies are NOT FREE. They had to be removed to make the event free. Sorry about the misinformation.)
It is tomorrow, Saturday, 25th April, from 9am to 6pm at I2IT campus, Hinjewadi. These are the sessions planned:
Keynote By Toby Patterson
Session 1: Hello Joomla! – Joomla in 45 mins
Case Study 1
Session 2: Still on Joomla 1.0? Migration from 1.0 to 1.5
Session 3: Joomla & Cloud Computing – Amazon S3
Session 4: Request to Response
Open Sessions
Case Study 2
For other tech events happening in Pune this weekend, check out the PuneTech calendar.
(PuneTech is honored to have Dr. Narayan Venkatasubramanyan, an Optimization Guru and one of the original pioneers in applying Optimization to Supply Chain Management, as our contributor. I had the privilege of working closely with Narayan at i2 Technologies in Dallas for nearly 10 years.
For Dr. Narayan Venkatasubramanyan’s detailed bio, please click here.
This is the second in a series of articles that we will publish once a week for a month. The first one was an ‘overview’ case study of optimization. Click here for the full series.)
this is a follow-up to optimization: a case study. frequent references in this article to details in that article would make this one difficult to read for someone who hasn’t at least skimmed through that.
it is useful to think of a decision-support system as consisting of 4 distinct layers:
data layer
visibility layer
predictive/simulation layer
optimization layer
the job of the data layer is to capture all the data that is relevant and material to the decision at hand and to ensure that this data is correct, up-to-date, and easily accessible. in our case, this would include master/static data such as the map of the field, the operating characteristics of the helicopter, etc as well as dynamic data such as the requirements for the sortie, ambient conditions (wind, temperature), etc. this may seem rather obvious at first sight but a quick reading of the case study shows that we had to revisit the data layer several times over the course of the development of the solution.
as the name implies, the visibility layer provides visibility into the data in a form that allows a human user to exercise his/her judgment. very often, a decision-support system requires no more than just this layer built on a robust data layer. for example, we could have offered a rather weak form of decision support by automating the capture of dynamic data and presenting to the radio operator all the data (both static and dynamic), suitably filtered to incorporate only parts of the field that are relevant to that sortie. he/she would be left to chart the route of the helicopter on a piece of paper, possibly checking off requirements on the screen as they are satisfied. even though this may seem trivial, it is important to note that most decision-support systems in everyday use are rather lightweight pieces of software that present relevant data to a human user in a filtered, organized form. the human decision-maker takes it from there.
the predictive/simulation layer offers an additional layer of help to the human decision-maker. it has the intelligence to assess the decisions made (tentatively) by the user but offers no active support. for instance, a helicopter scheduling system that offers this level of support would present the radio operator with a screen on which the map of the field and the sortie’s requirements are depicted graphically. through a series of mouse-clicks, the user can decide whom to pick up, where to fly to, whether to refuel, etc. the system supports the user by automatically keeping track of the weight of the payload (passenger+fuel) and warning the user of violations, using the wind direction to compute the rate of fuel burn, warning the user of low-fuel conditions, monitoring whether crews arrive at their workplace on time, etc. in short, the user makes decisions, the system checks constraints and warns of violations, and provides a measure of goodness of the solution. few people acknowledge that much of corporate decision-making is at this level of sophistication. the widespread use of microsoft excel is clear evidence of this.
the optimization layer is the last of the layers. it wrests control from the user and actively recommends decisions. it is obvious that the effectiveness of optimization layer is vitally dependent on the data layer. what is often overlooked is that the acceptance of the optimization layer by the human decision-maker often hinges on their ability to tweak the recommendations in the predictive layer, even if only to reassure themselves that the solution is correct. often, the post-optimization adjustments are indispensable because the human decision-maker knows things that the system does not.
the art (and science) of modeling
the term “decision-support system” may seem a little archaic but i will use it here because my experience with applying optimization has been in the realm of systems that recommend decisions, not ones that execute them. there is always human intervention that takes the form of approval and overrides. generally speaking, this is a necessary step. the system is never all-knowing. as a result, its view of reality is limited, possibly flawed. these limitations and flaws are reflected in its recommendations.
this invites the question: if there are known limitations and flaws in the model, why not fix them?
this is an important question. the answer to this is not nearly as obvious as it may appear.
before we actually construct a model of reality, we must consciously draw a box around that portion of reality that we intend to include in the model. if the box is drawn too broadly, the model will be too complex to be tractable. if the box is drawn too tightly, vital elements of the model are excluded. it is rare to find a decision problem in which we find a perfect compromise, i.e., we are able to draw a box that includes all aspects of the problem without the problem becoming computationally intractable.
unfortunately, it is hard to teach the subtleties of modeling in a classroom. in an academic setting, it is hard to wrestle with the messy job of making seemingly arbitrary choices about what to leave in and what to exclude. therefore, most students of optimization enter the real world with the impression that the process of modeling is quick and easy. on the contrary, it is at this level that most battles are won or lost.
note: the term modeling is going to be unavoidably overloaded in this context. when i speak of models, students of operations research may immediately think in terms of mathematical equations. those models are still a little way down the road. at this point, i’m simply talking about the set of abstract interrelationships that characterize the behaviour of the system. some of these relationships may be too complex to be captured in a mathematical model. as a result, the mathematical model is yet another level removed from reality.
consider our stumbling-and-bumbling approach to modeling the helicopter scheduling problem. we realized that the problem we faced wasn’t quite a text-book case. our initial approach was clearly very narrow. once we drew that box, our idealized world was significantly simpler than the real world. our world was flat. our helicopter never ran out of fuel. the amount of fuel it had was never so much that it compromised its seating capacity. it didn’t care which way the wind was blowing. it didn’t care how hot it was. in short, our model was far removed from reality. we had to incorporate each of these effects, one by one, because their exclusion made the gap between reality and model so large that the decisions recommended by the model were grossly unrealistic.
it could be argued that we were just a bunch of kids who knew nothing about helicopters, so trial-and-error was the only approach to determining the shape of the box we had to draw.
not true! here’s how we could have done it differently:
if you were to examine what we did in the light of the four-layer architecture described above, you’d notice that we really only built two of the four: the data layer and the optimization layer. this is a tremendously risky approach, an approach that has often led to failure in many other contexts. it must be acknowledged that optimization experts are rarely experts in the domain that they are modeling. nevertheless, by bypassing the visibility and predictive layers, we had sealed off our model from the eyes of people who could have told us about the flaws in it.
each iteration of the solution saw us expanding the data layer on which the software was built. in addition to expanding that data layer, we had to enhance the optimization layer to incorporate the rules implicit in the new pieces of data. here are the steps we took:
we added the fuel capacity and consumption rate of each helicopter to the data layer. and modified the search algorithm to “remember” the fuel level and find its way to a fuel stop before the chopper plunged into the arabian sea.
we added the payload limit to the data layer. and further modified search algorithm to “remember” not to pick up too many passengers too soon after refueling or risk plunging into the sea with 12 people on board.
we captured the wind direction in the data layer and modified the computation of the distance matrix used in the optimization layer.
we captured the ambient temperature as well as the relationship between temperature and maximum payload in the data layer. and we further trimmed the options available to the search algorithm.
we could have continued down this path ad infinitum. at each step, our users would have “discovered” yet another constraint for us to include. back in those days, ongc used to charter several different helicopter agencies. i remember one of the radio operator telling me that some companies were sticklers for the rules while others would push things to the limit. as such, a route was feasible or not depending on whether the canadian company showed up or the italian one did! should we have incorporated that too in our model? how is one to know?
this question isn’t merely rhetorical. the incorporation of a predictive/simulation layer puts the human decision-maker in the driver’s seat. if we had had a simulation layer, we would have quickly learned the factors that were relevant and material to the decision-making process. if the system didn’t tell the radio operator which way the wind was blowing, he/she would have immediately complained because it played such a major role in their choice. if the system didn’t tell him/her whether it was the canadian or the italian company and he didn’t ask, we would know it didn’t matter. in the absence of that layer, we merrily rushed into what is technically the most challenging aspect of the solution.
implementing an optimization algorithm is no mean task. it is hugely time-consuming, but that is really the least of the problems. optimization algorithms tend to be brittle in the following sense: a slight change in the model can require a complete rewrite of the algorithm. it is but human that once one builds a complex algorithm, one tends to want the model to remain unchanged. one becomes married to that view of the world. even in the face of mounting evidence that the model is wrong, one tends to hang on. in hindsight, i would say we made a serious mistake by not architecting the system to validate the correctness of the box we had drawn before we rushed ahead to building an optimization algorithm. in other words, if we had built the solution systematically, layer by layer, many of the surprises that caused us to swing wildly between jubilation and depression would have been avoided.
other articles in this series
this article is the second in a series of short explorations related to the application of optimization. i’d like to share what i’ve learned over a career spent largely in the business of applying optimization to real-world problems. interestingly, there is a lot more to practical optimization than models and algorithms. each of the the links below leads to a piece that dwells on one particular aspect.
articles in this series: optimization: a case study architecture of a decision-support system (this article) optimization and organizational readiness for change optimization: a technical overview
Dr. Narayan Venkatasubramanyan has spent over two decades applying a rare combination of quantitative skills, business knowledge, and the ability to think from first principles to real world business problems. He currently consults in several areas including supply chain and health care management. As a Fellow at i2 Technologies, he tackled supply chains problems in areas as diverse as computer assembly, semiconductor manufacturer, consumer goods, steel, and automotive. Prior to that, he worked with several airlines on their aircraft and crew scheduling problems. He topped off his days at IIT-Bombay and IIM-Ahmedabad with a Ph.D. in Operations Research from the University of Wisconsin-Madison.
He is presently based in Dallas, USA and travels extensively all over the world during the course of his consulting assignments. You can also find Narayan on Linkedin at: http://www.linkedin.com/in/narayan3rdeye
(PuneTech is honored to have Dr. Narayan Venkatasubramanyan, an Optimization Guru and one of the original pioneers in applying Optimization to Supply Chain Management, as our contributor. I had the privilege of working closely with Narayan at i2 Technologies in Dallas for nearly 10 years.
For Dr. Narayan Venkatasubramanyan’s detailed bio, please click here.
This is the first in a series of articles that we will publish once a week for a month. For the full series of articles, click here.)
the following entry was prompted by a request for an article on the topic of “optimization” for publication in punetech.com, a website co-founded by amit paranjape, a friend and former colleague. for reasons that may have something to do with the fact that i’ve made a living for a couple of decades as a practitioner of that dark art known as optimization, he felt that i was best qualified to write about the subject for an audience that was technically savvy but not necessarily aware of the application of optimization. it took me a while to overcome my initial reluctance: is there really an audience for this after all, even my daughter feigns disgust every time i bring up the topic of what i do. after some thought, i accepted the challenge as long as i could take a slightly unusual approach to a “technical” topic: i decided to personalize it by rooting in a personal-professional experience. i could then branch off into a variety of different aspects of that experience, some technical, some not so much. read on …
background
the year was 1985. i was fresh out of school, entering the “real” world for the first time. with a bachelors in engineering from IIT-Bombay and a graduate degree in business from IIM-Ahmedabad, and little else, i was primed for success. or disaster. and i was too naive to tell the difference.
for those too young to remember those days, 1985 was early in rajiv gandhi‘s term as prime minister of india. he had come in with an obama-esque message of change. and change meant modernization (he was the first indian politician with a computer terminal situated quite prominently in his office). for a brief while, we believed that india had turned the corner, that the public sector companies in india would reclaim the “commanding heights” of the economy and exercise their power to make india a better place.
CMC was a public sector company that had inherited much of the computer maintenance business in india after IBM was tossed out in 1977. quickly, they broadened well beyond computer maintenance into all things related to computers. that year, they recruited heavily in IIM-A. i was one of an unusually large number of graduates who saw CMC as a good bet.
not too long into my tenure at at CMC, i was invited to meet with an mid-level manager in electronics & telecommunications department of the oil and natural gas commission of india (ONGC). the challenge he posed us was simple: save money by optimizing the utilization of helicopters in the bombay high oilfield.
the problem
the bombay high offshore oilfield, the setting of our story
the bombay high oilfield is about 100 miles off the coast of bombay (see map). back then, it was a collection of about 50 oil platforms, divided roughly into two groups, bombay high north and bombay high south.
(on a completely unrelated tangent: while writing this piece, i wandered off into searching for pictures of bombay high. i stumbled upon the work of captain nandu chitnis, ex-navy now ONGC, biker, amateur photographer … who i suspect is a pune native. click here for a few of his pictures that capture the outlandish beauty of an offshore oil field.)
movement of personnel between platforms in each of these groups was managed by a radio operator who was centrally located.
all but three of these platforms were unmanned. this meant that the people who worked on these platforms had to be flown out from the manned platforms every morning and brought back to their base platforms at the end of the day.
at dawn every morning, two helicopters, flew out from the airbase in juhu, in northwestern bombay. meanwhile, the radio operator in each field would get a set of requirements of the form “move m men from platform x to platform y”. these requirements could be qualified by time windows (e.g., need to reach y by 9am, or not available for pick-up until 8:30am) or priority (e.g., as soon as possible). each chopper would arrive at one of the central platforms and gets its instructions for the morning sortie from the radio operator. after doing its rounds for the morning, it would return to the main platform. at lunchtime, it would fly lunchboxes to the crews working at unmanned platforms. for the final sortie of the day, the radio operator would send instructions that would ensure that all the crews are returned safely to their home platforms before the chopper was released to return to bombay for the night.
the challenge for us was to build a computer system that would optimize the use of the helicopter. the requirements were ad hoc, i.e., there was no daily pattern to the movement of men within the field, so the problem was different every day. it was believed that the routes charted by the radio operator were inefficient. given the amount of fuel used in these operations, an improvement of 5% over what they did was sufficient to result in a payback period of 4-6 months for our project.
this was my first exposure to the real world of optimization. a colleague of mine — another IIM-A graduate and i — threw ourselves at this problem. later, we were joined yet another guy, an immensely bright guy who could make the lowly IBM PC-XT — remember, this was the state-of-the-art at that time — do unimaginable things. i couldn’t have asked to be a member of a team that was better suited to this job.
the solution
we collected all the static data that we thought we would need. we got the latitude and longitude of the on-shore base and of each platform (degrees, minutes, and seconds) and computed the distance between every pair of points on our map (i think we even briefly flirted with the idea of correcting for the curvature of the earth but decided against it, perhaps one of the few wise moves we made). we got the capacity (number of seats) and cruising speed of each of the helicopters.
we collected a lot of sample data of actual requirements and the routes that were flown.
we debated the mathematical formulation of the problem at length. we quickly realized that this was far harder than the classical “traveling salesman problem”. in that problem, you are given a set of points on a map and asked to find the shortest tour that starts at any city and touches every other city exactly once before returning to the starting point. in our problem, the “salesman” would pick and/or drop off passengers at each stop. the number he could pick up was constrained, so this meant that he could be forced to visit a city more than once. the TSP is known to be a “hard” problem, i.e., the time it takes to solve it grows very rapidly as you increase the number of cities in the problem. nevertheless, we forged ahead. i’m not sure if we actually completed the formulation of an integer programming problem but, even before we did, we came to the conclusion that this was too hard of a problem to be solved as an integer program on a first-generation desktop computer.
instead, we designed and implemented a search algorithm that would apply some rules to quickly generate good routes and then proceed to search for better routes. we no longer had a guarantee of optimality but we figured we were smart enough to direct our search well and make it quick. we tested our algorithm against the test cases we’d selected and discovered that we were beating the radio operators quite handily.
then came the moment we’d been waiting for: we finally met the radio operators.
they looked at the routes our program was generating. and then came the first complaint. “your routes are not accounting for refueling!”, they said. no one had told us that the sorties were long enough that you could run out of fuel halfway, so we had not been monitoring that at all!
ONGC’s HAL Dhruv Helicopters on sorties off the Mumbai coast. Image by Premshree Pillai via Flickr
so we went back to the drawing board. we now added a new dimension to the search algorithm: it had to keep track of fuel and, if it was running low on fuel during the sortie, direct the chopper to one of the few fuel bases. this meant that some of the routes that we had generated in the first attempt were no longer feasible. we weren’t beating the radio operators quite as easily as before.
we went back to the users. they took another look at our routes. and then came their next complaint: “you’ve got more than 7 people on board after refueling!”, they said. “but it’s a 12-seater!”, we argued. it turns out they had a point: these choppers had a large fuel tank, so once they topped up the tank — as they always do when they stop to refuel — they were too heavy to take a full complement of passengers. this meant that the capacity of the chopper was two-dimensional: seats and weight. on a full tank, weight was the binding constraint. as the fuel burned off, the weight constraint eased; beyond a certain point, the number of seats became the binding constraint.
we trooped back to the drawing board. “we can do this!”, we said to ourselves. and we did. remember, we were young and smart. and too stupid to see where all this was going.
in our next iteration, the computer-generated routes were coming closer and closer to the user-generated ones. mind you, we were still beating them on an average but our payback period was slowly growing.
we went back to the users with our latest and greatest solution. they looked at it. and they asked: “which way is the wind blowing?” by then, we knew not to ask “why do you care?” it turns out that helicopters always land and take-off into the wind. for instance, if the chopper was flying from x to y and the wind was blowing from y to x, the setting was perfect. the chopper would take off from x in the direction of y and make a bee-line for y. on the other hand, if the wind was also blowing from x to y, it would take off in a direction away from y, do a 180-degree turn, fly toward and past y, do yet another 180-degree turn, and land. given that, it made sense to keep the chopper generally flying a long string of short hops into the wind. when it could go no further because they fuel was running low or it needed to go no further in that direction because there were no passengers on board headed that way, then and only then, did it make sense to turn around and make a long hop back.
“bloody asymmetric distance matrix!”, we mumbled to ourselves. by then, we were beaten and bloodied but unbowed. we were determined to optimize these chopper routes, come hell or high water!
so back we went to our desks. we modified the search algorithm yet another time. by now, the code had grown so long that our program broke the limits of the editor in turbo pascal. but we soldiered on. finally, we had all of our users’ requirements coded into the algorithm.
or so we thought. we weren’t in the least bit surprised when, after looking at our latest output, they asked “was this in summer?”. we had now grown accustomed to this. they explained to us that the maximum payload of a chopper is a function of ambient temperature. on the hottest days of summer, choppers have to fly light. on a full tank, a 12-seater may now only accommodate 6 passengers. we were ready to give up. but not yet. back we went to our drawing board. and we went to the field one last time.
in some cases, we found that the radio operators were doing better than the computer. in some cases, we beat them. i can’t say no creative accounting was involved but we did manage to eke out a few percentage point of improvement over the manually generated routes.
epilogue
you’d think we’d won this battle of attrition. we’d shown that we could accommodate all of their requirements. we’d proved that we could do better than the radio operators. we’d taken our machine to the radio operators cabin on the platform and installed it there.
we didn’t realize that the final chapter hadn’t been written. a few weeks after we’d declared success, i got a call from ONGC. apparently, the system wasn’t working. no details were provided.
i flew out to the platform. i sat with the radio operator as he grudgingly input the requirements into the computer. he read off the output from the screen and proceeded with this job. after the morning sortie was done, i retired to the lounge, glad that my work was done.
a little before lunchtime, i got a call from the radio operator. “the system isn’t working!”, he said. i went back to his cabin. and discovered that he was right. it is not that our code had crashed. the system wouldn’t boot. when you turned on the machine, all you got was a lone blinking cursor on the top left corner of the screen. apparently, there was some kind of catastrophic hardware failure. in a moment of uncommon inspiration, i decided to open the box. i fiddled around with the cards and connectors, closed the box, and fired it up again. and it worked!
it turned out that the radio operator’s cabin was sitting right atop the industrial-strength laundry room of the platform. every time they turned on the laundry, everything in the radio room would vibrate. there was a pretty good chance that our PC would regress to a comatose state every time they did the laundry. i then realized that this was a hopeless situation. can i really blame a user for rejecting a system that was prone to frequent and total failures?
other articles in this series
this blog entry is intended to set the stage for a series of short explorations related to the application of optimization. i’d like to share what i’ve learned over a career spent largely in the business of applying optimization to real-world problems. interestingly, there is a lot more to practical optimization than models and algorithms. each of the the links below leads to a piece that dwells on one particular aspect.
Dr. Narayan Venkatasubramanyan has spent over two decades applying a rare combination of quantitative skills, business knowledge, and the ability to think from first principles to real world business problems. He currently consults in several areas including supply chain and health care management. As a Fellow at i2 Technologies, he tackled supply chains problems in areas as diverse as computer assembly, semiconductor manufacturer, consumer goods, steel, and automotive. Prior to that, he worked with several airlines on their aircraft and crew scheduling problems. He topped off his days at IIT-Bombay and IIM-Ahmedabad with a Ph.D. in Operations Research from the University of Wisconsin-Madison.
He is presently based in Dallas, USA and travels extensively all over the world during the course of his consulting assignments. You can also find Narayan on Linkedin at: http://www.linkedin.com/in/narayan3rdeye
Normally, PuneTech covers mostly software technology. However, today, we feel an exception is warranted. The Tata Nano, the world’s cheapest car, which could change the world, was designed entirely in Pune. On this occassion, Amit Paranjape, a proud Punekar (and also chief evangelist of PuneTech) wrote this article on his blog, about the achievements of Pune in automotive technology. It is reproduced on PuneTech with permission.
Today, India and possibly the entire automotive world commemorate the customer launch of the ‘Nano’ – the world’s cheapest car. The brainchild of the Indian corporate legend Ratan Tata is finally available to the Indian consumer. I am sure that the Nano will raise a whole bunch of debates around urban traffic-management issues; but today is not the time for those. Today is a time for celebration!
Pune too celebrates this historic occasion; but I am not sure how many Punekars realize the significance of Pune’s role in creating this and other automotive history in India.
The Nano was completely designed and developed at the Tata Motors facility in Pimpri-Chinchwad Pune. The initial manufacturing will also be carried out here.
1980s: Manufacturing of India’s first automatic (non-geared) scooter: Kinetic Honda.
1990s: India’s first fully indigenous car: Tata Indica.
2008-09: Launch of world’s cheapest car: Tata Nano.
You can also add the development of India’s most popular Truck-Line to this list. Pune also leads the nation in various automotive suppliers, ancillary units and industrial equipment.
India’s biggest, one of the most innovative and world’s 2nd largest forging company – Bharat Forge has been at the forefront of this pack.
India’s largest Diesel Engines & Generator Manufacturer – Cummins has been active in Pune’s industrial landscape since the 1960s.
Research and Software for Automotive Engineering also have strong presence in Pune.
It’s no coincidence that all major global CAD/CAM software and services companies have significant presence in Pune: Ansys, AutoDesk, Catia, Geometric, PTC and UGS-Siemens. I doubt if there’s any city in the world that has the presence of all these entities! (See all PuneTech articles about Computer Aided Design at http://punetech.com/tag/CAD -ed)
ARAI (Automotive Research Association of India) based in Pune, is the premier automotive research institute in India, that is responsible for research and testing & certification of every vehicle model on Indian roads.
I am confident that in the coming decades, Pune will continue to innovate and be at the forefront of automotive engineering in India, and the world.
So now remember – next time you see a Nano on Pune Streets (traffic jams not withstanding), it is as ‘Puneri’ as the ‘Puneri Pagdi’ or ‘Chitale Bakarwadi’!
After we carried a few quick articles on why you should learn more about Ruby and Ruby on Rails (take 1, take 2) last month, we decided that we wanted to give people a much deeper article on why these new languages (Ruby, Python, PHP) and frameworks (Rails, Django) are setting the web world on fire. We invited Dhananjay Nene to write an article with an in depth discussion of the technical reasons how these new languages differ from the older ones and when to choose one over the other. He responded with this article which, as an added bonus, also includes the business reasons for your decisions. At the request of the community, Dhananjay is also giving a talk on the relative strengths and weaknesses of different programming languages on Saturday, 28th March, 4pm, at SICSR. All those who found this article interesting should definitely attend.
Introduction
Programing language selection is often a topic that elicits a lot of excitement, debate and often a bit of acrimony as well. There is no universally superior programming language that one can recommend, so I tend to generally disregard most language opinions which say ‘X language is the best’, without specifying the context under which it is superior. Finally most language debates often deal with the technical issues and not the ROI issues. Hopefully I shall be able to address this topic without being guilty of any of these problems.
The range of languages that fall under Dynamic Programming Languages category is rather extensive. My experience is primarily limited to Python and to a lesser extent PHP, Ruby, Javascript, and Groovy. For the rest of this article, I shall be primarily referring to Python or Ruby when I use the word dynamic languages, though many of the references may continue to be applicable and relevant for a number of other dynamic programming languages.
As I describe the technical characteristics, I shall also continue to attempt to address the business aspects as well, so you might find this article at a little techno-business level. Assuming I am able to excite their interest, the tech guys would not find sufficient technical details and would be hungry to hunt for more, and while the business guys would get a little teased with the possibilities, they will not quite get the ROI served in the traditionally formatted excel spreadsheets. Being aware of that, I continue down this path with a feeling that this perhaps will be the most appropriate level for me to abstract this article to.
Characteristics of Dynamic Programming Languages.
Let us quickly review some of the characteristics :
Object Oriented : Many dynamic languages support full object orientation. There are many who don’t necessarily buy the benefits of Object Orientation, but it is my strong belief, that once a piece of software grows beyond a certain threshold of complexity and / or size, Object Orientation starts delivering very strong dividends. There are a few areas such as highly complex, algorithmic processing which might be better suited for functional programming. However a majority of the medium-to-large sized web applications are better served by OO. The empirical evidence at least bears out the fact that most of the most popular languages today (except C) are Object Oriented. However this still is a very very large class of languages which in them include C++, Java, PHP, Python, Ruby etc. The one area where some dynamic languages separate themselves from the others is in the notion of “everything is an object”, ie. primitives such as numbers, functions are all objects by themselves.
Business implications: OO code well designed and implemented allows for a substantial reduction in maintenance costs. When working with a team which is up the curve on OO, it is likely to lead to lower costs and time on inital coding as well. On the other hand, both training costs and skill requirements are higher for fully OO languages. If you are already using partialy OO / hybrid languages such as PHP, C++ or Java, and are convinced about OO, using fully OO languages such as Python or Ruby will help you leverage the OO capabilities even further.
Duck Typing : In very loose terms, duck typed languages do not require you to declare an explicit interface. You send an object a message (ie. invoke a function or access an attribute) and if it can respond to it, it will, and if it can’t it will result in an error. Duck typing is a specific typing system which is a subset of a broader system called Dynamic Typing, which often makes for an interesting debate with its counterpart – Static typing : Static and Dynamic Type checking in practice. For people well grounded in static typing alone, this can sometimes seem to be sacrilegious. I am convinced that duck typing makes writing code much much faster for two reasons – a) You now require to write fewer lines of code and b) You often don’t have to keep on regularly waiting for the compiler to do its work. There is also a substantial capability enhancement that dynamic typing makes to the language type system, which allow the frameworks to build dynamic types on the fly. This in turn offers the framework users many more capabilities than frameworks written in other languages. That is why it is nearly impossible to write frameworks like Rails or Django in Java (You can modify the class loaders and use byte code generation to generate the new types, but the compiler can’t see them so you cant use them). That is also why there is a lot of anticipation of using JRuby, Jython and Grails on the JVM since the languages underlying them (Ruby, Python and Groovy respectively) bring the dynamic typing capabilities to the JVM platform.
Business Implications :Writing code is much much faster. Maintenance depending upon the situation can sometimes be more or less difficult in case of dynamic typed languages. Refactoring is usually a lot more difficult in case of dynamically typed languages since the underlying type system is not able to infer sufficiently about the code to help the refactoring tools, as is possible in case of statically typed languages. It is my opinion that a skilled and trained development team using dynamic languages can generally substantially outperform another equally capable team using static languages. Insufficiently or poorly skilled development teams however can lead to very very different kind of pitfalls in these class of languages. In both cases the code becomes difficult to change or maintain due to a) cryptic code in case of dynamically typed languages and b) extremely large code bases in case of statically typed languages. Both are undesirable situations to be in but if I had to choose between one of the two, I would go for being in the cryptic mess since it is at least manageable by bringing in external skilled help.
Metaprogramming : Metaprogramming is in loose terms the ability of programs to write programs. A large proportion of developers may not use this capability too frequently. Specifically in web application development it gets used as a mechanism to transform one set of datastructures which a programmer specifies into code at runtime. As I point out later in this article, it in fact is a very important element in designing common frameworks and libraries which in turn offer substantial capabilities including small code and easier maintenance. A quick note to state that metaprogramming is not code generation. In case of code generation, one uses the generator to generate code which is then compiled. A big limitation with this is the fact that often people modify the generated code leading to really tough maintenance nightmares and the fact that it is a two stage process which is prone to more errors. Metaprogramming results in new code “coming to life” so to speak while your program is running.
Business Implications : Read on, they will get covered in the final roundup. They are large and they are positive.
Function blocks/objects, iterators, closures, continuations, generators: I will not go into any substantial details of this issue except to say that small pieces of code logic can be handled in a much much more concise way than if these weren’t supported. While many situations may not need closures support, you will be glad to have them on your side when needed.
Business Implications : Helps having shorter, cleaner code leading to lesser development and maintenance costs. Another significant positive is that your developers are just likely to be so much happier since they get some truly nice building blocks for concise and elegant expression of their logic. Can’t think of any significant negatives.
There are a full range of other capabilities, but none come to mind immediately as something that have strong business implications as well.
When did these languages say Ruby and Python originate ? Most people are likely to be a little surprised if the answer is in the last millenium. Yet Guido von Rossum started working on Python in 1986 and Ruby was released in 1992. Python has been rather well known within the scientific community and perhaps a bit within the systems / OS utility programming communities for quite some time. However both languages grabbed a large mindshare only post 2005. A big reason for their popularity (especially in case of Ruby’s case) came from the popularity the frameworks which used them. Ruby on Rails for ruby and Django (to the best of my knowledge) for python. These frameworks combined the language capabilities with the learnings of good design practices for internet applications (eg MVC, declarative validations, simple ORM etc) into a simple usable package, which developers could take and build web applications quickly. There are examples of people having built simple web apps within a day and medium complexity apps in 1-3 weeks using these frameworks. The languages are the ingredients, the frameworks are the cooks – a great combination for serving great meals. Now you will find many such frameworks in these languages, including some which have better capabilities for building more sophisticated / complex applications eg. Merb and Pylons.
I am not too sure of how many people are exactly aware of the role of metaprogramming in the frameworks’ successes. I am willing to believe that but for metaprogramming, these frameworks simply would not have achieved anywhere close to the success they achieved. It is metaprogramming which takes the datastructures as defined by a developer and converts it into runtime code implicitly, saving the developer lots of time and effort. So even if most developers don’t actively write metaprograms, their lives are so much easier. Metaprogramming capabilities are also the reason why it is virtually impossible to write similar frameworks in Java. However if you are on the .NET or JVM environments, things are definitely looking encouraging with the possibilities to use IronPython or IronRuby on .NET or JRuby or Jython or Groovy+Grails on the JVM.
Business implications : If you are focused on scientific or desktop or highly algorithmic applications, where python especially is used extensively, you are likely to get benefits from these languages on their own merit alone. For web applications you will see the maximum benefits by using the web MVC frameworks along with the languages. I submit that on the whole you are likely to see very substantial reduction in development, enhancement and maintenance times – sweet music for any end user, investor or project manager.
Increased Business Agility
There is one more reason why I believe these languages are especially helpful. They help by increasing development agility to an extent where it now allows for the business to be more agile. You can get a first prototype version up in weeks, take it around to potential users, and gather feedback on the same. Incorporate elements of this feedback into the next release of working code quickly. The business benefits of such a scenario are tremendous. You might wonder that this is a process issue, so what does it have to do with a language selection. I would submit, that languages which allow changes to be made faster, help support this process in a far superior way. Another equally important facet is the superior risk management. Since you are able to build features with lower investments, you are able to get a series of customer feedbacks into your decision making process much faster. This helps being able to come up with a product that really meets the customer expectations much earlier. This happens by allowing the better features to come in earlier and also by allowing the lesser important or lesser relevant features to be decided to be deferred earlier. That’s precisely the reason why the dynamic languages have found a strong acceptance in the startup world. I believe the increasing agility which is often required in the startup world, is and will continue to be increasingly required of established enterprises. Precisely the reason why I believe these languages will continue to do better in the enterprise space as well. Finally, these languages make it relatively easier to tell your business sponsor – We will work with you on imprecise requirements rather than spending months on nailing down requirements which anyways are likely to change later. This has both a pro and a con especially for outsourcing situations. It is likely to allow for tremendous customer delight in terms of a vendor that works with him in such a flexible manner, yet it does introduce challenges in terms of how the commercials and management of the project are handled.
The reason I would like to especially point out increased business agility is because programmers don’t often visualise or evangelise it much, but when I wear a manager’s hat, it is perhaps the most compelling benefit of these languages.
Concluding
As I said earlier, there is no single universal language which is the best for all scenarios. There are some scenarios where using dynamic languages will not be helpful
A Treemap view of sales of programming language books by O’Reilly Media in 4Q2008. The size of a box represents the total sales of a book. The color represents the increase or decrease in sales compared to same quarter in 2007. Green = increase, bright green = big increase, red = decrease, bright red = large decrease. See full article at O’Reilly Radar for lots of interesting details.
When not to use these languages
You are building a simple / small application and don’t have the available skill sets. One exception to this is where you decide to use it in a simple application to allow yourself a non risky mechanism of building these skillsets.
Extremely High performance requirements. However please make sure that you really need the high performance capabilities of say a C, C++ or Java. In my experience 80% of developers like to believe that they are building highly performant applications where the maximum speed is a must have. Yet the top 10% of them are facing far far more critical performance requirements than the remainder. Unless you are convinced you are in the top 10%, you should certainly consider dynamic languages as an option. Moreover in case of most high performance requirements, these can sometimes be boiled down to a few inner loops / algorithms. Consider implementing the same in C, / Java or other .NET languages (depending upon the choice of your dynamic language interpreter implementation)
You have an architecture standard in place which does not allow using these languages. If you are convinced your applications are better served by using dynamic languages both from your individual application and an overall enterprise perspective, consider taking the feedback to your standards setting body to see if you can pilot a different approach. Also evaluate if the .NET or JVM versions can help you comply with the architecture guidelines.
You are unable to commit to the retraining requirements. While these languages are easy and powerful to use, leveraging that power can require some amount of retraining. If that does not fit your business plans, since the retraining effort could impact immediate and urgent requirements, that could be a reason to not use these languages. However in such situations do consider investing in building this skill sets before you get to another similar decision point.
You need a very high levels of multithreadinging as opposed to multi processing support. While this is not a typical situation for web applications, you should be aware that most dynamic languages have some limitations in terms of multi threading support. This actually is not necessarily an issue with the language as with the implementation eg. the C implementation of python has the notorious Global Interpreter Lock which constrains you from being able to use more than a handful of threads per processes efficiently. However the same restriction is not present in Jython (the jvm implementation of python). This is likely to be an issue for a miniscule percentage of the web applications market for the primary reason that multi process / shared nothing architecture styles often work quite well for many web applications and they don’t really need multi threading.
So where’s my return on investment ?
First of all lets talk of the investment part. If you get into it in a paced approach, the investment may not be that great. Start with a team size of anywhere between 2-6 people (depending upon your organisation and project size). Think of 15 days of intensive training followed by a 2-6 months coming up the curve effort (more likely 2 than 6). Make sure your first project is not a critical one under tremendous business pressure. This can be subsequently followed by more people getting retrained as necessary. In the longer term it might actually help reduce your incremental investment, since it might be much easier to ramp up new programmers in Ruby or Python than say Java or C#.
Secondly lets look at the incrementally higher costs. You are likely to need people who are a little bit more capable in terms of understanding and debugging the same logic expressed in fewer lines of code (that sometimes can be a challenge) and then be able to modify and enhance the same. This may increase your testing and fixing costs in the earlier days. Finally while the fewer lines of code can make refactoring easier, you could find that your total refactoring costs are a little higher.
Now the returns part. I am convinced that the increased business agility is the strongest return in business terms. Immediately after that is the substantial reduction in development, enhancement and maintenance times. If neither of these benefits are appealing, when contrasted with some other issues that you might perceive, maybe considering dynamic languages in your context is not such a great idea.
One more factor that I would of course encourage you to evaluate from a business perspective are the implications for you if your competition (assuming it is not already using them) started using these languages. The implications would vary from case to case, but it could also help you decide how important this issue is for you.
About the author – Dhananjay Nene
Dhananjay is a Software Engineer with around 17 years of experience in the field. He is passionate about software engineering, programming, design and architecture. He did his post graduation from Indian Institute of Management, Ahmedabad, and has been involved in Senior Management positions and has managed team sizes in excess of 120 persons. His tech blog, and twitter stream are a must read for anybody interested in programming languages or development methodologies. Those interested in the person behind the tech can check out his general blog, and personal twitter stream. For more details, check out Dhananjay’s PuneTech wiki profile.
(This article giving an overview of the field of Computer Aided Engineering has been written for PuneTech on our request by Dr. Ajey Walavalkar, regional manager of services and support at AnsysFluent India, a company specializing in CAE and CFD applications and services, which has a development center in Pune. See the end of this article for more about Ajey.)
In the earlier PuneTech article “An Overview of CAD,” Yogesh and Amit have provided a very good overview of the CAD technology spectrum. There, we learnt where and how “Analysis” and “Simulations” fit in the overall scheme of CAD. This article is intended to provide a broad overview of the area called Engineering Simulations, which spans across the above mentioned “Analysis” and “Simulations” fields on the CAD canvas.
What are Engineering Simulations?
The analysis of vibrations and the dynamical behaviour due to excitation forces have been applied routinely to cable-stayed bridges using SMR devised analysis tools. Note that similar techniques have been used to analyse the resonance behaviour of large floating bridge structures.
Let’s say you want to build a bridge. Not just any bridge, but a massive suspension bridge to rival the Golden Gate Bridge in San Francisco Bay Area. How do you decide the type of steel, the span length, the tower height, the thickness of the cables, the depth of the foundations, and other design parameters? You will wonder that if this problem was solved in the 1930s then why is it tough today? The simple answer is, ‘Solved? Yes! But at what cost and effort?’ Previously, the simple solution for most tough engineering problems was to ‘over-engineer’ the solution by building in a ‘huge factor of safety’. Today, this design process is lot more accurate and efficient. Today, the engineering team will start off by considering the effects of vehicles (weights and speeds) plying on the bridge, the wind forces that will sway the bridge, the waves that will hit the foundation, the steady long-term corrosive effects of weather, etc. These effects can studied by mathematically modeling these factors and ‘simulating’ them with a help of a computer. Such ‘simulations’ greatly help today’s engineers to come up with the exact design that is safe and that saves time and costs.
Wikipedia defines Simulation as follows:
Simulation is the imitation of some real thing, state of affairs, or process. The act of simulating something generally entails representing certain key characteristics or behaviors of a selected physical or abstract system.
Engineering simulations are simulations applied to engineering problems. These can range from simple “back of the envelope” calculations to full fledged systems simulations using High Performance Computing. Mathematical modeling and physical modeling are two main aspects of engineering simulations. Mathematical modeling refers to the description of a particular phenomenon in terms of mathematical equations. E.g. If you wish to find out the distance a canon ball will travel when fired from a canon at certain angle, at a certain speed; you can write some mathematical expressions and solve them to calculate this distance. This is mathematical modeling. Now suppose you are firing that canon ball at a mountain cliff and the ball is going to bounce off and roll down the slopes of the mountain, how do you find out where that ball will end up? Here you need to also consider the physical structures and that leads you to physical modeling.
Uses of Engineering Simulations
Today, computer aided engineering simulations are used extensively in designing and development of almost all industrial products. In addition to product designing, simulations find their uses in troubleshooting of existing systems and research & development of new processes. Many times the field of computer aided engineering simulations is also termed as Computer Aided Engineering (CAE). There are numerous software products available in the market and they offer variety of engineering simulation capabilities. Many organizations also use their in-house built simulation software products that capture their time proven design practices.
There are many objectives in performing engineering simulations:
During product design cycle, engineering simulations enable the designers to evaluate various design options without getting into costly prototyping and testing. Use of simulations in the design process can help narrow the design choices to very small set that can be taken to the prototyping and testing phases. Even with physical testing, one can inspect and measure only a small number of variables at few select locations. Whereas simulations can provide visual information on all the variables of interest at all locations inside the simulated space.
Simulations can help troubleshoot deviation of actual performance of system/product from the desired one.
Simulations can help designers and analysts to evaluate response from the product or system, to ‘highly-off’ design conditions, e.g. evaluating the safety of a nuclear reactor facility in hurricane conditions etc.
What is involved in performing engineering simulations?
The domain of engineering simulations is mainly divided based on the physics that is captured in the simulation. Based on this criterion, the domain is broadly categorized as follows
In any type of engineering simulation, there are three main stages.
Pre-processing: This involves determining the region in space that one will need to use in simulation. This region, (with all the necessary geometry details that are needed to attain the goals of the simulation) is then drafted on a computer using various CAD software tools. This space domain is then discretized into various small volumes or elements called computational cells. This is called meshing or gridding of the domain. Depending on the physics involved, the extent of the physical domain, the accuracy desired and the computational resources that are available, this mesh or grid can range from a few hundred cells to few hundred million cells. Ansys recently broke the Billion cell barrier.
Solving the governing equations: This step involves solving the governing mathematical equations that describe the physics that one desires to capture at all the computational cells. Finite Element Method (FEM), Finite Volume Method (FVM) and Finite Difference Method (FDM) are the most commonly used numerical techniques that enable solving the governing partial differential equations on discretized domain on computers. To perform these calculations, many inputs need to be provided. The results achieved from the simulations are directly dependent on the inputs provided. Techniques of parallel computing enable use of multiple cpus on a network to be used for solving a single simulation. In this technique, if you have a mesh with say 2 million cells, and have a network cluster of 8 cpus, each CPU can solve equations for 0.25 million cells and thus the simulation time can be reduced significantly as compared to a single CPU solving equations for all 2 million cells.
Post-processing: This stage involves understanding and analyzing the results of the simulations. The simulation software tools provide a mix of visual as well as alpha-numeric reporting of various variable of interest to the user so that the user can derive the information from the simulation needed to fulfill their objectives.
Most of the engineering simulation software tools provide connectivity to variety of CAD drafting packages so that geometries can be imported in from various different sources. Many of them provide ability to customize the solvers such that users can add their own/ proprietary physics/knowledge in the simulations. The post-processing allows results to be ported to various other analysis tools including optimization tools. Through these customizations, many users of these software tools have embedded the engineering simulation technology deep into their design process. Of the software tools available in the market, many are general tools that can be used by any industry vertical where as there are few tools that are developed for only one or few industry verticals and are easier to use to simulate applications in that particular industry.
At present, most of the software tools available in the market are solving various physics involved in the real life process or equipment separately. However, in reality all these physics occur simultaneously and affect one another. The world of engineering simulations is moving rapidly towards incorporating multiple physics and their interactions to provide more reliable predictions. The engineering product development community is exploring, what is known as “Simulation Driven Product Development”, so that full benefits of engineering simulation technology can be leveraged to their competitive advantage. Some of the major software providers in this space have already started offering multi-physics solvers that enable organizations to march in this direction.
Another new facet that is coming in focus now is of knowledge management. Use of these simulation software tools, is generating a lot of engineering knowledge which the companies would like to leverage in their future design processes. With this need in mind, integrated engineering knowledge management platforms that will work seamlessly with the engineering simulation tools are being developed.
Engineering Simulations scene in Pune
Pune has attracted most of the main players in this exciting domain.
Ansys Inc, one of the leading companies in developing the engineering simulation software tools is present in Pune. Ansys has development, testing, sales, support and services functions for India based as well as worldwide customers being conducted out of the Pune office. Ansys develops structural analysis, CFD, EAD as well as optimization tools that are used widely in almost all industry segments.
In addition to Ansys, companies such as Siemens PLM, MSC Software, Abaqus also have presence in Pune.
Pune also has a growing list of companies that are using these engineering simulation softwares. Companies such as Cummins, John Deere, Dow Chemicals, Eaton Corp, Honeywell, etc have set up their product development centers in Pune which use these tools. Tata Motors, Tata Technologies, research wing of TCS, TRDDC and Tata’s new venture in HPC, CRL are also exploring the field of engineering simulations actively. Additionally engineering services consultants such as Tridiagonal, Pacific Mindware are also based in Pune.
Education institutes such as COEP too are now well equipped and have elective courses that allow students exposure to this interesting field.
About the author – Dr. Ajey Walavalkar
Ajey has over 10 years of experience in the Computational Fluid Dynamics industry. Currently, he is a Regional Support and Services Manager for Ansys Fluent India, in Pune. He has experience in successfully simulating various applications across industry domains, building teams of engineers for delivering CFD projects, support and services in an offshore setting. Ajey has a Ph.D. in Mechanical Engineering from Pennsylvania State University, USA, and a B.E. in Mechanical Engineering from COEP.
PlanetAikon is a forum that allows “community powered innovation”. It is a Pune-based startup which allows its members to “connect, collaborate and co-create”. It strives to be a unique innovation ecosystem bringing together a community of ideators, contributors and sponsors – in other words, it connects the guys with the bright ideas, the guys with the money, and the guys with skills and time to implement the idea. 
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_b.png?x-id=64370af9-7c39-49f0-88c5-fd5f0fc7251e)
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_b.png?x-id=9a3be87e-dceb-4e25-889d-bf5691a385c8)
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_b.png?x-id=14c4d6df-3b6b-4d53-9773-b45cbde4f984)
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_b.png?x-id=a94ed4f5-a95b-4b6a-b4fb-d6261e766fe4)
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_b.png?x-id=a70c6581-d85e-4b4d-b969-2b9f0a193054)
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_b.png?x-id=8db837a3-6b5d-477c-bf69-6740d0c4e82b)
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_b.png?x-id=4b7c32a0-1264-42f1-b98c-8f7fadffb135)
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_b.png?x-id=4608fc72-dcb8-4511-9b86-967d13574a92)
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_b.png?x-id=b7ed0643-f578-4076-8dd2-03de3e329a10)
![Reblog this post [with Zemanta]](http://img.zemanta.com/reblog_b.png?x-id=06f84c1f-59ae-470f-80a9-775c23973a03)