Tag Archives: techweekend

TechWeekend 6 (#tw6): NoSQL and Databases In the Cloud – 15 Jan

TechWeekend Pune and Microsoft present #TW6, a deep-dive into the world of databases in the cloud – NoSQL, SQL Azure, MongoDB, Redis and more. On Saturday, January 15, from 10am to 1pm.

As you’re aware, TechWeekend is a monthly series of lectures/discussions on technical topics. It happens on the third Saturday of every month, at the same time and same place.

Schedule

  • Introduction to NoSQL – Navin Kabra
  • MongoDB the Infinitely Scalable – Baishampayan Ghose
  • SQL Azure – The Database of the Clouds – Saranya Sriram
  • When NoSQL is not appropriate for you – Dhananjay Nene
  • Experiences with Redis – Gautam Rege

About the Sponsor – Microsoft

Many thanks to Microsoft for sponsoring the venue for Techweekend. Microsoft wants to get more closely involved with the tech community in Pune, and particularly the open source enthusiasts – with the intention of making everybody aware that their cloud technologies (like Azure) actually play well with open source, and that you can deploy your php applications, your drupal/joomla installs on Azure.

Logistics

It is from 10am to 1pm on Saturday 15th January at the Sumant Moolgaonkar Auditorium, Ground Floor, Wing A, ICC Trade Towers, SB Road. As usual, TechWeekend is free for anyone to attend, but please register here.

LiveBlog #tw5: Intro to Functional Programming & Why it’s important

This is a live-blog of TechWeekend 5 on Functional Programming. Please keep checking regularly, this will be updated once every 15 minutes until 1pm.

Why Functional Programming Matters by Dhananjay Nene

Dhananjay Nene started off with an introductory talk on FP – what it is, and why it is important.

FP is a language in which functions have no side-effects. i.e., the result of a function is purely dependent on its inputs. There is no state maintained.

Effects/Implications of “no side effects”

  • Side-effects are necessary: FP doesn’t mean completely side-effect free. If you have no side-effects, you can’t do IO. So, FP really means “largely side-effect free”. Specifically, there are very few parts of the code that have side-effects, and you know exactly which those are.
  • Testability: Unit Testing becomes much easier. There are no “bizarre interactions” between different parts of the code. “Integration” testing becomes much easier, because there are no hidden effects.
  • Immutability: There are no “variables”. Once a value has been assigned to a ‘name’, that value is ‘final’. You can’t change the value of that ‘name’ since that would be ‘state’ and need ‘side-effects’ to change it.
  • Lazy Evaluation: Since a function always produces the same result, the compiler is free to decide when to execute the function. Thus, it might decide to not execute a function until that value is really needed. This gives rise to lazy evaluation.
  • Concurrency control is not so much of a problem. Concurrency control and locks are really needed because you’re afraid that your data might be modified by someone else while you’re accessing it. This issue disappears if your data is immutable.
  • Easier parallelization: The biggest problem with parallelizing programs is handling all the concurrency control issues correctly. This becomes a much smaller problem with FP.
  • Good for multi-core: As the world moves to multi-core architectures, more and more parallelism will be needed. And humans are terrible at writing parallel programs. FP can help, because FP programs are intrinsically, automatically parallelizable.

Another important feature of functional programming languages is the existence of higher order functions. Basically in FP, functions can be treated just like data structures. They can be passed in as parameters to other functions, and they can be returned as the results of functions. This makes much more powerful abstractions possible. (If you know dependency injection, then higher-order functions are dependency injection on steroids.)

FP gives brevity. Programs written in FP will typically be much shorter than comparable imperative programs. This is probably because of higher-order functions and clojures. Compare the size of the quicksort code in Haskell vs. Java at this page

You need to think differently when you start doing functional programming.

Think different:

  • Use recursion or comprehensions instead of loops
  • Use pattern matching instead of if conditions
  • Use pattern matching instead of state machines
  • Information transformation instead of sequence of tasks
  • Software Transactional Memory FTW!

Advantages of FP:

  • After initial ramp-up issues, development will be faster in FP
  • Code is shorter (easier to read, understand)
  • Clearer expression of intention of developer
  • Big ball of mud is harder to achieve with pure functions. You will not really see comments like “I don’t know why this piece of code works, but it works. Please don’t change it.”
  • Once you get used to FP, it is much more enjoyable.
  • Faster, better, cheaper and more enjoyable. What’s not to like?

The cost of doing FP:

  • Re-training the developers’ brains (this is a fixed cost). Because of having to think differently. Can’t just get this from books. Must do some FP programming.
  • You can suffer from a lack of third-party libraries(?), but if you pick a language like Clojure which sits on the JVM, then you can easily access java libraries for the things that don’t exist natively in your language.

Should a company do it’s next project in a functional programming language? Dhananjay’s recommendation: start with small projects, and check whether you have the organizational capacity for FP. Then move on to larger and larger projects. If you’re sure that you have good programmers, and there happens to be a 6-month project for which you’re OK if it actually becomes a 12-month project, then definitely do it in FP. BG’s correction (based on his own experience): the 6-month project will only become a 8-month project.

Some things to know about Erlang by Bhasker Kode

Bhasker is the CEO of http://hover.in. They use Erlang in production for their web service.

Erlang was created in 1986 by developers at Ericsson for their telecom stack. This was later open-sourced and is now a widely used language.

Erlang is made up of many “processes”. These are programming language constructs – not real operating system processes. But otherwise, they are similar to OS processes. Each process executes independently of other processes. Processes do not share any data. Only message passing is allowed between processes. There are a number of schedulers which schedule processes to run. Normally, you will have as many schedulers as you have cores on your machine. Erlang processes are very lightweight.

Garbage collection is very easy, because as soon as a process dies, all its private date can be garbage collected because this is not shared with anyone else.

Another interesting thing about Erlang is that the pattern matching (which is used in all functional programming languages) can actually match binary strings also. This makes it much easier to deal with binary data packets.

Erlang has inbuilt support and language features for handling failures of processors, and which process takes over the job and so on, supervisor processes, etc.

Erlang allows you to think beyond for loops. Create processes which sit around waiting for instructions from you. And then the primary paradigm of programming is to send a bunch of tasks to a bunch of processes in parallel, and wait for results to come back.

Some erlang applications for developers:

  • Webservers built in erlang: Yaws, mochiweb, nitrogen, misultin
  • Databases built in erlang: amazon simpledb, riak, couch, dynomite, hibari, scalaris
  • Testing frameworks: distil, eunit, quickcheck, tsung

Who is using erlang? Amazon (simpledb), Facebook (facebook chat), microsoft, github, nokia (disco crawler), ea (the games company), rabbitmq (a messaging application), ejabberd (the chat server, which has not crahsed in 10 years). Indian companies using erlang: geodesic, http://hover.in.

How Clojure handles the Expression Problem by Baishampayan Ghose

If you’ve gone deep into any programming language, you will find a reference to lisp somewhere. So, every programmer must be interested in lisp. To quote Eric Raymond:

LISP is worth learning for the profound enlightenment experience you will have when you finally get it. That experience will make you a better programmer for the rest of your days, even if you never actually use LISP itself a lot.

BG had conducted a 2 day Clojure tutorial in Pune a few months back, and he will happily do that again if there is enough interest. This talk is not about the basics of Clojure. It is talking about a specific problem, and how it is solved in Clojure, in the hope that it gives some interesting insights into Clojure.

Clojure is a dialect of lisp. And the first thing that anybody notices about lisp is all the parantheses. Don’t be afraid of the parantheses. After a few days of coding in lisp, you will stop noticing them.

Clojure has:

  • first-class regular expressions. A # followed by a string is a regular expression.
  • arbitrary precision integers and doubles. So don’t worry about the data-type of your numbers. (It internally uses the appropriately sized data types.)
  • code as data and data as code. Clojure (and lisp) is homoiconic. So lisp code is just lists, and hence can be manipulated in the program by your program to create new program constructs. This is the most ‘difficult’ and most powerful part of all lisp based languages. Google for “macros in lisp” to learn more. Most people don’t “get” this for a long time, and when they “get” lisp macros, the suddenly become very productive in lisp.
  • has a nice way to attach metadata to functions. For example, type hints attached to functions can help improve performance
  • possibility of speed. With proper type-hints, Clojure can be as fast as Java

_(Sorry: had to leave the talk early because of some other commitments. Will try to update this article later (in a day or two) based on inputs from other people.)

Clojure, Erlang, & Functional Programming – Intro to FP & Why It’s Important – TechWeekend5 18 Dec

Have you heard of Clojure, Erlang, Scala, F# and wondered why people are getting all excited about these new fangled languages? Then this is your chance to find out. And if you are a programmer or are otherwise working in the software technology space and have not heard any of those names, then you need to start reading more, and you certainly need to attend this TechWeekend5 in Pune this Saturday. Register for the event here.

Vayana Services and TechWeekend Pune presents a detailed session on Functional Programming this Saturday, 18th December from 10am to 1pm, at Sumant Moolgaonkar Auditorium, MCCIA in ICC Trade Tower (A Wing, Ground floor), S.B. Road. You must attend.

Object-Oriented Programming is now passe, and all the cool kids (i.e. the star programmers) have started looking very seriously at functional programming languages like Clojure and Erlang. The more visionary ones (like our speakers this week: Dhananjay Nene, Bhasker Kode, and Baishampayan Ghose) are building the next generation of products in these languages.

Find out the What, the Why and the How on Saturday.

There will be three talks, listed below, and some time for general discussions around this topic.

Why you should care about functional programming – by Dhananjay Nene

This talk will focus on important characteristics of functional programming and the current landscape in terms of variety of languages and its adoption. The talk will also refer to how leveraging it can help you in terms of brevity, concurrency, better abstractions, testability, economics and particularly enjoyability. A small part of the talk will also focus very superficially on the Scala programming language.

About the Speaker – Dhananjay Nene

Dhananjay is a passionate programmer and a consulting software architect. He loves to learn, research, prototype and deploy new technologies and languages even as he is strongly focused on ensuring that the choices are made consistent with the business objectives and landscape. He currently writes code for and advises Vayana Enterprises in his role as its Chief Architect.

An Introduction to Erlang – by Bhasker Kode

While ideating hover.in towards the end of 2007 Bhasker soon become an ardent evangelist of Erlang and it’s fault tolerant nature traditionally intended for use in telecom & messaging circles. Following it’s rising use in building real-time and low-latency applications at web scale Bhasker has presented Hover’s erlang growth stories at Commercial Users of Functional Programming Conference in Edinburgh along with Facebook, Erlang Factory in London, and Foss.in in Bangalore talking about the role of functional programming. Hover’s engineering efforts can be tracked at http://developers.hover.in

About the Speaker – Bhasker Kode

Bhasker is the CEO and Co-Founder of Pune-based Hover Technologies, a user-engagement platform that allows web publishers to add a new channel of earning ad revenue through the use of in-text “tooltip” based ads. He has always been captured by the potential of the internet as part of the core team behind several destination portals and startups from his college days in Chennai. His introduction to functional programming came from his stint as the first few developers at Bangalore based Tutorvista where he built the calendar, syndication, whiteboard among other products used by thousands across the world everyday.

Clojure & its solution to the Expression Problem – Baishampayan Ghose

The “Expression Problem” arises when we want to add new functionality to a library that we don’t control. Most popular programming languages accomplish this task by Monkey Patching, Wrapper Classes, etc. In this talk, BG will discuss the demerits of traditional approaches to the problem and how Clojure solves this problem using Protocols. This talk is intended to show-off the real power of Clojure in solving complex problems.

BG has chosen to talk about a particular feature of Clojure in depth instead of skimming over many things in a hurry because he believes that Clojure’s approach to solving the Expression Problem clearly demonstrates the thought process that has gone into designing the language and shows how it’s different from most other programming languages. I will also cover the very basics of reading Clojure code in just a few minutes which will also demonstrate the simplicity of the language itself.

About the Speaker – Baishampayan Ghose

Baishampayan Ghose (mostly known as BG) is the co-founder & CTO of http://Paisa.com. He has been a career Functional Programmer and has programmed professionally in Common Lisp, Clojure & now Erlang.

About the Sponsor – Vayana Services

Vayana Services offers an easier option for small and medium enterprises to obtain working capital financing from banks by electronically sourcing, transferring and tracking digitally signed trade documents across trading parties and banks. It is a financial service backed by a cloud based offering with its development and operations management team based in Pune. With a strong belief that healthy businesses are greatly assisted by using healthy technology, Vayana Services looks forward to an increasingly frequent and high quality interaction within the software technology community in Pune and welcomes you all to Techweekend 5.

Logistics

This event is free for all to attend, but please register here. The event is in MCCIA’s Sumant Moolgaokar Auditorium, ICC Towers, Wing A, Ground Floor. From 10am-1pm. The hashtag for the event is #tw5

Web Scalability and Performance – Real Life Lessons (Pune TechWeekend #3)

Last Saturday, we had TechWeekend #3 in Pune, on the theme of Website Scalability and Performance.  Mukul Kumar, co-founder, and VP of Engineering at Pubmatic, talked about the hard lessons in scalability they learnt on their way to building a web service that serves billions of ad impressions per month.

Here are the slides used by Mukul. If you cannot see the slides, click here.
Web Scalability & Performance

The talk was live-tweeted by @punetechlive and @d7ylive. Here are a few highlights from the talk:

  • Keep it simple: If you cannot explain your application to your sales staff, you probably won’t be able to scale it!
  • Use JMeter to monitor performance, to a good job of scaling your site
  • Performance testing idea: Take 15/20 Amazon EC2 servers, run JMeter with 200threads on each for 10 hours. Bang on your website! (a few days later, @d7y pointed out that using openSTA instead of JMeter can give you upto 500 threads per server even on old machines.)
  • Scaling your application: have a loosely coupled, shared nothing, stateless, distributed architecture
  • Mysql scalability tip: Be careful before using new features, or new versions. Or don’t use them at all!
  • Website scalability: think global. Some servers in California, some servers in London, etc. Similarly, think global when designing your app. Having servers across the world will drive architecture decisions. When half your data-center is 3000 miles from the other half, interesting, non-trivial problems start cropping up. Also, think carefully about horizontal scaling (lots of cheap servers) vs vertical scaling (few big fat servers)
  • memcache tip: pre-populate memcache with most common objects
  • Scalability tip: Get a hardware load balancer (if you can afford one). Amazon AWS has some load-balancers, but they don’t perform so well
  • Remember the YouTube algo for scaling:
    while(1){
    identify_and_fix_bottlenecks();
    eat_drink();
    sleep();
    notice_new_bottleneck();
    }

    there’s no alternative to this.
  • Scalability tip: You can’t be sure of your performance unless you test with real load, real env, real hardware, real software!
  • Scalability tip – keep the various replicated copies of data loosely consistent. Speeds up your updates. But, figure out which parts of your database must be consistent at all times, and which ones can have “eventual consisteny”
  • Hard lessons: keep spare servers at all times. Keep servers independent – on failure shouldn’t affect other servers
  • Hard lessons: Keep all commands in a script. You will have to run them at 2am. Then 3am. Then 7am.
  • Hard lessons: Have a well defined process for fault identification, communication and resolution (because figuring these things out at 2am, with a site that is down, is terrible.)
  • Hard lessons: Monitor your web service from 12 cities around the world!
  • Hard lesson, Be Paranoid – At any time: servers can go down, DDOS can happen, NICs can become slow or fail!

Note: a few readers of of the live-tweets asked questions from Nashik and Bombay, and got them answered by Mukul. +1 for twitter. You should also start following.

Reblog this post [with Zemanta]

TechWeekend #3: Website Performance, Scalability and Availability: Sept 5

Scalability (Source: Domas Mituzas, Wikipedia)
Click on the image to see other PuneTech articles on Scalability (Image Source: Domas Mituzas, Wikipedia)
What: TechWeekend featuring “Website Scalability and Performance” by Mukul Kumar, VP Engineering at Pubmatic, and “Website Availability and Recovering from Failures and Disasters” by Sameer Anja, Associate Director at KPMG
When: Saturday, 5th Sept, 4pm
Where: Symbiosis Institute of Computer Studies and Research, Atur Centre, Model Colony. Map.
Registration and Fees: This event is free for all to attend. Please register here.

Website Scalability and Performance – Mukul Kumar

Mukul will talk about the various aspects of what it takes to run a very high traffic website – something that he has a lot of experience with at Pubmatic, the ad optimization service for web publishers, where they serve over a billion requests per month.

Mukul Kumar (mukul.kumar [at] pubmatic [dot] com) is a Co-Founder and VP of Engineering at Pubmatic, and Mukul is responsible for PubMatic’s engineering team and resides in Pune, India. Mukul was previously the Director of Engineering at PANTA Systems, a high performance computing startup. Previous to that he joined VERITAS India as the 13th employee and helped it grow to over 2,000 individuals as Director of Engineering for the NetBackup group, Veritas’ main product. He has filed for 14 patents in systems software, storage software, and application software and proudly proclaims his love of Π and can recite it to 60 digits. Mukul is a graduate of IIT Kharagpur with a degree in electrical engineering.

Website Availability and Recovery from Disasters – Sameer Anja

While everyone looks at security and focuses on confidentiality, privacy and integrity; an oft neglected parameter is of availability. While “neglected” may be seem like a strong term, the truth is that we overlook basic data on availability and do not even implement simple to-dos which would help in remediating the situation. The session is aimed at identifying such simple remedies, look at impacts, the assessment model and put forward various scenarios and possible solutions available. The session does not focus on specific products and instead endeavours to use existing technologies used for web site development and how they can be used for ensuring availability. Some principles of disaster recovery will also be covered.

Sameer is a Senior Manager in the IT Advisory practice and is working with KPMG since January 2007 and has 12+ years of work experience in the areas of Information Security, Product design and development, system and network administration. Worked on process and technology areas of Information Security. Worked on Governance and Compliance areas like SOX, Basel II, ISO 15048, SSE -CMM, Data Privacy apart from ISO 27001, Identity Management and Business Continuity design and testing. Experience working with startups and established setups. Speaker at various conferences/ seminars within India and abroad. Trained for six sigma green belt.

TechWeekend: Three tech talks on Google Android, 4:30pm, 1st Aug

Image representing Android as depicted in Crun...
Image via CrunchBase

What: TechWeekend featuring “What makes Google Android different from other systems, and from regular Java” by Navin Kabra, “Maps, GPS and sensors in Android” by Rohit Ghatol, with a demo on an Android G1 phone, and more
When: Saturday, 1st August, 4:30pm
Where: Symbiosis Institute of Computer Studies and Research, Atur Centre, Model Colony. Map.
Registration and Fees: This event is free for all to attend. Please register here.

What makes Google Android different from other systems – Navin Kabra.

Google’s Android is a brand new platform for mobile phones, and has been created from scratch specifically for this purpose. This means that it is a “modern” system that does not suffer from any legacy issues, and has taken the best ideas from various other projects to build a system that is arguably better than any of the other, competing, systems. Thus, for example, it uses the Java language as the development language, but has rejected the rest of the Java ecosystem. Specifically it uses a compeletely new virtual machine (Dalvik) which is redesigned with mobiles in mind – and has a number of very interesting design decisions that we will discuss. Similarly, the Android application framework represents a departure from the traditional way of doing things, and has a learning curve, but once you get used to it, it is great, especially for allowing different apps to share data, code, and in general co-operate.

We will explore and discuss this and various other design decisions in Android.

Maps, GPS and Sensor Capability – Rohit Ghatol

Rohit Ghatol is a co-founder of the Pune Google Technologies User Group (PuneGTUG), and one of the early adopters of the Google Android platform. He has already built a number of applications on Android, and is working on some interesting ideas in this area. In this talk, he will be discussing the mapping in the Android framework, and how the GPS and sensor capabilities can be combined with it to give powerful and rich experiences to users. He will be using his Google Android G1 phone to demonstrate these capabilities.

There will be one more speaker, but unfortunately, the details of that talk were not ready in time for this announcement. Please check back in a day for that update.

Also check out Dhananjay Nene‘s slides from the first TechWeekend: “REST Explained.”

Reblog this post [with Zemanta]