Overview of Business Intelligence and Data Warehousing

Live Blogging, , , ,

I am liveblogging the CSI Pune lecture on Business Intelligence and Data Warehousing. These are quick-n-dirty notes, so please forgive the uneven flow and typos. This page is being updated every few minutes.

There’s a large turnout – over 100 people here.

Business Intelligence is an area that covers a number of different technologies for gathering, storing, analyzing and providing access to data that will help an large company make better business decisions. Includes decision support systems (i.e. databases that run complex queries (as opposed to databases that run simple transactions)), online analytical processing (OLAP), statistical analysis, forecasting and data mining. This is a huge market, with major players like Microsoft, Cognos, IBM, SAS, Business Objects, SPSS in the fray.

What kind of decisions does this help you with? How to cut costs. Better understanding of customers (which ones are credit worthy? which one are at most risk of switching to a competitor’s product?) Better planning of flow of goods or information in the enterprise.

This is not easy because amount of data is exploding. There’s too much data. Humans can’t make sense of all of them.

To manage this kind of information you need a big storage platform and a systematic way of storing all the information and being able to analyze the data (with the aforementioned complex queries). Collect together data from different sources in the enterprise. Pull from various production servers and stick it into an offline, big, fat database. This is called a data warehouse.

The data needs to be cleaned up quite a lot before it is usable. Inconsistencies between data from different data sources. Duplicates. Mis-matches. If you are combining all the data into one big database, it needs to be consistent and without duplicates. Then you start analyzing the data. Either with a human doing various reports and queries (OLAP), or the computer automatically finding interesting patterns (data mining).

Business Intelligence is an application that sits on top of the Data Warehouse.

Lots of difficult problems to be solved.

Many different data sources: flat files, CSVs, legacy systems, transactional databases. Need to pick updates from all these sources on a regular basis. How to do this incrementally and efficiently?  How often – daily, weekly, monthly? Parallelized loading for speed. How to do this without slowing down the production system. Might have to do this during a small window at night. So now you have to ensure that the loading finishes in the given time window.

This is the first lecture of a 6-lecture series. Next lectures will be Business Applications of BI. This will give an idea of which industries benefit from BI – specific examples: e.g. banking for assessing credit risk, fraud, etc. Then Data Management for BI. Various issues in handling large volumes of data; data quality, transformation and loading. These are huge issues, and need to be handled very carefully, to ensure that the performance remains acceptable in spite of the huge volumes. Next lecture is technology trends in BI. Where is this technology going in the future. Then one lecture on role of statistical techniques in BI. You’ll need a bit of a statistical background to appreciate this lecture. And final session on careers in BI. For detailed schedule and other info of this series, see the Pune Tech events calendar, which is the most comprehensive source of tech events info for Pune.

SAS R&D India works on Business Applications of BI (5 specific verticals like banking), on Data management, on some of the solutions. A little of the analytics – forecasting. Not working on core analytics – that is only at HQ.

We are trying to get the slides used in this talk from the speaker. Hopefully in a few days. Please check back by Monday.

Related articles by Zemanta
Reblog this post [with Zemanta]