I never questioned that way of accessing the news. I always perceived the fact that I carefully choose my sources as very positive.

Now, what I read in the current issue of Books (a French literary magazine with an English name!) might change my view. Their special report “Internet contre la démocratie” (that I would translate to “Internet Against Democracy”) introduced me to Cass Sunstein’s work and to his “Group Polarization” theory.

That fact that I have the power to filter the news that reaches me tends to reinforce my preestablished thoughts. I basically cut myself from any alternative opinions. In the long run, that could lead to a radicalisation of my thoughts…

I found that idea very interesting and I have to admit that I prefer to read someone who shares my views. Now, how is the act of choosing a website different from the act of choosing a newspaper? The simple fact that I read that article is a proof that I am still exposed to new ideas… Is that only because I read a paper copy of the magazine?

I will try to answer that question, but obviously my only interest is to read your answers!!!

I should maybe start by defining what my interests in BI are. As discussed in a previous post, I think that there is more in that field than just BI suite of products. Also, my interests cover a wide variety of topics: I want to find out how BI can address specific business problems, I want to know about methodologies that can be used to successfully run BI projects, I want to learn about new technologies that can be used to support BI needs and finally I am interested in what BI vendors are releasing.

I gather most of that information on the Internet: I have subscribed to some RSS feeds, I follow a few people on Twitter, and obviously I have a list of sites in my bookmarks.

As of today, here is what is on my radar:

RSS feeds:

BI Questions Blog
Claudia Imhoff on BeyeNETWORK
Industry Research on BeyeNETWORK
Articles by Bill Inmon on BeyeNETWORK
Stacia’s blog on Data Inspirations
Erika’s blog on Data Inspirations
North Star BI
Open Data Protocol

Twitter:

I have created a list @HerveSchnegg/bi following @pentaho, @herveschnegg, @W_T, @jamiet, @businessobjects.

Bookmarks:

Here are my Delicious bookmarks tagged BI and datawarehouse.

After a year, I finally followed Lauren’s recommendation and attended a Power Vinyasa class at Yogaworks.

She was right! The school is nice and cosy, the teacher is motivating and the class is… hard!

The class was not sticking to the traditional Ashtanga primary series as the emphasis was mainly on standing postures linked through vinyasas. I was pleased to still feel some sense of alignment after such a long break. My main problem has been to breathe properly as I struggled to follow the class. I found out that despite my attempts to run, my level of fitness is quite low.

Once again, the feeling after the class was great (obviously, I am not talking about the aching shoulders!). This state of serenity is something that I only experience after yoga. I really hope to find the time and energy to come back to a regular practice.

I wrote about my first experience with Pentaho Data Integration (PDI) a fortnight ago. Yesterday’s buzz about the announcement of PDI 4.0 makes me think that my choice was quite right!

With PDI 4.0, Pentaho also announced Agile BI and the promise to move from "data source to dashboard in minutes".

Agile BI is a methodology based on Agile Development where business users are involved in an iterative cycle of development.

To support that, PDI 4.0 has been organised in three perspectives:

• Data Integration Perspective: Classic ETL features
• Visualisation Perspective: Representation of the data in reports
• Modeling Perspective: Manipulation of OLAP and Reporting metadata

(PDI 4.0 new capabilities)

That means that all the stages of BI development from sources to reports are managed from within PDI.

The conclusion of my last post was that I wanted to learn more about Pentaho. Yesterday’s announcement only reinforced that view!

That is very frustrating for someone who developed some expertise in all the jobs encompassed in a typical DW/BI project…

In my view, the ideal DW/BI job should give me opportunities to use my experience in the following roles:

Business Analyst

As suggested by the name, Business Intelligence is about delivering the right information to the business. Trying to do BI without understanding the business needs in reporting and analysis is nonsense. BI professionals need to gain an in depth understanding of the business activity. Therefore, every BI professional has to be a Business Analyst.

Data Modeler

Both Kimball‘s and Inmon‘s followers know that it is not possible to build a successful BI project on a flawed data warehouse. Hence, DW/BI professionals need to understand data warehousing modeling principles.

Data Base Administrator (DBA)

Reporting and analysis projects usually imply the manipulation of huge volumes of data. Tuning an RDBMS to cope with that requires DBA skills.

OLAP Developer

Many analytical and data mining needs cannot be addressed without using OLAP cubes. Designing those multidimensional structures requires some specific skills.

ETL Developer

The data warehouse is usually loaded by ETL jobs. Designing and maintaining those jobs is an integral part of the DW/BI project.

BI Suite Developer

The tip of the iceberg… Deploying a BI suite of products and developing reports is what will bring the required information to the business. Obviously, that is an important part of the DW/BI activity. However, if any of the other bricks is missing, that job cannot be successful. In my view, it is sad that all the fuss around BI is at that level only…

Project Manager

Finally, strong Project Managers are required to coordinate all of those tasks.

I started to build a prototype for a complex dashboard. Part of the work required to integrate data stored in heterogeneous sources (Oracle, MySQL, spreadsheets). A classic dashboarding project… However, the situation became trickier when more and more users started to use the prototype. Starting from there, the time spent maintaining and refreshing the dashboard prevented me to normally carry on with my development work…

The obvious solution would have been to start industrialising the process and to develop ETL (Extract, Transform and Load) jobs. Unfortunately, as part of my sources were sitting on test servers and part of them on live servers, it was not possible to use the available corporate ETL tool to achieve that.

Here enters Pentaho Data Integgration (PDI), formerly known as KETTLE.

Installation

PDI is a Java application and its installation is very easy. First, make sure that a Java Runtime Environment (version 1.5 or higher) is running on your machine.

Then, download the latest PDI version (3.2.0 in my case).

Unzip the binary distribution and run spoon.bat (on Windows) or spoon.sh on other platforms.

When you first start PDI, a repository needs to be created. For that, you will need an access to a database instance. Create an empty database in the RDBMS of you choice, point PDI to that newly created database and let PDI create and populate the required entities.

Initial user / password for the newly created  repository is admin / admin.

Graphical Interface

That was a very nice surprise to find a complete graphical interface in an open source product. The screen is composed of a Core Object Palate (toolbox) in the left pane and a Workspace.

Jobs and Transformations are designed by dragging tools from the Core Object Palate to the Workspace. The connectors used to link those tools in the Workspace are called Hops in PDI.

Available Tools

All the tools required to achieve what I wanted to do were easy enough to find. The additional time needed to get used to PDI was almost zero (most of my ETL experience is with Microsoft SSIS).

Connections to various sources (Oracle 10g, MySQL and MS Excel) were very easy to set up. All the required drivers were available.

Conclusions

First of all, the problem I initially wanted to address has been solved. I now have a level of automation that leaves me with enough time to add what needs to be added to my dashboard.

I also discovered a product that can be used to quickly test ETL concepts. The fact that the installation footprint is almost nonexistent means that PDI can be easily used in those situations where political barriers would prevent me to use another ETL tool.

Notes:

• I am not saying that PDI should not be used as a corporate ETL solution!
• I know that there is more than Spoon to PDI and I know that PDI is more than a desktop application. However, Spoon is all I needed on my project. I hope to find out more about PDI in the future.

Finally, I hope that I will soon find an opportunity to discover more of the Pentaho suite. As an OLAP specialist, I am particularly looking forward to doing some work with Pentaho Analysis Services (Mondrian).

Cognos
Business Objects
Information Builders

Against The Day

A combination of the number of plots, the number of characters and the number of pages coupled with the fact that I do not have so much time to read right now is blocking me. I never gave up reading a book that I liked. Now as the pile of books to read is growing, I do not know what to do… If I leave it behind, I will certainly never come back to it…

Procrastination?

That is a funny definition. Everything is said, but is it not too generic? At least, that would be a good explanation of what the objective of a BI project should be.

Then as I thought again about it, I wondered who came first with that definition. I was sure enough that it was not mine as I heard it hundreds of times in the past! I googled it and found that everyone used it: BI vendors, consultants, bloggers…

Do we have both a definition and a “public domain” slogan?

datamorphose ltd was the company through which I billed my work when I used to work as a BI contractor. The company no longer exists, but I kept the domain name as so many people only know that address.

Also, and despite the fact that the name always sounded weird to English speaking people, I have a sentimental attachment to it… For a French speaker, the name sounded quite relevant!