Will Data Integration Break Out of the Silo?

by: Rick Sherman

How’s this for irony: a lot of integration projects create more silos.

How can this happen? It’s because companies implement integration technology and products on a project-focused basis. The integration projects concentrate on specific applications and processes, in other words they integrate silos. Each tactical integration task is completed, and eventually the company discovers that they need to integrate the silos they created with their integration projects.

The bad news is that many companies have already created these integration silos. The business groups that invested in these integration projects are now back to using spreadsheets, i.e. data shadows systems, to reconcile the conflicting numbers coming out of these silos.

The good news is that many IT and business groups are educated consumers, i.e. they have seen the result of their myopic integration efforts and understand they need to change their ways.

The Integration Sea Change

What should you do if you find yourself in this situation? Step back from the tunnel vision of tactical data integration projects and design an overall integration architecture. After you have this overall architectural blueprint you can then design your individual projects, fitting them into this overall blueprint. Just like you hand your house blueprint to various contractors so that everything fits together, you need your data integration blueprint to get things to work together too. This blueprint encompasses architectures for data, technology, product and information (business data transformed.)

Enterprise Data Management [EDM] is the blueprint (or holistic approach in consultant lingo) that people should be striving for. Integration efforts, be it enabled using EAI (Enterprise Application Integration), SOA (Service Oriented Architecture), EII (Enterprise Information Integration) or ETL (Extract Transform and Load) technologies, all involve integrating data.

Data integration processes involve mapping one or more data sources to a data target and transforming the data along the way. Regardless of whether you are using messaging, services or batch-driven technologies you are performing the same processes (only the transport is different.)

Rather than taking a common approach, most integration projects start with an entirely different set of technologies or products. However, these products and technologies overlap and, more importantly, the data they integrate overlaps. The result is silos and business people manually reconciling the data they just spent a lot of money and effort integrating.

Fortunately, if your company is in this situation you are not alone (misery loves company!) and great options have emerged in the market. First, there are best practices to design your data integration framework [DIF] or blueprint. Second, there are proven program and project approaches that incrementally build an EDM and migrate from your silos.

Finally, some of the top ETL (extract, transform & load) vendors have recognized the need for comprehensive approach and have transformed their ETL products to data integration suites. These suites have expanded beyond batch-oriented ETL to include EAI, EII and SOA. In addition, it is becoming increasingly common for these suites to offer data profiling and data quality functions. No longer are you forced to buy separate best-of-breed products supporting niche technologies. You can find products that will support your DIF.

The market leaders, according to Gartner Research (Magic Quadrant for Data Integration Tools, 2007), are Informatica (NASDAQ:INFA) and IBM (NYSE:IBM). Both companies have expanded their data integration capabilities through acquisitions. Most notably, IBM’s Information Server obtained a significant portion of its functionality from its acquisition of Ascential Software.

Forrester Research, concurs with the leaders (The Forrester Wave™: Enterprise ETL, Q2 2007)(pdf file) but feels that Oracle (NASDAQ:ORCL) (via its Sunopsis acquisition) and SAP (via its BusinessObjects acquisition) are catching up. I’d also watch that smaller competitors such as Pervasive Software (NASDAQ:PVSW) and Sybase (SY) are busy putting together their own suites.


Change is in the air, so companies need to shift from their tactical, project-focused approach to integration to an enterprise perspective. Many in the industry are realizing that they need to make their integration efforts provide comprehensive and consistent business information, and not create yet another silo to then reconcile with your other silos. The will and ability (products) are in place. This trend will gain more momentum this year providing case studies and knowledge to expand beyond technology’s early adopters.

FYI: This post was originally titled “No Quarter: Data Integration Suites enable EDM (Enterprise Data Management).” I have heard that some of my readers are not Led Zeppelin fans and a few have not even heard of them, making my obscure references a bit confusing.

(This is part of a series of posts on business intelligence & data warehousing trends for 2008.)