Will Data Integration Break Out of the Silo?

How’s this for irony: a lot of integration projects create more silos.

How can this happen? It’s because companies implement integration technology and products on a project-focused basis. The integration projects concentrate on specific applications and processes, in other words they integrate silos. Each tactical integration task is completed, and eventually the company discovers that they need to integrate the silos they created with their integration projects.

The bad news is that many companies have already created these integration silos. The business groups that invested in these integration projects are now back to using spreadsheets, i.e. data shadows systems, to reconcile the conflicting numbers coming out of these silos.

The good news is that many IT and business groups are educated consumers, i.e. they have seen the result of their myopic integration efforts and understand they need to change their ways.

The Integration Sea Change

What should you do if you find yourself in this situation? Step back from the tunnel vision of tactical data integration projects and design an overall integration architecture. After you have this overall architectural blueprint you can then design your individual projects, fitting them into this overall blueprint. Just like you hand your house blueprint to various contractors so that everything fits together, you need your data integration blueprint to get things to work together too. This blueprint encompasses architectures for data, technology, product and information (business data transformed.)

Enterprise Data Management [EDM] is the blueprint (or holistic approach in consultant lingo) that people should be striving for. Integration efforts, be it enabled using EAI (Enterprise Application Integration), SOA (Service Oriented Architecture), EII (Enterprise Information Integration) or ETL (Extract Transform and Load) technologies, all involve integrating data.

Data integration processes involve mapping one or more data sources to a data target and transforming the data along the way. Regardless of whether you are using messaging, services or batch-driven technologies you are performing the same processes (only the transport is different.)

Rather than taking a common approach, most integration projects start with an entirely different set of technologies or products. However, these products and technologies overlap and, more importantly, the data they integrate overlaps. The result is silos and business people manually reconciling the data they just spent a lot of money and effort integrating.

Fortunately, if your company is in this situation you are not alone (misery loves company!) and great options have emerged in the market. First, there are best practices to design your data integration framework [DIF] or blueprint. Second, there are proven program and project approaches that incrementally build an EDM and migrate from your silos.

Finally, some of the top ETL (extract, transform & load) vendors have recognized the need for comprehensive approach and have transformed their ETL products to data integration suites. These suites have expanded beyond batch-oriented ETL to include EAI, EII and SOA. In addition, it is becoming increasingly common for these suites to offer data profiling and data quality functions. No longer are you forced to buy separate best-of-breed products supporting niche technologies. You can find products that will support your DIF.

The market leaders, according to Gartner Research (Magic Quadrant for Data Integration Tools, 2007), are Informatica (INFA) and IBM (IBM). Both companies have expanded their data integration capabilities through acquisitions. Most notably, IBM’s Information Server obtained a significant portion of its functionality from its acquisition of Ascential Software.

Forrester Research, concurs with the leaders (The Forrester Wave™: Enterprise ETL, Q2 2007)(pdf file) but feels that Oracle (ORCL) (via its Sunopsis acquisition) and SAP (via its BusinessObjects acquisition) are catching up. I’d also watch that smaller competitors such as Pervasive Software (PVSW) and Sybase (SY) are busy putting together their own suites.

Conclusions

Change is in the air, so companies need to shift from their tactical, project-focused approach to integration to an enterprise perspective. Many in the industry are realizing that they need to make their integration efforts provide comprehensive and consistent business information, and not create yet another silo to then reconcile with your other silos. The will and ability (products) are in place. This trend will gain more momentum this year providing case studies and knowledge to expand beyond technology’s early adopters.

FYI: This post was originally titled “No Quarter: Data Integration Suites enable EDM (Enterprise Data Management).” I have heard that some of my readers are not Led Zeppelin fans and a few have not even heard of them, making my obscure references a bit confusing.

(This is part of a series of posts on business intelligence & data warehousing trends for 2008.)

Comments

[Data Analyst guy]

The point you make is correct. Data overlaps across systems but it is organized and structured differently and over time, while the overlap continues, inconsistencies pop up. It is these inconsistencies that cause business heartburn. What data do I trust from which system and when do I trust it?

The problem is that no one knows what the rules are that relate the common data across systems. In addition, the tools that the major vendors provide do a good job of moving data... only after you know those cross-system rules. Those same vendors provide data analysis tools. But those tools only analyze one data source at a time. That means that to figure out how two systems relate to each other, a data analyst has to do that work by hand. Since data structures are different, and data formats are different and the rules that relate data across systems can be complex, this work can take a long time, is not accurate and puts integration projects at risk before they event get off the ground.

There is a new but small vendor, Exeros, whose product my company has purchased and they are the only company so far that does cross system data analysis and automates what we have previously had to do manually. The other integration vendors need to catch up to the Exeros functionality if our industry is going to be truly efficient at data integration.

However, the critical issue you miss is that the ETL companies d