Data Warehousing, Part 1: Building the Virtual Organization

Since the technology of data warehousing has risen to become common currency, it has been at the epicenter of leading-edge advances in database, data analysis and decision support systems theory and practice -- and for good reason.

With experts estimating that the amount of data in any given organization doubles every five years, and with much of this data dispersed across private and public networks using a variety of standards, protocols and operating systems, the question of how to best access, store, organize and make use of it is being examined in a new light.

From the basic and applied sciences across the engineering disciplines to public policy, planning and administration, and on through to commercial applications in enterprise sales , marketing and finance, data warehouse (DW) researchers are harnessing the power of the Internet and other information networks to design and build truly intuitive and stable data warehouse systems. These new systems are capable of supporting business intelligence (BI) applications that can put the combined data and information of entire libraries at a user's fingertips.

Cross-Fertilization

One of the great benefits of the Internet, and networking in general, is its ability to provide people working across organizations with the ability to exchange and share vast amounts of up-to-date historical data at the click of a mouse.

This is clearly evident in the increasing amount of cross-fertilization taking place within the diverse groupings of data warehouse and database system specialists around the world, as well as among the business executives and academic researchers working in a wide range of fields.

"We can make decisions and discoveries with data collected within our own business or [area of] research. But we improve decisions or increase the chance and scope of discoveries when we combine information from multiple sources," wrote Prof. Malcolm Atkinson in an article for the UK's National e-Science Center in Edinburgh. "Then correlations and patterns in the combined data can support new hypotheses, which can be tested and turned into useful knowledge. We see this requirement in medicine, engineering, sciences, economics and business."

Now serving as director of the e-Science Institute, Prof. Atkinson has been widely recognized for his work in the information sciences, which includes participating in the Center's OGSA-DAI (Open Grid Services Architecture Data Access and Integration) systems project. Part of an ongoing project stream, OGSA-DAI is one of the e-Science Center researchers' attempts to develop an information systems framework capable of supporting the data, information and communications needs of a very large-scale "virtual organization" for scientific research.

OGSA-DAI researchers are developing "middleware which supports a range of projects and applications by providing a Java-based product that allows multiple databases to be exposed via Web services," explained Neil P. Chue Hong, OGSA-DAI project manager.

"By using this standards-based approach, it is possible to provide a useful set of basic functionality, as well as facilitating the efficient development of plug-ins for application-specific data processing," Hong added. "This enables individuals to bring together public and private data from many different sources -- your own personal data store."

Spatially Enabled Data Warehouses

It has taken some time and a lot of doing, but data warehouse and DBMS specialists working in widely different fields are continually exchanging ideas, methods and techniques with their counterparts, as well as collaborating on ambitious data warehouse development projects. This is amply illustrated by the growing use of GIS (geographic information systems) data warehouse and business intelligence applications.

As we all know, humans are visually oriented animals. No other sense we possess is as well-developed as our vision. Yet our spatial thinking is relatively underdeveloped and underused. Unless one works in a visual, spatially-oriented field -- one of the earth sciences, for instance -- chances are, you pretty much stopped thinking spatially when you left high school.

That's been slowly changing, however. The idea of somehow fusing the distinct methods and applications used in the growing field of business intelligence and GIS, to more intuitively store and use data and information, has piqued the interest of many a business executive, academic researcher and practicing database specialist.

"The use of data warehouses has matured at most user sites and has proven to be [a valuable tool] in addressing the issues originally intended for them. These users are now looking to derive more value from their DWs, and the exploitation of the location data already contained in most DWs -- addresses for instance -- is a relatively inexpensive way to accomplish this," said Bob Ruschman, general press and computer trade media relations coordinator at ESRI, a pioneer and leading GIS software and data services vendor based in Redlands, Calif.

While this has been a "relatively obvious path to take," Ruschman noted, it's been somewhat slow to catch on. "However, with the availability of in-car navigation, personal GPS, and most recently Google (Nasdaq: GOOG) Maps and Microsoft (Nasdaq: MSFT) Virtual Earth/Windows Live Local, private organizations have become geographically aware, and are looking at the location data in their DWs as an untapped resource."

Data Warehousing, Part 2: When Worlds Collide

Data Warehousing, Part 3: One Step Beyond