Data Warehousing - Clearing the Path to Knowledge ManagementData warehouses and data marts often fall short of expectations because inherently they do not address the translation of data into information or the dissemination of the information to those who need to know.
To move from data warehousing to knowledge management requires one to become successful at the art of turning data into information. The fact that data warehouses have numerous pitfalls and many failures led to data marts. But, data marts are only successful when done with data warehouses (or operation data stores).
Management and user expectations from technology are high, while awareness of the performance, functions and resulting business advantage is low. Data warehousing fits neatly into this dichotomy. Impressive and interesting as data may be, it means nothing until it is analyzed, applied or forwarded to those who may facilitate its transformation into useful actions and ideas. Put simply, trends in data warehousing are paving the way to knowledge management.
The main business incentive driving the growth of data warehouse expectations with senior management is the desire to better understand the company's business and turn this new found knowledge into a competitive advantage. For example, by analyzing its own internal data such as the typical buying patterns of large customers, a manufacturing company can streamline its production process, boosting profits and shareholder value. Similarly, businesses can gain important competitive knowledge by analyzing "external data."
Imagine how valuable it would be for a manufacturing company to know not just the purchasing patterns of its existing customers, but also those of the customers it would like to sell its products to. Building a new data warehouse or "tuning up" an existing data warehouse can provide more efficient access and use of disparate data sources within an organization, while combining this information with external sources can help measure the business against its competition. Opportunities for business process improvement become more apparent when a data warehouse combines data from other systems applications with external data.
But as many of the early adapters of data warehousing have learned, data warehouses can be more than just identifying new markets and exploiting niche and new opportunities; it can be about lowering costs, increasing business efficiency and meeting rising customer expectations. Data warehousing enters the next cycle of adoption as businesses push for more benefits from this exciting tool.
Data warehousing emerges as the first step on the way to achieving solutions with knowledge management initiatives. The information in a data warehouse provides a vital component to the central goal of knowledge management -- data interpretation and decision support. Applying data mining and interpreting that data facilitates knowledge management.
In addition, breaking up the information in a data warehouse into data marts for specific user groups can aid the progression toward knowledge management for two reasons. First, the preponderance of users can better handle data as a data mart and second, a data mart avoids some of the technology barriers associated with managing a large data warehouse. The case for data marts is best represented by the following paraphrased story attributed to Admiral Grace Hopper (the inventor of COBOL). A young farmer buys ten acres of land and a tractor to tend the land. The farmer plants a successful crop and decides to buy more land. This cycle continues until the farmer now owns thousands of acres. The farmer is faced with a problem, either buy more tractors or buy larger and larger tractors. Seems simple. The farmer buys more tractors. We should apply this same lesson to our data warehouses, data mart or decision support applications (and for that matter all our information technology problems). Bigger is not always the answer.
However, data marts must embody a well thought-out architecture. The architecture should define data marts derived from a data warehouse (or operational data store) so that business advantage can be achieved without creating a drag on technology staff or resources. A data mart must be built with an enterprise data model in mind to prevent "stove pipe" unintegrated or independent decision support systems, inconsistent business information and the inability to grow.
A data mart like a data warehouse is a decision support application, but this application or system focuses on solving a specific business problem in a single department or subject area (also referred to as the breadth of the data mart). Not all data marts are necessarily small. Single subject area data marts can often be very large and hold lots of historical data (also referred to as the depth of the data mart). This might seem like a contradiction; however, more breadth creates confusion, while more depth creates clarity and better understanding of the trends.
Unfortunately, too many businesses implement the "all data is better" and "bigger is better" data warehouse. This results in a business building an unmanageably large data warehouse containing all the data of the corporation, making it much more difficult for knowledge workers to sift for their golden nuggets.
Having established that a data warehouse can be a portal to knowledge management, several trends are emerging:
Data warehousing is expanding into an enterprise-wide application and will increasingly become commonplace on the path to knowledge management. As businesses gather information, automatically run analyses and then distribute that information -- most often via the Internet -- they enable their knowledge workers to achieve further competitive advantage. An example is a telecommunications company providing billing status, history and trends to customer service reps via an Intranet to better market to its customers.
Companies implementing Enterprise Resource Planning systems will need to analyze the information from the system not just report it. ERP will require analysis and reporting systems that I call Enterprise Data Analysis systems or EDA for short. The architecture for these new EDA systems will closely follow the architecture described earlier, a data warehouse and a set of dependent data marts. These EDA systems will include robust analysis, mining and reporting, along with automatic analysis and distribution (or even broadcasting via the Internet) of key trends and reports to the subscribing users in the company and even to key vendors and customers outside the company.
Simply building or implementing a client-server transaction system will no longer comprise a total solution. The transaction system must be accompanied by a management system to organize, analyze and interpret the data. Eventually, the planning and implementing of the data warehouse to accompany the new transaction system will become commonplace.
Problem definition and exploration of alternatives will drive the development of data warehousing and knowledge management. Companies must use the tremendous power of a data warehouse as a tool to help build its future. For example, if a company wants to improve its marketing, it must first focus on defining its problem. A well-designed data warehouse can facilitate this type of strategic analysis. Improvements may be required, not just in its customer relationship management, but also in order processing systems to add upselling or cross-selling, for example.
Data warehousing and knowledge management will mature as businesses say "let's use a data warehouse to analyze our problem," instead of "build a data warehouse and that will fix our problem."
Data warehousing solutions will move from Unix to Windows NT, allowing more and more businesses to build data warehouses. While Unix-based hardware and software vendors presently have greater mindshare than Windows NT-based hardware and software, this is changing with the introduction of Microsoft's SQL Server 7, the generally lower price point, and ongoing improvement in data warehousing tools.
While it is clear that data warehousing and knowledge management can provide a business with a competitive advantage, there remains confusion about why and how to apply data warehousing and knowledge management to an operation.
The functions, performance and reliability of data warehousing are often misunderstood and it is this misunderstanding that causes obstacles. This leads to people building large data warehouses for the sake of building a data warehouse without first understanding the business problem.
As business decision-makers realize not all the data in the company's ERP system needs to be in the data warehouse, then data warehousing will emerge as an enterprise-wide application instead of just a specialty tool. Every transaction system and every ERP system will have a companion data warehouse to provide the knowledge worker the ability to analyze the corporate data assets. EDA systems will automatically analyze and distribute, via the Internet, company information to enable their knowledge workers to gain further competitive advantage.
David R. May is the principal of the Metamor DataXpertsSM practice at Metamor Technologies Ltd., Chicago. His e-mail address is firstname.lastname@example.org.