Articles

Data warehousing is a technique for collecting, managing, and presenting data to help people analyze and use that data effectively. It involves a large database designed to support management-level staff by providing all the relevant historical data for analysis. This chapter begins with a definition of data warehousing, followed by an overview of large-scale policy management to highlight the need for data warehousing. Next, an overview of an ETL framework is presented, along with a discussion of advanced ETL techniques. The chapter concludes with an outline of performance optimization techniques for data warehousing. Data warehousing is considered a key enabler for efficient reporting and analysis, with implementation choices ranging from cost-effective desktop systems to large-scale, mission-critical data marts and warehouses containing petabytes of data. Extract, transform, and load (ETL) systems remain one of the largest cost and effort areas within data warehouse development projects, requiring significant planning and resources to build, manage, and monitor the flow of data from source systems into the data warehouse. The technology and techniques used for ETL can greatly influence the success or failure of a data warehouse. Complex business requirements for data cleansing, loading, transformation, and integration have intensified, while operational plans for real-time and near-real-time reporting add additional challenges. Parallel loading mechanisms, incremental data loading, and runtime update and insert strategies not only improve ETL performance but also optimize data warehousing performance, particularly for large-scale policy management.

Seamless integration of information in organizations promotes not only the operational efficiency but also the quality of decisions made by managers. Real-time decision support systems enable organizations to evaluate organizational changes immediately and ideally gives a hint of problems before they even appear in the organization. Such real time systems are nowadays regarded as the front-line solutions for managing organizations effectively. The technological possibilities seem not to conquer management. For most companies the data is still dealt with traditional solutions, data is collected and reports are generated to evaluate the past occurrences which only gives information on what has happened in the organization. The problem with these non-real-time systems is the reflection of organizational condition very late. These are the common rear-mirror descriptions for what already has been. Managers are receiving information from their organizations too late and often too little to make optimal decisions. Is it not possible to manage operations in real-time? Is real-time decision support really needed? If so, why most organizations still rely on traditional reporting systems.