CDIT Comset Data Integration Toolkit

Accelerate the development of your reporting data warehouse with the Data Integration Toolkit.

CDIT OVERVIEW

The Comset Data Integration Toolkit (CDIT) complements the functionality of your chosen database and Data Warehousing technology.  It enables us to deliver robust, responsive reporting data warehouses and the supporting data transfer processes quickly and cost-effectively, using tried and tested components to dramatically reduce the time-to-value.

A reporting data warehouse provides a central data management platform optimised for the management information needs of a business.  By being independent of the underlying operational business systems, the data warehouse makes available a consistent, consolidated and integrated view of the business, structured in a way to specifically address reporting and analytic requirements.

Building on the experience of many data warehouse implementations, Comset have compiled a package of templates and processes to help ensure rapid and cost-effective development. 

CDIT - the Comset Data Integration Toolkit - Data warehouse schematic

DELIVERABLES

A development project using the Comset Data Integration Toolkit delivers a reporting data warehouse that includes:

  • A reporting data warehouse framework;
  • A documented working solution that can be expanded and maintained as your reporting and analysis requirements continue to evolve to meet the changing needs of your business;
  • A library of Comset Data Integration Toolkit templates to further enhance the reporting data warehouse, and that can be utilised by your own personnel;
  • A library of derived components from the toolkit that support the delivered system.

TOOLKIT COMPONENTS

A reporting data warehouse needs a very different structure to a transaction-oriented operational database.  It needs to be designed for ease and speed of data retrieval and navigation to meet reporting and analysis requirements, rather than speed of processing transactional data.  The dimensional modelling design technique that Comset consultants use seeks to present the data in a standard, intuitive framework that allows for high-performance query access.

The Comset Data Integration Toolkit supports the rapid development of the dimensional data warehouse by bringing together a set of templates and processes that encompass the knowledge and experience gained by Comset over many years of delivering such applications.  The Toolkit comprises the following components:

  • Standard SQL scripts for creation of tables, views, and stored procedures;
  • Standard SQL scripts, including tables, routines, and population processing, providing specific functionality for:
    • Global variables
    • Control
    • Auditing
  • Template routines, and required database structures, for various styles and types of data loading, processing, and validation:
    • Full reload
    • Incremental load
    • Track changes
    • Snapshots
  • Standard scripts for MS SQL Server Integration Services (SSIS) functionality;
  • Template MS SQL Server Integration Services (SSIS) packages;
  • System documentation templates.

DATA WAREHOUSE DEVELOPMENT

The Comset Data Integration Toolkit includes processes and procedures to manage these four key areas of data warehouse development:

Dimension Maintenance

  • Automatic Updates from source systems:  Data is extracted direct and used to update the data warehouse (or raise errors);
  • Manual Updates:  Data is updated either using manually maintained files such as Excel spreadsheets or through an application to allow updating of the database tables.  The technique can be used for both full maintenance and problem correction;
  • Data Bridge: Transformation / Mapping:  This functionality supports the mapping of data from source systems to different dimensional structures, and can include defaults or rejections for error conditions such as unknown members;
  • Support for Multi-Level Hierarchies:  The maintenance process allows the creation of even and ragged dimensional trees;
  • Time Dimension Support:  This handles time logic and automatic dimension updating.

Measure Loading

  • Aggregation:  The data warehouse can store data at the aggregated level required for reporting, this is not necessarily at the full transactional level (which is maintained in the source system but can also be held in other tables in the data warehouse if that level of detail is needed);
  • Calculated Measures:  These can either be defined in the reporting system for generation at the point of reporting/extraction or held in the data warehouse, in which case they are calculated during loading of the data warehouse.  Which option is selected depends upon requirements and properties of the measures, and the capabilities of the reporting tool.

Versioning & Snapshots

  • Historic Querying:  This enables the querying of data as it was at a particular point in time, rather than the latest version;
  • Rollback:  This function enables you to roll the data warehouse back to a previous state.  Consideration needs to be given to how far back this should be allowed.

Logging & Auditing

  • Tracking:  This process maintains data about load success, issues, data volumes, etc;
  • Validation:  This handles checking the data for problems and recording the issues; if rejecting data then this allows for subsequent updating and reloading.

SUPPORTED PLATFORMS

The Comset Data Integration Toolkit supports Microsoft SQL Server from Microsoft SQL Server 2008 R2 onwards for the reporting data warehouse. 

When using the Toolkit, the supported source database technologies include, but are not restricted to:

  • Microsoft SQL Server
  • Oracle
  • IBM DB2
  • SAP Sybase
  • OLE DB and ODBC

BI TOOLS

The Comset Data Integration Toolkit shapes your corporate data allowing you to easily perform Operational and Analytic Reporting using your chosen BI Tool, such as:

Comset Data Integration Toolkit Summary

The CDIT Toolkit is a tried and tested set of templates and processes that encompasses the knowledge and experience of Comset gained over many years of developing reporting data warehouses.  The Toolkit enables Comset consultants to deliver robust, responsive reporting data warehouses by configuring standard SQL scripts and building on template routines to match specific business requirements.  This enables our clients to save on development time and costs. 

The data warehouse and supporting processes can be readily maintained and enhanced by your own personnel, to meet new and evolving business requirements.

Comset Capabilities

Using technology to help you make the right business decisions is at the heart of Comset’s service. Here’s what’s new in the field of Data.

Learn more

Microsoft’s BI offering took another leap forward with the launch of Fabric in May 2023. Here are its main components, and how it can help you…

Learn more

Business Objects remains one of the premium BI solutions, and part of SAP’s core BTP Platform. We’ve been an expert partner since its earliest days…

Learn more