Six Sigma In Data Warehousing

Most quality professionals recognize Six Sigma as a breakthrough strategy. They understand that the methodology uses data to measure current process effectiveness and to validate improvement; uses proven statistical and quality tools to identify process gaps and improvement solutions; and uses change management processes for institutionalizing and integrating improvements in an organizations' operational and business strategy. Still, it is curious that so many do not understand how Six Sigma can be used in the information technology industry to improve processes and produce bottom-line benefits.

Now when it comes to data warehousing, the primary reason that corporations introduce Six Sigma into data warehousing boils down to cost reduction. With large amount of data piled up everyday, there is a need to store these data and with this comes cost. Well you cannot neglect it as it helps for future prediction. There is no denying the fact that data warehousing is in a way, the powerhouse of Six Sigma deployment. In early stages of projects, data warehousing allows for better planning of deployment, design and tuning of the production environment.

If we think of the problem faced during managing data we may end up with a long list.  We should also look at the causes, here are few...

    * Wrong logic for data filter and transformation.

    * Missing or duplicate data in the data warehouse.

    * Record rejection in the extract-transform-load process.

    * Data missing in the input file.

    * Improper recovery and restart of load during interim load failure.

    * Missing loads.

It is not new that modern day data warehouses are built for auto refreshing and/or compatible for at least real time updating. ETL, as extraction, transformation and loading of data flow is a very resource-consuming exercise in data warehousing. The importance of data warehousing increases several times, considering the fact that data structures are both strategic and functional.

The recent trend in data warehouse development is to treat them as belonging to the same family or group. Consider dedicating each family to a particular geographical location, and other subsets of respective hierarchical data.

Applying Six Sigma elements into software development typically helps in identifying potential problems in production if the development is done in the early stages of the project. Secondly, the mammoth task of data warehousing can return positive results if deployment plans are fine tuned before implementation. The self-assessing nature and the provisions for internal auditing shed light on the course of implementation. At the same time, one cannot forget that databases developed remain tied to the system architecture on which they are built and bear heavily on the accuracy of predictions in a fluctuating business environment, ironically for which they are built.

Now if we want to structure these six sigma solution then we have

• Automating the process of checking the input file layout correctness and data completeness.
• Identifying and fixing the extract-transform-load logic.
• Analyzing the rejected records during the loading process and reloading them.
• Auditing interim data in various critical tables of the data warehouse.
• Using a checklist for tracking the loading process.
• Preparing and using guidelines for the load recovery process in case of interim load failure.
• Defining and using critical data validation processes to check data accuracy in the data warehouse before releasing it to users.

Six sigma has proved its utility in many different fields and sectors. Dataware housing will be benefited with the application of six sigma.