Preventing the “Great DQM Fire”

For German readers: Es gibt eine deutsche Version dieses Blogeintrags.

When dealing with the Middle Ages, there are a lot massive fires that destroy entire cities. An example is the "Great Fire of London" (see Wikipedia article). It lasted four full days in September 1666 and destroyed four fifths of the City of London, including most of the medieval buildings, and resulted in about 100,000 homeless people.

Source: Wikipedia, Great Fire London
Today, there are numerous provisions for fire protection, so that fires occur much less frequently than before. And in the case of a fire, every town and every village has a professional or volunteer fire department that can respond quickly and prevent extensive damage.

This concept can also be applied to the data quality management.

Data Quality Middle Ages

Again and again, problems result from bad data quality. Typically, these issues occur in major projects (eg, data migration or process optimization), changes in personnel or when there is a high visibility to upper management. To achieve its objectives, the project must then take care of the problems in an ad-hoc fashion. After the immediate, pressing problems solved and the projects objectives are achieved, there is no need to introduce a regular DQM. Until the next DQ emergency occurs …

This "ad-hoc DQM" has many negatives, some examples:

• No single approach, no standard tools
• Only the main symptoms are eliminated, without resolving the root causes
• No preventive measures
• Investment in DQ tools cannot be justified for a single project (either too expensive or too time consuming)
• DQ issues are handled by various people and departments
• No predictability of budgets and resources, therefore external support is often needed
• When new problems occur, everything has to start again from scratch, resulting in high recurrent costs

Modern Era Data Quality

It therefore makes a lot of sense to establish a unit with the main task of regular, active data quality management. The objectives of this unit are to establish a general process, organization and technical platform to continuously monitor and improve of data quality. The relevant aspects will be discussed in further posts.