For German readers: Es gibt eine deutsche Version dieses Blogeintrags.
The relationship between a data quality team and a migration project (such as from an in-house, host-based system to a standard ERP system such as SAP R/3) can be tricky. Often times, the project sees data quality as one more foe on their way to project success and tries to block access to the project. This blog post explores some ideas so that the two sides can become friends and the relationship can become mutually beneficial.
If migrations are often pursued ambitious targets for improving data quality. The project life is usually not much for them and after the migration has been then not only scrap grapple with the old data, but by migration have been added new problems. But it need not be: With an explicit monitoring of migration by DQM to data quality can actually be improved – to relieve the migration project and for the benefit of the company.
Deterioration of Data Quality in Migrations
When a new migration project is started, there are usually ambitious goals for improving data quality:
- We can (finally!) get rid of all the junk in the old system.
- Only good data can enter the new system.
- Automatic checks in the new system will stop the data quality from deteriorating.
Unfortunately, reality is often different. In order to make a deadline constraints are relaxed and the new data is squeezed into the new system. This can lead to the “worst of both worlds”: The old junk data is still around, migration has created new issues and there is no one around who knows how to use the new system. In that case, the focus has to shift on “running the new system”, instead of being able to fix the issues that were inherited.
The following sections present ideas how this deterioration can be prevented by cooperation between the migration project and DQM.
DQM before migration
Even before the the actual migration DQM can play an important role in supporting the migration project:
- Collect DQ problems
There are a lot of DQ problems that are already known before the migration project or that are uncovered during the trial runs of the migration. The DQM team can help in tracking these issues. - Prioritization of the DQ problems
In a second step, the accumulated problems prioritized in order to distinguish between important issues that must be handled before migration and those that are somewhat less urgent. - Resolve important DQ problems
With increasing pressure to keep the deadline the project will focus on resolving the issues that are preventing the migration. This can be achieved using the standard DQ process and tools. - Tracking of DQ problems
Identified DQ problems can be measured and tracked with the help of a DQ cockpit. A new cockpit can be implemented or an existing cockpit can be extended for the use of the migration project. The cockpits helps to track the progress towards acceptable error levels and makes sure that resolved issues do not flare up again. - Develop preventive measures in the new system
When developing or customizing the new system, DQM can help to design preventive measures to help improve data quality in the new system. For example, if one of the problem areas in the old system has incorrect ZIP codes, a function for address validation can be integrated.
At the beginning of cooperation, there are often reservations in the project, which sees in DQM, a new "source of trouble." But as soon as the project realizes that DQM helps them solve problems, DQM can quickly take a supporting role to the benefit of both sides.
DQM during migration
There are always problems during the actual migration of data from the old system to the new system. Maybe there are some known problems that were not deemed as preventing migration. But as it is not possible to completely simulate the migration in tests, some new, unexpected problems occur. Only in very rare cases the migration is postponed, and the problems have to be fixed in the new system after the migration is complete. Again, DQM can help in collecting, prioritizing and tracking of issues.
Another supporting function DQM can take during the migration is by helping with DQ tools such as a dedicated analysis environment. For example, this environment could be used to compare the data before and after migration, making sure all records have been migrated or identifying records that have to be manually edited after the migration.
DQM after migration
After the migration event is over, the migration project usually ends pretty quickly, and a lot of resources are suddenly no longer available. Therefore, it is critical that the work of the DQ team is not over after the migration as there is a number of important tasks left:
- Resolving open DQ problems
Both the problems deferred to after the migration and the new problems that occurred during the migration have to be prioritized and resolved. Again, this can be done using the standard DQ process and tools. - Ongoing tracking of all DQ problems
The DQ cockpit that has been filled before the migration has to be adapted to the new system and expanded as necessary. This enables an ongoing tracking of all DQ problems. Often this also results in preventive measures in the new system.
Again, the project is often grateful to have a pre-defined recipient for issues that could not be resolved by the project. (Of course this does not mean that all problems should be declared to be DQ issues.)
In many cases, an ongoing DQ practice can be built and established during a migration project.
Leave a Reply