Have you built your DQ trust today?

May 10, 2010 by
Filed under: English 

For German readers: Es gibt eine deutsche Version dieses Blogeintrags.

In the last time, there have been quite a few posts on using “shame” as a tool for improving data quality. Here are just three from the top of my head:

Picture by Okinawa Soba, taken from flickr with a cc license

I’ve added some comments to these posts that I think that they are missing something. I wasn’t quite able to put my finger to it, not sure how to grab the “missing thing”, not really able to give it a name. In order to move the discussion forward, I’ve decided to go with “DQ trust” and try to explain my thinking a bit more. Let me know in the comments what you think!

The problem I see with the “public humiliation” aspect of what Rob and Jim are describing: It will only work in a certain environment – when the “riot act” gets what I would call a “wink wink, nudge nudge” aspect.  The “culprit” understands why the reaction is coming, but the whole thing is so much over the top that it can’t really be taken seriously. This results in taking the sting out of the “public humiliation” aspect and the riot act achieves its purpose.

In order for this to work, there has to be one of two things: Either you have to be a really good comedian (and I’m certainly not) so that you can spring this on a person you’ve hardly ever dealt with before. If your act backfires, you’ll also have to deal with that person’s boss, and I have found humor to decline when moving up the corporate ladder. Pretty risky to rely on that.

That leaves the second option: Your riot act has to have a background to it, and you must have built a reputation as a fervent defender of data quality in your organization – you must have built a trust in your data quality judgment. This way, a person or his boss can understand that your reaction is aimed at improving data quality, and not at public humiliating data quality villains.

Too often I find that people do not take enough time to build this data quality trust. As they say it takes a long time to build trust, but only a moment to destroy it forever. Here are some ideas of what to do to build the trust:

  • reserve judgment on someone’s actions for as long as possible – try to find out why people do things a certain way before telling them they are idiots
  • admit that you don’t know everything and try to learn constantly by interacting with different people from different departments to get a 360° view on the issues
  • help people to solve their problems – then they will be much more willing to help you when you need their support
  • make sure to explain data quality in terms the person understands – a business user doesn’t care too much about referential integrity unless you can explain how it affects his daily work
  • don’t be too academic in your data quality requirements – it doesn’t make sense to require perfect data quality for data that is never used

Even with this, whenever a new data quality issue comes and I’m shaking my head why anyone would come up with this harebrained scheme, I ask myself whether I’ve built enough trust to shame the person about it or not. Almost always, I come out on the side of caution and try to be firm on the issue, but avoid assigning personal blame. In the short term, this may not be quite as satisfying as “venting”, but has a much better chance of long-term success.


7 Comments on Have you built your DQ trust today?

    […] Dieser Eintrag wurde auf Twitter von Henrik L Sørensen, Thorsten Radde erwähnt. Thorsten Radde sagte: New post on my #dataquality blog: Have you built your DQ trust today? (http://cli.gs/9D3Wr) […]

  1. Ken O'Connor on Mon, 10th May 2010 17:43
  2. Hi Thorsten,

    This is an important topic – I’m glad you’ve raised it.

    In terms of “carrot and stick” approach – I greatly favour the carrot. I believe that a “public humiliation” approach is flawed, doomed to failure, and will cause bad feeling in an organisation.

    We need to make Data Quality “matter” to the person entering the data.

    I often refer to, and recommend, the “Ryanair Data Entry model”. This is the model now used by all low cost airlines. As passengers, we are required to enter our own data. We take care to ensure that each piece is correct – because it matters to us.

    With Ryanair, it is impossible to enter an Invalid date (e.g. 30Feb), but it is easy to enter the “wrong date” for our needs. E.g. We may wish to Fly on a Sunday, but by mistake we could enter the date for Monday.

    We ensure that we select the correct number of bags, since each one costs us money. We try to avoid having to pay for insurance, despite Ryanair’s best efforts to force it on us.

    It may not be easy to have data entry “matter” to the persons performing it – but this is what we must do. To achieve this, we must measure data quality at the point of entry, provide immediate feedback to the data entry person (helping them to get it right first time). Where appropriate, data quality should be included in a person’s performance review – reward for good data quality, and lack of reward for poor data quality (but not public humiliation).

    Apologies for long comment – but this is a topic I feel strongly about.

    Rgds Ken

  3. Thorsten on Mon, 10th May 2010 18:29
  4. Ken,

    no need to worry about the length of your comment – I’m glad you liked the idea! I agree with you that data entry should be correct – but that it is hard to motivate the people doing it. (Also, data entry is usually a “low level job”.) Maybe we can come up with some ideas on how to improve things on this front?


  5. Jim Harris on Mon, 10th May 2010 19:15
  6. Great post Thorsten,

    I definitely agree wasting time on blame and shame tactics, although somewhat cathartic, doesn’t help anyone truly understand the nature of data quality, nor does it help the organization move forward and make the necessary adjustments to minimize the likelihood of data quality issues recurring in the near future.

    And I really like the list that you have provided, with excellent recommendations for building trust.

    To Ken’s point about the “carrot” being better than the “stick,” I am currently reading the excellent book Drive – The Surprising Truth About What Motivates Us by Daniel Pink, where he shares research showing that using positive financial incentives to correct undesirable behavior often paradoxically has the opposite effect. Phil Wright also wrote an excellent blog post Can motivations impact the state of data quality? based on his reading of the book.

    Best Regards,


  7. Thorsten on Mon, 10th May 2010 21:30
  8. Jim,

    thanks for sharing the links you provided. Some more stuff for thought …


  9. Julian Schwarzenbach on Tue, 11th May 2010 09:12
  10. Thosten,

    I’ve put up a blog post partly in response to Jim’s Data Quality Jar posting – The Data Accident Investigation Board. This presents an alternate view and probably goes some way to address your concerns.


  11. Thorsten on Wed, 12th May 2010 09:26
  12. Julian,

    your “Data Accident Investigation Board” describes the frame of mind I’m looking for when looking at Data Quality issues. Thanks for sharing!


Tell me what you're thinking...
and oh, if you want a pic to show with your comment, go get a gravatar!