I'm the head of IT at Statistics Iceland. I describe myself as a hacker turned manager so that other hackers don't have to (but I sure do miss hacking all the time).
Nice post. I'm the author of an upcoming book (in early access program now) called The Art of Data Usability which is at its core about data quality. I've never thought of data quality as technical debt. That's a really nice way to frame it. I really like it :)
One thing I'd recommend is setting up monitoring of your quality attributes (like the incoherency you talk about). You monitor the attributes to make sure the quality continues to stay at the level you want (that you don't start collecting technical debt again) but you do it from the start (when you start the working on lowering the debt) to know when you've reached that level of quality. As you said, we start making mistakes, have a bad day or something. Monitoring quality helps us stay focused.
You can think of it as data quality tests. You monitor afterwards for regression testing and you develop the metrics and monitoring before you start as some sort of a TDD approach.
Again, a really good post and a fresh perspective on data quality.
Miguel Barba is licensed in Computer Engineering by the Instituto Superior Técnico, in Lisbon. He joined Accenture in November 2007 as a Junior Programmer. Since then he has been involved in Telco ...
"One thing I'd recommend is setting up monitoring of your quality attributes" - Yes, that would be the ideal scenario and it used to happen here but unfortunately the team responsible for doing it is from another department and our priorities and approaches to problem solving aren't always as aligned as they should be, so this ends up having a negative impact when it comes to detect and correct data issues on a regular basis.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Nice post. I'm the author of an upcoming book (in early access program now) called The Art of Data Usability which is at its core about data quality. I've never thought of data quality as technical debt. That's a really nice way to frame it. I really like it :)
One thing I'd recommend is setting up monitoring of your quality attributes (like the incoherency you talk about). You monitor the attributes to make sure the quality continues to stay at the level you want (that you don't start collecting technical debt again) but you do it from the start (when you start the working on lowering the debt) to know when you've reached that level of quality. As you said, we start making mistakes, have a bad day or something. Monitoring quality helps us stay focused.
You can think of it as data quality tests. You monitor afterwards for regression testing and you develop the metrics and monitoring before you start as some sort of a TDD approach.
Again, a really good post and a fresh perspective on data quality.
Thanks for the feedback.
And congrats on your book, by the way!
"One thing I'd recommend is setting up monitoring of your quality attributes" - Yes, that would be the ideal scenario and it used to happen here but unfortunately the team responsible for doing it is from another department and our priorities and approaches to problem solving aren't always as aligned as they should be, so this ends up having a negative impact when it comes to detect and correct data issues on a regular basis.