Day 2 - Data Quality and Validation: Glue Data Quality and Quality Gates

"Garbage in, garbage out." No matter how robust the pipeline, if incoming data is wrong, results are unreliable. Today we measure and enforce data quality, filter bad data, and prepare for reprocessing.

Six Dimensions of Data Quality

Data quality typically spans six dimensions:

• Completeness: Required values not empty (NULL ratio).
• Accuracy: Match actual fact.
• Consistency: No contradictions across systems/columns.
• Validity: Follow defined format, range, domain.
• Uniqueness: No duplicates (PK duplicates).
• Timeliness: Data sufficiently fresh.

💡 Related Theory: Place quality validation as far upstream (right after collection) as possible