Day 2 - Data: Structured vs Unstructured, Splitting Data, and the Power of Quality

📌 Key Takeaways

• Data divides into structured (tabular) and unstructured (images, audio, text), and most of the world's data is unstructured.
• There is only one reason to split data into training/validation/test sets: fair evaluation. The test set stays hidden until the very end.
• Data leakage, where test data gets mixed into training, inflates scores and then collapses in production.
• The 5 axes of data quality: accuracy, completeness, consistency, timeliness, and representativeness. Poor representativeness leads directly to bias.
• With class imbalance, plain accuracy is meaningless. A model that just answers "all normal" scores 99.9%.

Structured Data vs Unstructured Data

The most basic way to categorize data is whether it has a fixed structure.

• Structured: a table with fixed columns, like "customer ID, age, purchase amount."
• Unstructured: a single photo, an audio recording, or one line of a tweet, where there are no fixed columns.

There is also semi-structured data. JSON, XML, and log files are not full tables, but they carry some structure such as tags and keys. For the exam it is enough to know the two extremes, "structured vs unstructured," and that semi-structured sits in between.

A bank's transaction history (date, amount, account number) is classic structured data. A call-center recording or a free-form complaint written by a customer is unstructured data. Extracting meaning from unstructured data is exactly where deep learning and generative AI shine.

Why Split Data into Three Pieces

You don't use all the data you collect for training. In a word, it is "to take a fair exam."

• If a student saw the exam questions in advance, the score wouldn't show real ability.
• Likewise, measuring a model on the data it trained on only shows "it repeated what it memorized", which is meaningless.
• That is why test data is never used for training and is kept hidden until the end.

The difference between validation and test is easy to confuse. Validation data is used during training to ask "is this setting good?" and to tune the model (the practice test). Test data is used only once, after all tuning is finished, to measure final ability (the real exam). Typical split ratios are about 60-80% training, 10-20% validation, and 10-20% test (there is no fixed right answer).

When test data gets mixed into training data, it is called data leakage. The evaluation score then comes out unrealistically high, and you end up with a model that performs poorly in actual operation. Keeping the three sets cleanly separated is the first condition of fair evaluation.

Data Quality - What Decides a Model's Fate

"Garbage In, Garbage Out" - no matter how sophisticated the algorithm, you can't build a good model from bad data.

• Representativeness is especially important and is emphasized on the exam.
• Example: if a face recognition model is trained only on photos of one ethnicity, it recognizes faces of other ethnicities poorly.
• When data doesn't capture reality evenly, the model ends up with bias.

Data bias is a core topic in Responsible AI. A model's bias mostly comes from its data. If the data is skewed to one side, the model's judgments are skewed too. So "collecting diverse and representative data" is not just a performance issue; it is also a fairness and ethics issue.

🎯 Scenario: "We built a model to evaluate job applications, but trained it on data where past hires were mostly one gender. What problem arises?" → The data lacks representativeness and fairness, so the model learns and reproduces past bias as-is. This is a textbook failure of data quality (especially representativeness).

Data Quantity and Balance

• Generally, the more data you have, the more varied patterns the model can learn.
• For classification problems, class balance matters.
• Example: in fraud detection with 99.9% normal / 0.1% fraud, the model falls into the trap of looking 99.9% "accurate" even when it answers "all normal".

⚠️ Pitfall: With such imbalanced data, plain accuracy is almost meaningless. A model that answers "all normal" gets 99.9% accuracy but catches none of the fraud it was supposed to catch. That is why metrics beyond accuracy (precision and recall) are needed, which we cover tomorrow in Day 3.

Seeing the Imbalance Trap in Numbers

Suppose a model answers "all normal" on a dataset with 10 fraud cases (0.1%) out of 10,000 transactions.

• Accuracy is dragged along by "how well you got the majority right." The rarer the thing you are looking for, the less accuracy tells you about it.
• There are two responses. Change the metric (recall and precision, which you learn tomorrow). And fix the data: collect more examples of the rare class, or rebalance by raising the share of the rare class in the training data or reducing the common class.

Data Types the Exam Guide Talks About

Besides structured/unstructured, the exam guide names data along several other axes. Because the axes differ, one dataset can carry several names.

• They can be combined, as in "labeled images" or "unlabeled text." Data that is both structured and time-series (a daily sales table) is also common.

⚠️ Pitfall: Time-series data must not be split by random shuffling. If you shuffle, you can end up training on March data and testing on February data, so the model is evaluated as if it had "seen the future." Keep time order, train on the past and test on the most recent period, so the setup matches real usage.

The Three Faces of Data Leakage

Leakage isn't only "test data was accidentally put into training." What they all share is one thing - information that can't be known in production flowed into training.

• All three show the same symptom: the test score is suspiciously high, and the real service never reaches that score.
• The third is the hardest to find. The check question is "do we really know this value at the moment of prediction?"

Quality Problems: From Symptom to Cause

• In the Responsible AI domain, dataset characteristics are also asked about: inclusion, diversity, curated sources, and balance. Today's representativeness and balance are the foundation for that.
• AWS has Amazon SageMaker Clarify, which measures before training whether data is skewed toward certain groups (covered in Week 5).

Clearing Up Common Misconceptions

• "More data is always better" → Collecting more skewed data only entrenches the bias. Whether it covers things evenly comes before quantity.
• "If the test score is bad, just fix things and measure again on the same test data" → The more you repeat, the more you fit to the test data. Do tuning with validation data, and use test data last.
• "Unstructured data can't be used for machine learning" → It just needs an extra step to convert it to numbers. It is actually where deep learning does best.
• "Structured data has no quality concerns" → Even when neatly in a table, values can be wrong, missing, or skewed to one side. Structure and quality are separate matters.
• "Validation data and test data are the same thing" → Validation is used for choosing; test is used once, after all choices are made.

Tomorrow we look at how to measure how well a model built from this prepared data performs: accuracy, precision, recall, and overfitting and underfitting.

📖 Glossary

• Structured data: Data in a table with fixed rows and columns. Excel, DB tables, and CSV are typical.
• Unstructured data: Data that is hard to fit into table cells. Photos, audio, video, free-form text, and so on.
• Semi-structured data: Data that isn't a full table but has some structure such as tags or keys. JSON, XML, log files.
• Training data: Data the model studies from. Equivalent to the textbook.
• Validation data: Data used during training to check "is this setting OK?" Equivalent to the practice test.
• Test data: Data used just once, after all tuning is done, to measure final ability. Equivalent to the real exam.
• Data leakage: An incident where test data that should have stayed hidden gets mixed into training and inflates the score.
• Representativeness: How evenly the data captures the various cases of reality. When lacking, the model skews to one side.
• Bias: A state where a model's judgments skew toward a particular group or direction. Mostly comes from data.
• Class imbalance: A state where one class among the classification targets is overwhelmingly common and another is extremely rare. It makes accuracy untrustworthy.

📝 Practice Questions

Click a choice to reveal the answer and explanation.

Question 1

Which of the following is unstructured data?

• AA customer database table organized in rows and columns
• BA transaction-history CSV made of date, amount, and account number
• CA free-form voice call recording left by a customer
• DAn Excel sheet with age and gender columns

Question 2

What is the most fundamental reason to split data into training, validation, and test sets?

• ATo reduce data storage costs
• BTo evaluate performance fairly with data not used for training
• CTo speed up model training
• DTo encrypt the data

Question 3

What is it called when data set aside for testing gets mixed into the training data, producing unrealistically high evaluation scores?

• AData Drift
• BData Leakage
• CUnderfitting
• DClass Imbalance

Question 4

A face recognition model was trained only on photos of one ethnicity and then recognized faces of other ethnicities poorly. Which aspect of data quality was lacking?

• ATimeliness
• BConsistency
• CRepresentativeness
• DStorage capacity

Question 5

When evaluating a fraud detection model on data that is 99.9% normal transactions and 0.1% fraudulent transactions, what is the best reason not to rely on plain accuracy alone?

• AAccuracy always comes out as 0
• BAnswering "all normal" gives 99.9% accuracy, so a model that catches no fraud can be mistaken for a good one
• CThe data is structured, so accuracy can't be calculated
• DThe fraud data is unstructured