Day 4 - Data Labeling: SageMaker Ground Truth, Active Learning, Label Quality

The workforce (who does the labeling) is chosen based on data sensitivity and cost.

💡 Related theory: Exposing sensitive data (patient imaging, PII) to a public workforce like Mechanical Turk is a compliance violation. In such cases you must use a Private workforce (internal) or a trusted Vendor. The Specialty exam frequently asks "which labeling workforce for sensitive medical/financial data?", and the answer is almost always Private or Vendor. Choosing Mechanical Turk based on cost and scale alone is walking into the trap.

Active Learning: Cutting Costs with Automated Labeling

Ground Truth's core cost-saving feature is automated data labeling (active learning). How it works:

1. Humans label a portion of the data (the seed).
2. A model is trained on those labels.
3. The model scores predictions over the full dataset, and high-confidence predictions are adopted as automatic labels.
4. Only the low-confidence (ambiguous) items are sent back to humans.
5. The model is retrained on the newly human-labeled data, and the cycle repeats.

The key insight: by concentrating human effort on the borderline cases the model is unsure about, you achieve high quality even though humans label only a fraction of the full dataset.

💡 Related theory: Active learning starts from the premise that "not all data is equally useful for training." Samples the model is already confident about carry low information value for additional labels, while ambiguous samples near the decision boundary (high-uncertainty samples) carry the most information. The strategy of selecting only these ambiguous samples to ask humans about is called uncertainty sampling. The more data you have and the tighter the labeling budget, the greater the payoff.

Guaranteeing Label Quality: Consensus and Verification

Humans err too. A single labeler makes mistakes from slips, bias, and fatigue. Ground Truth provides several mechanisms for quality.

• Consensus: Setting NumberOfHumanWorkersPerDataObject to 2-5 has multiple workers label the same object, and the results are aggregated (majority vote or weighted) into the final label.
• Annotation consolidation: The logic that merges multiple answers. Provided by default and replaceable with a custom Lambda.
• Quality verification workflows: A separate workforce reviews and corrects the labeling output.

A metric for measuring label quality is inter-annotator agreement (agreement between labelers, e.g., Cohen's kappa). Low agreement is a signal that the task guidelines are ambiguous or the task is inherently hard.

💡 Related theory: Label noise directly cuts into the upper bound of model performance. Consensus-based labeling cancels random errors through majority voting, reducing noise, but systematic bias — where all labelers err in the same direction (e.g., due to ambiguous guidelines) — cannot be caught by consensus. That is why you also need clear labeling guidelines and a procedure that validates labelers against a golden set (samples with known ground truth). Cost trades off against accuracy and labeler count, and reducing the amount of human labeling itself via active learning is the standard play that wins on both cost and quality.

Labeling Alternatives: Do You Really Need All That Data?

There are questions to ask before labeling. (1) Can a pretrained model + transfer learning get by with fewer labels? (2) Can data augmentation multiply the existing labels? (3) Can weak supervision provide heuristic labels? The Specialty exam often asks "ways to reduce labeling cost," and active learning and transfer learning show up as correct answers.