Day 4 - Data Labeling: SageMaker Ground Truth, Active Learning, Label Quality

📌 Key Takeaways

• The ceiling of a supervised model is set not by the algorithm but by label quality. "Garbage in, garbage out."
• A Ground Truth labeling job consists of four things: input data, task type, workforce, and UI template.
• For the workforce: Private or Vendor for sensitive data; Mechanical Turk for public, high-volume, low-cost work. Mechanical Turk is unsuitable for sensitive data.
• Active learning (automated data labeling) — the model auto-labels what it is confident about, and only the ambiguous items go to humans. This is the standard play for cutting cost.
• Consensus reduces random errors but cannot catch systematic bias. You need clear guidelines and a golden set.

SageMaker Ground Truth: The Labeling Workflow

The cost of having humans hand-label hundreds of thousands of images is hard to bear. The Specialty exam asks "how do you attach labels cheaply and accurately?", and Ground Truth sits at the center of the answer.

Ground Truth composes a labeling job from four pieces.

The output comes in the standard augmented manifest (JSON Lines) format, ready to use directly for training.

import boto3
sm = boto3.client("sagemaker")
 
sm.create_labeling_job(
    LabelingJobName="cat-dog-bbox",
    LabelAttributeName="annotations",
    InputConfig={"DataSource": {"S3DataSource": {
        "ManifestS3Uri": "s3://my-lake/labeling/input.manifest"}}},
    OutputConfig={"S3OutputPath": "s3://my-lake/labeling/output/"},
    RoleArn=role_arn,
    LabelCategoryConfigS3Uri="s3://my-lake/labeling/labels.json",
    HumanTaskConfig={
        "WorkteamArn": private_workteam_arn,        # workforce selection (table below)
        "PreHumanTaskLambdaArn": prehuman_lambda,
        "TaskTitle": "Draw boxes around cats and dogs",
        "NumberOfHumanWorkersPerDataObject": 3,     # 3 workers per object → consensus
        "TaskTimeLimitInSeconds": 300,
    },
)

Label Shape by Task Type

What humans do, and the shape of the result, differ by task type. Reading "what must be attached" in the passage determines the type.

• The higher up the table, the shorter the time per task; the further down, the more expensive. Segmentation is overwhelmingly more expensive than classification.
• To cut cost, first ask "do we really need segmentation, or are bounding boxes enough?"

Workforce Selection Table

Choose by looking at data sensitivity and cost together.

Exposing sensitive data such as patient imaging or PII to a public workforce like Mechanical Turk is a compliance violation. In that case you must use a Private workforce (internal) or a Vendor with NDAs and expertise. When the Specialty exam asks "which labeling workforce for sensitive data?", the answer is almost always Private or Vendor.

⚠️ Pitfall: Choosing Mechanical Turk by looking only at cost and scale is the classic trap in this area. If the passage contains even one of the words "patient," "PII," "regulated," or "confidential," immediately eliminate the public-workforce options.

Active Learning: Cutting Costs with Automated Labeling

Ground Truth's core cost-saving feature is automated data labeling (active learning). The sequence is as follows.

1. Humans label a portion (the seed)
        ▼
2. Train a model on those labels
        ▼
3. The model predicts over the full dataset
        ├─ High confidence → adopted as an automatic label (no human involvement)
        └─ Low confidence  → routed to humans
                          ▼
4. Humans label only the ambiguous items → return to step 2 and repeat

• Key insight: by concentrating human labor on the samples near the decision boundary that the model finds ambiguous, you get high quality while labeling only a fraction of the whole.
• The larger the dataset and the tighter the labeling budget, the greater the payoff.

# Adding LabelingJobAlgorithmsConfig to create_labeling_job turns on active learning
labeling_algorithm = {
    "LabelingJobAlgorithmsConfig": {
        # Built-in algorithm ARN for the task type (image classification example)
        "LabelingJobAlgorithmSpecificationArn":
            "arn:aws:sagemaker:us-east-1:027400017018:labeling-job-algorithm-specification/image-classification"
    }
}
# High-confidence objects are auto-labeled by the model; only ambiguous ones go to humans → lower cost

Active learning starts from the premise that "not all data is equally useful for training." For samples the model is already confident about, an additional label has low information value, while ambiguous samples near the decision boundary (high-uncertainty samples) have the highest information value. The approach of focusing only on those ambiguous samples is called uncertainty sampling.

Guaranteeing Label Quality: Consensus and Verification

Humans make mistakes. A single labeler can be wrong because of carelessness, bias, or fatigue. Ground Truth protects quality with several mechanisms.

• Consensus: Set NumberOfHumanWorkersPerDataObject to 2-5 so that several workers label the same object, and combine the results (majority vote or weighted).
• Annotation consolidation: The logic that merges multiple answers into one. A built-in default exists, and you can replace it with a custom Lambda if needed.
• Quality review workflow: A separate workforce reviews and corrects the results.

# Consensus: 3 workers label each object → the consolidation algorithm produces the final label
"NumberOfHumanWorkersPerDataObject": 3,
# Consolidation logic (majority vote by default) or a custom Lambda
"AnnotationConsolidationConfig": {
    "AnnotationConsolidationLambdaArn": consolidation_lambda_arn
}

One metric for measuring label quality is inter-annotator agreement (e.g., Cohen's kappa). Low agreement is a signal that the guidelines are ambiguous or the task itself is inherently difficult.

Response Table by Error Type

Label noise directly lowers the ceiling of model performance. Consensus-based labeling reduces random errors through majority vote but cannot catch systematic bias (all labelers erring in the same direction because of ambiguous guidelines). A procedure that validates against clear labeling guidelines and a golden set (samples whose correct answers are already known) is essential. There is a cost trade-off among accuracy, number of labelers, and active learning, and reducing the amount of human labeling itself is the standard approach that improves cost and quality together.

⚠️ Pitfall: If you memorize only "more labelers means better quality," you will get questions about systematic bias wrong. If the options include "clarify the guidelines" or "golden set," those are the likely answers.

Labeling Alternatives: Do You Really Need All That Data?

Before starting labeling, check three things first.

1. Can a pretrained model + transfer learning do the job with only a few labels?
2. Can data augmentation stretch the existing labels further?
3. Weak supervision — can heuristic rules produce rough labels?

When the Specialty exam asks "how do you reduce labeling cost?", active learning + transfer learning is the candidate answer.

Comparison of Cost-Reduction Measures

Human Review in Production: Amazon A2I

Labeling is needed not only before training. There is also a flow in which predictions the model is unsure of in production are handed to humans for review.

Inference request ─▶ Model prediction ─▶ Confidence threshold check
                              ├─ At or above threshold → respond as is
                              └─ Below threshold → human review queue (A2I)
                                                    ▼
                                          Store review result → accumulate as retraining data

• Amazon Augmented AI (A2I) is the service that builds this "human review loop." It shares the workforce concept with Ground Truth.
• The value is twofold. (1) Humans correct the quality of low-confidence predictions, and (2) those review results become high-quality labels for the next retraining.
• The idea is the same as active learning. The principle of sending only the ambiguous items to humans applies at both the training stage (Ground Truth) and the production stage (A2I).

Choosing Ground Truth for a passage saying "humans review low-confidence predictions in production" feels off. Ground Truth is a tool for attaching labels to a training dataset, and inserting human review into inference results is the place for A2I.

Tomorrow we synthesize all of Week 1 (lifecycle, storage, ingestion, labeling) into a single data pipeline.

📖 Glossary

• Labeling job: The Ground Truth execution unit that defines the task of attaching ground truth to data.
• Manifest: A JSON Lines file listing the targets to label. One line is one object.
• Augmented manifest: An output file in which the label results are appended to the original information. It can be used directly as training input.
• Workforce: The group of people who actually perform the labeling. Three kinds: Mechanical Turk, Private, and Vendor.
• Active learning: A labeling strategy that auto-processes what the model is confident about and sends only the ambiguous items to humans.
• Uncertainty sampling: Selecting for labeling the samples the model is most confused about first.
• Consensus: The procedure of having several people label the same object and producing a single answer by majority vote or similar.
• Annotation consolidation: The consolidation logic that merges the answers of multiple labelers into one.
• Golden set: A validation sample whose correct answers are already known. Used to check the quality of labelers and guidelines.
• Systematic bias: An error where everyone is wrong in the same direction. Majority vote does not filter it out.