Day 4 - Retraining and Model Maintenance: Automated Retraining Pipelines and A/B and Shadow Testing

📌 Key Takeaways

• Mature ML operations link drift detection → automatic retraining → safe redeployment into a single closed loop.
• For retraining triggers, event triggers based on drift or performance are more cost-efficient than schedule-based ones.
• The automation chain is Model Monitor violation → CloudWatch Alarm → EventBridge/Lambda → SageMaker Pipelines.
• Use a ConditionStep to register to the Model Registry only when the evaluation criteria pass, preventing performance regressions.
• Redeploy in order of risk: shadow (zero user impact) → A/B and canary (a small share exposed) → Blue/Green (guardrails and automatic rollback).

The Closed Loop of Maintenance

If monitoring has detected drift or performance degradation, what comes next? Having a person retrain and deploy the model by hand every time is slow and error-prone. The whole picture goes around one full cycle like this.

        ┌────────────────── Endpoint in operation ─────────────────┐
        │                                                        │
        ▼                                                        │
  Model Monitor violation                                         │
  (data/model quality, bias, attribution)                          │
        │                                                        │
        ▼                                                        │
  CloudWatch metric → Alarm                                       │
        │                                                        │
        ▼                                                        │
  EventBridge rule (or SNS → Lambda)                              │
        │                                                        │
        ▼                                                        │
  SageMaker Pipelines execution                                   │
   ├ Preprocessing (Processing)                                   │
   ├ Training                                                     │
   ├ Evaluation                                                   │
   └ ConditionStep ──fail──▶ Don't register (keep existing model) ─┤
              │                                                   │
            pass                                                  │
              ▼                                                   │
   Model Registry registration (Approved / PendingManualApproval)  │
              │                                                   │
              ▼                                                   │
   Redeploy: shadow → A/B and canary → blue/green ────────────────┘

From an MLOps perspective, MLA-C01 asks "how do you respond automatically when drift is detected?" and "which deployment strategy do you use to validate a new model version without risk?"

Signals That Retraining Is Needed

When should a model be retrained? There are three main triggers.

• Drift-based: When Model Monitor reports a data/model quality violation.
• Performance-based: When the accuracy or F1 of the model quality monitor falls below a threshold.
• Schedule-based: Periodically (for example, weekly) in domains where data changes quickly.

💡 Related theory: Retraining as often as possible is not the right answer. Retraining comes with compute cost, validation time, and the risk that the new model turns out worse. That is why an event-driven trigger like "retrain when drift is detected" is cost-efficient. Schedule-based is simple, but it can retrain even in periods with no change and waste resources. When the exam says "to reduce unnecessary retraining cost," choose a drift-based trigger over a schedule-based one.

Composition of an Automated Retraining Pipeline

On AWS, the center of retraining automation is SageMaker Pipelines. You define the steps data processing → training → evaluation → (conditional) model registration, and trigger it with an external event.

A typical event-driven flow:

1. Model Monitor sends the drift violation as a CloudWatch metric.
2. The CloudWatch Alarm goes into the ALARM state and fires an EventBridge rule (or SNS → Lambda).
3. Lambda (or an EventBridge target) starts a SageMaker Pipeline execution.
4. The pipeline retrains on the latest data, and if performance passes the criteria in the evaluation step, it registers the new version in the Model Registry.

from sagemaker.workflow.pipeline import Pipeline
from sagemaker.workflow.condition_step import ConditionStep
from sagemaker.workflow.conditions import ConditionGreaterThanOrEqualTo
 
# A condition step that registers the model only when evaluation accuracy meets the threshold
cond = ConditionGreaterThanOrEqualTo(
    left=eval_accuracy, right=0.85,        # Don't register if accuracy is below 0.85
)
condition_step = ConditionStep(
    name="CheckAccuracy",
    conditions=[cond],
    if_steps=[register_model_step],         # On pass, register in the Model Registry
    else_steps=[],                          # On fail, do nothing (keep the existing model)
)
pipeline = Pipeline(name="retrain-pipeline", steps=[process_step, train_step,
                                                    eval_step, condition_step])

Conditional registration is important. If the retrained model falls short of the criteria, it is not deployed automatically and the existing model is kept, preventing a regression.

The side that starts the pipeline from Lambda is this simple.

import boto3
sm = boto3.client("sagemaker")
 
def lambda_handler(event, context):
    # Receive the event passed by CloudWatch Alarm → SNS/EventBridge and start retraining
    sm.start_pipeline_execution(
        PipelineName="retrain-pipeline",
        PipelineParameters=[
            {"Name": "InputDataUrl", "Value": "s3://my-bucket/latest/"},
            {"Name": "AccuracyThreshold", "Value": "0.85"},
        ],
    )

The Model Registry manages model versions as a catalog and gives each version an approval status (Approved/Rejected). If the retraining pipeline registers the new version as "PendingManualApproval," you can make the deployment pipeline (for example, CodePipeline) trigger only when a person approves after review. You choose between fully automatic (deploy immediately when the condition passes) and human approval (human-in-the-loop) according to the level of risk.

Redeployment Strategies at a Glance

Even if a new model did well on the evaluation data, there is no guarantee it will be better on real traffic. So you choose a deployment strategy by "how much risk you are willing to take."

Shadow and A/B have different purposes. Shadow is a technical validation that checks "does the new model not blow up on production traffic?", and A/B is an outcome validation that checks "does the new model actually produce better results?" Shadow has no user response, so business metrics like click-through rate and conversion rate can't be measured at all. So the practical order is shadow → A/B.

Safe Redeployment 1: A/B Testing

A/B testing puts several production variants on one endpoint and splits traffic to compare them. For example, you start with 90% on the existing model and 10% on the new model, and raise the share gradually if the new model performs well.

from sagemaker.session import production_variant
 
variant_a = production_variant(model_name="model-v1", instance_type="ml.m5.xlarge",
                               initial_instance_count=1, variant_name="ModelV1",
                               initial_weight=9)        # 90% of traffic
variant_b = production_variant(model_name="model-v2", instance_type="ml.m5.xlarge",
                               initial_instance_count=1, variant_name="ModelV2",
                               initial_weight=1)        # 10% of traffic
# Deploy both variants on one endpoint
session.endpoint_from_production_variants(name="ab-endpoint",
                                          production_variants=[variant_a, variant_b])

The weights (initial_weight) control the traffic split, and since you can change the share in real time, it is used for gradual rollout (canary).

# Raise the new model's share to 50% by adjusting only the weights, without redeployment
sm.update_endpoint_weights_and_capacities(
    EndpointName="ab-endpoint",
    DesiredWeightsAndCapacities=[
        {"VariantName": "ModelV1", "DesiredWeight": 5},
        {"VariantName": "ModelV2", "DesiredWeight": 5},
    ],
)

• Latency and error metrics appear in CloudWatch split by the VariantName dimension, so you can compare per variant.
• To compare business metrics (such as conversion rate) too, you need to link prediction results to subsequent user behavior.

Safe Redeployment 2: Shadow Testing

In A/B testing, the new model actually responds to some real users. If that response is bad, those users are affected. Shadow testing removes this risk. It also sends a copy of production traffic to the new (shadow) model, but the shadow model's response is not returned to the user and is only recorded. The user always receives the existing model's response.

             ┌──▶ Production model v1 ──▶ Response ──▶ [User]
  Request ───┤
             └──▶ Shadow model v2 ─────▶ Store only logs and metrics (does not go to the user)

# SageMaker Shadow Test: deploy a production variant and a shadow variant together
# The shadow model receives the same input and runs inference, but the result is only logged and not sent to the user
shadow_config = {
    "ShadowModelVariants": [
        {"ShadowModelVariantName": "model-v2-shadow", "SamplingPercentage": 100},
    ],
}

Shadow testing is the right answer when you want to "validate how the new model behaves on real production traffic without risk." After comparing latency, errors, and prediction distribution without user impact, if you are satisfied you move on to A/B or a full replacement. Lowering SamplingPercentage reduces the shadow inference cost.

⚠️ Trap: It is easy to confuse A/B testing and shadow testing. In A/B, the new model's response actually goes out to real users (some risk exists, and outcomes can be compared with business metrics). In shadow, the new model's response does not go out to users and is only recorded (zero user impact, purely technical validation). If it is "validate the new model with production data without affecting users at all," it is shadow, and if it is "split traffic and shift gradually," it is A/B.

Deployment Guardrails

For a full replacement, you shift safely with deployment guardrails. You bring up a new environment with Blue/Green, then shift all at once or gradually with canary or linear, and if a CloudWatch alarm fires, it automatically rolls back.

• Automatic rollback works only when tied to a CloudWatch alarm. If you don't set an alarm, you end up in a state of "a guardrail deployment that can't roll back."
• During the shift, the old and new environments are up at the same time, so capacity cost temporarily doubles.

When Retraining and Redeployment Go Wrong: Symptom → Cause → Action

In the next post we do a comprehensive review of all of Week 8: data/model quality drift, bias and explainability drift (Clarify), operational monitoring (CloudWatch/X-Ray), and retraining and redeployment.

📖 Glossary

• Closed loop: An operations structure that goes around by itself: detect → trigger → retrain → redeploy → detect again.
• Event-based trigger: A method that starts retraining only when an event such as a drift or performance violation occurs, not at a fixed time.
• SageMaker Pipelines: An ML workflow service that runs preprocessing, training, evaluation, and registration steps together. The center of retraining automation.
• ConditionStep: A conditional branching step that runs the next step only when the evaluation metric exceeds the criteria. A safeguard against performance regressions.
• Model Registry: A repository that manages model versions as a catalog and assigns an approval status (Approved/PendingManualApproval, etc.).
• Production variant: A model slot that coexists within one endpoint. Weights divide the traffic share.
• A/B testing: A method that divides traffic with variant weights to compare the new model's actual outcomes. The new response goes to users.
• Shadow testing: A method that sends a copy of production traffic to the new model but only records the response. Zero user impact.
• Canary / linear shift: Gradual shift methods that move traffic to the new version a small amount first / in steps of a fixed percentage.
• Deployment guardrails: A safe deployment mechanism that attaches CloudWatch alarm-based automatic rollback to a Blue/Green or canary shift.